<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavel Mrázek on Medium]]></title>
        <description><![CDATA[Stories by Pavel Mrázek on Medium]]></description>
        <link>https://medium.com/@pavel.mrazek?source=rss-36832e81eeb8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ckTM-xIEiSeAGcT3</url>
            <title>Stories by Pavel Mrázek on Medium</title>
            <link>https://medium.com/@pavel.mrazek?source=rss-36832e81eeb8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:10:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pavel.mrazek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Approach Software Process Improvement]]></title>
            <link>https://medium.com/manta-engineering-blog/how-to-approach-software-process-improvement-c937b2473832?source=rss-36832e81eeb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c937b2473832</guid>
            <dc:creator><![CDATA[Pavel Mrázek]]></dc:creator>
            <pubDate>Fri, 21 Apr 2023 19:44:54 GMT</pubDate>
            <atom:updated>2023-04-21T19:50:21.261Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="software process improvement" src="https://cdn-images-1.medium.com/max/1007/1*Wwa38KXBJsjxe7h4oC3nDQ.png" /></figure><p>In the first part of this two-part blog series, I talked about why I care about software process improvement. If you haven’t already read it, you can do so here. Now, we’ll talk about the phases you’ll go through when putting your software process improvements in place.</p><h3>Phase 0: Before We Start</h3><h4>Take Ownership</h4><p>I always try to complete my homework first. Simply said, the basic prerequisites should be fulfilled before I proceed. To start with, I do my best to perform in my current role with the tasks I’m assigned, while any improvements are considered something extra. Secondly, taking ownership is necessary. If the team needs to narrow the Maven repository configuration, looking for experts is usually not the fastest way. We probably need to gain a reasonable level of expertise quickly enough to be able to change anything soon.</p><h4>Quantifying the Costs</h4><p>I also try to do at least some basic calculations around whether the effort will pay off. Probably the first question your boss will ask you is “How much will it cost? How many man-days will you need? “ The bad news is that saving money in the long term will cost you something upfront. But being able to quantify the costs and benefits can give you the right advantage and convince your boss.</p><p>We considered purchasing the licenses of IntelliJ Idea Ultimate for our full-stack development team. The team wanted to leverage the embedded application server with its hot-swap functionality. This would speed up the three-minute build and redeploy turn-around.</p><p>Doing this 10 times per day wastes 30 minutes every day. Multiply it by 150 to get the impact in one year. The license price per day was about three dollars. Was saving 30 minutes of the programmer’s productivity worth it? Absolutely. Was the work comfort and satisfaction worth the price? Of course (although no calculation can be used to justify it).</p><p>Is it always possible to quantify the benefits? No. How many man-days or bugs do we save when introducing static code analysis, fostering a code review process, improving the quality of the project documentation, and involving an agile approach? In such cases, relying on the industry’s best practices and trusting common sense are the only weapons we have.</p><h4>Diplomacy</h4><p>Last but not least, in today’s complex development environments and projects — and big teams working within them — we would barely leave any track if we played the game alone. I always make sure I have the support of my team, the project manager, and the customer … Sooner or later, I will ask them for help, a time slot in the release schedule, paying for licenses, extra CPUs in the testing environment, backup server, consultations, or more importantly, for changing their mindset.</p><p>I listen and engage in informal small talks in the kitchenette over coffee, and joint lunches, trying to absorb the whole story, spreading the word, and slowly buying my colleagues in. Then, at the team meeting, when folks hear me talking for the third time about that urgent change, it’s no surprise for them anymore and a couple of heads may even nod.</p><h3>Phase 1: Observing and Understanding the Project</h3><p>What helps me identify the most neglected areas and most severe bottlenecks is the understanding of the big picture. Otherwise, any change would be just a guess with a random result. I need to understand all the puzzles of the development process. Who does what and why?</p><p>Prioritize. If you gathered many observations, select the most serious insufficiencies solvable in a reasonable amount of time with minimal resources, while bringing the most appreciable effect. Such a quick win can buy you the trust of management or your team for your next bigger challenges. I personally use a good old Excel sheet with the following columns: severity, difficulty, area, current state, consequences if not solved, proposed solution, assignee, dependencies, and effort estimate.</p><h4><strong>Example</strong></h4><p><strong>Severity: </strong>High</p><p><strong>Difficulty: </strong>Medium</p><p><strong>Area: </strong>Testing</p><p><strong>Current state</strong></p><ul><li>Qualification testing takes place in the very last week at the end of the development period and is often not completed in full.</li><li>The process is not systematically planned (resources, time, quality, scope), repeatable, and controllable.</li></ul><p><strong>Consequences</strong></p><ul><li>Degraded quality, a high number of errors</li><li>Team leaders are overwhelmed with bugs from production and they don’t have enough time for improving the quality of the development process itself and managing their teams properly</li></ul><p><strong>Proposed solution</strong></p><ul><li>Implement test methodology/strategy and test plan</li><li>Collect metrics from the development, testing, and production</li><li>Introduce continuous (dry-run) testing of all new features</li><li>Hire more testers</li><li>Prolong the qualification testing period by one week</li><li>Implement automated end-to-end tests</li></ul><p><strong>Assignee: </strong>Test lead, Delivery manager</p><p><strong>Dependencies: </strong>Make the developers responsible for the quality of the developed features</p><p><strong>Effort estimate</strong></p><ul><li>2MD test strategy and plan (one-time)</li><li>1MD collecting metrics (each release)</li><li>50MD one additional tester doing the dry-run testing (each release)</li><li>40MD prolonged QT</li></ul><p><strong>Benefits</strong></p><ul><li>50% fewer production bugs</li><li>30 MD less production support</li><li>Predictable qualification testing</li><li>Predictable production quality</li><li>Improved customer satisfaction, lower customer churn, and higher annual revenue</li></ul><h3>Phase 2: Planning</h3><p>You can see your improvement as a regular project. Usually, it needs a justifiable business case, clear scope (who should do what, when, and how), resources, time (possibly split into several phases or milestones), and a rollout of your project. Having a plan significantly improves the transparency and manageability of the whole activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*VCfan-6QVbfZfT7bd6ZhbQ.png" /><figcaption>List of the activities comprising the project of codebase migration from SVN to Git</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIBnl8S6wjuOPCsGerIjSQ.png" /><figcaption>A simple Gantt chart of these activities showed the dependencies and the expected schedule</figcaption></figure><h3>Phase 3: Rollout</h3><p>The final day has come and the shiny rocket is ready to be launched. However, for risky or big changes, I tend to progress gradually in small steps. For instance, we enabled the SonarQube pull-request analysis in one team only. Once the expected behavior was verified and Jenkins resources monitoring didn’t show any extremes, we enabled the analysis for the rest of the teams as well.</p><p>Similarly, having a rollback option, a plan B, will buy you a good night’s sleep. In the case of Git migration, we had a backup server ready. Even if everything goes smoothly, I always like to communicate all changes in advance and provide aftercare support to resolve potential issues, assist others, and answer any questions.</p><h3>Conclusion</h3><p>Let me summarize what we found useful and what helped us make some of the improvements possible.</p><ol><li>Start the observation right after joining the team/company before you get used to overlooking all the shortcomings.</li><li>Every project is unique. Technology comes and goes,. but the principles remain and are worth getting right from the start.</li><li>The entire development process deserves your attention, not just the implementation and code.</li><li>Ask multiple people for their opinion, with open-ended but specific questions. This doesn’t work: “What would you improve?” Try this instead: “How many development bugs were there, and how many did you manage to correct and test by the end of release?” or “In which module do you have the least test coverage and what percentage is it?”</li><li>Use checklists to repeat the successes and avoid stupid mistakes. Create your personal checklist where you take away acquired knowledge and lessons from every project.</li><li>Find out what the biggest thorn in your side is. Support your software process improvement proposal with clear arguments and/or calculate the benefits.</li><li>Be grateful for the critics in your team. They bring your overly optimistic ideas back down to earth. Divide the expected benefit by two.</li><li>Get your colleagues involved in the game and you will have more fun.</li><li>Be prepared that the activity may delay you now, but it will help us in long term.</li><li>Start small, in your immediate surroundings, and gradually, with some quick wins. Do first things first and evaluate the impact of the changes individually.</li><li>It does not end with the introduction of change. You should also introduce a control mechanism to adopt and keep the change.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c937b2473832" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manta-engineering-blog/how-to-approach-software-process-improvement-c937b2473832">How to Approach Software Process Improvement</a> was originally published in <a href="https://medium.com/manta-engineering-blog">MANTA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Care About Software Process Improvement]]></title>
            <link>https://medium.com/manta-engineering-blog/why-i-care-about-software-process-improvement-748b5dbacd4c?source=rss-36832e81eeb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/748b5dbacd4c</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Pavel Mrázek]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 12:01:35 GMT</pubDate>
            <atom:updated>2023-03-23T12:01:35.033Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzLbdSziKpeGx6mOLvDgvw.png" /></figure><p>A quick Google search will tell you that at least 25% of all software projects fail. Only 67% of projects are completed within budget, and a similar number of projects are completed on time. This makes me consider how mature the development process is at MANTA. What happens when we neglect the technical debt or simply stop evolving the development process? How can I raise the bar as an individual tester, developer, analyst, team leader, or project manager? How can we successfully suggest, implement, and adopt an improvement?</p><p>When I joined MANTA as a delivery manager, the current level of engineering was already mature enough to satisfy the requirements of our global customers. At that point, MANTA’s major version was already 24 releases — 24 times, the team delivered a working solution to the production.</p><p>My boss told me: “Everything kind of works here.” He was right in many aspects. The development process was settled and generally followed. Highly modular architecture allowed the team to work in parallel without unnecessary conflicts when pushing the code. The continuous integration (CI) system made the best use of the unit tests, static code analysis tools, precisely configured build configuration, and dependency management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*P5zW7DW_5hB2rn-UpwwOhw.png" /></figure><p>The team had automated major parts of the release process and employed various development tools to make their work more effective (validator of Maven POM configurations, OWASP vulnerability check in external libraries). An issue tracking system had been in place in connection with defined configuration management.</p><p>Having said all of that, my boss concluded: “…and it can be infinitely fine-tuned”. I was lucky enough to get this empowerment and trust to work on it. Long-term development projects and our product development environment allow us to observe the trends, measure, compare recurring moments, improve, and evaluate the impact of the changes introduced.</p><h3>Software Development is More than Programming</h3><p>Since then, the engineering department has grown and matured a lot and many experienced developers, testers, UX professionals, and DevOps engineers have enriched our 13 teams. But initially, the team was a diverse mixture of motivated young engineers with strong connections to technical universities. Thus, it was quite natural that the focus was strongly programming-centric, while other disciplines of software engineering were not getting the attention they deserved. We decided to balance this.</p><p>I know that saying “SW development is not just programming” sounds provocative, and that’s absolutely intentional. I am in no way diminishing the importance of implementation. I’m a software developer at heart, probably like most of you. But let’s have a look at the development as a whole, with a broader perspective. In my experience, the usual share of implementation effort is between 30–50% out of all commercial project activities, depending on the project size. The bigger the project, the less programming is usually involved and the more share is taken by project management, architecture, analysis, documentation, integrations, and testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*kMRguQaVRksdLNiz9l_XAw.png" /></figure><p>With an implementation share of 35% at MANTA, the importance of the other slices of the cake becomes obvious. Of course, one can still deliver a top-notch product with an inferior production line, but having a top-notch production line increases the probability of producing much better output.</p><h3>Major Risks of SW Projects</h3><p>To support the above-mentioned claim, let’s look at the importance of a holistic view from another perspective. What represents the main problems in the development of SW systems?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uapAUoHtWcS52NKvwW2T-g.png" /></figure><p>A few years spent in the industry or a few minutes of Googling could probably compile a similar list of risky areas to be cautious of. Although implementation-related risks are very important, they are not the only risks and aren’t prevalent. The causes for failure can also be “political” (e.g., reorgs, budget cuts, competition taking advantage, and game-changing decisions by the department of architecture). For now, let’s stay with the engineering practices we can more or less influence.</p><p>Another perspective. My favorite first step after entering a new company or team is to ask developers about the biggest problems they’re facing on their projects. Let’s interview our fictitious developers Adam, Ben, and Caroline to see the common classes of answers I frequently get.</p><ul><li><strong>A</strong>dam: “I don’t like the duplication in CSS. Oh, and the <em>DoItAllServiceImpl</em> class has 3,000 lines and is a total mess.” Alright, this is a bit too detailed of a problem. Let’s see if we can reveal more severe impediments.</li><li><strong>B</strong>en: “Let’s plug in a new check-style rule. If correct code formatting is not observed, the build will fail.” Well, in cases when the release is at stake, this improvement actually makes things rather worse.</li><li><strong>C</strong>aroline: “What do you mean by design? Everything’s OK.” She doesn’t know what she doesn’t know. That’s probably the most serious case.</li></ul><p>As we can see, typically little is said about analysis, documentation, testing, CI, releasing, organization of work, or the number of regression errors in production. If you ask the developers, you’ll get bits and pieces, but you’ll have to stitch together the big picture on your own to have a complete understanding of and identify the areas of improvement that could bring the most value.</p><h3>Why Improve?</h3><p>Why should we care about the development process that much? Because we have to succeed. Let me give an example regarding project management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*004QJayFCagyipM33lPPKw.png" /><figcaption>Source: <a href="https://www.theknowledgeacademy.com/blog/the-skills-and-values-of-efficient-project-managers">The Knowledge Academy</a></figcaption></figure><p>At MANTA, project management is only one-tenth of the whole pie to eat. And even <a href="https://www.theknowledgeacademy.com/blog/the-skills-and-values-of-efficient-project-managers">this small part can ruin 32% of all projects</a>. That’s very alarming. Generally, projects with malformed or totally lacking a well-shaped development process, or projects with growing undetected technical debt, are sooner or later deemed to fail.</p><p>It’s even harder to spot those at first glance. Such projects just sort of keep working, until they get stuck slowly or fail fast. I always like to remind myself of this metaphorical whip behind my back. Luckily, there is always a carrot hanging in front of our noses making me strive for a better development process. Besides that, there are many pragmatic reasons to kickstart the improvements:</p><ul><li>I’ll get to know the project from the perspective of processes or architecture</li><li>I can learn the good and bad of the libraries, frameworks, and tools</li><li>I will get in touch with other people</li><li>I will explore the dead ends. Hopefully, I will be able to show it can be done</li><li>I’ll help my colleagues simplify their work or level up the results of their work</li><li>I will make my boss happy and maybe even boost my career</li></ul><h3>“We’ve Always Been Doing it That Way Here”</h3><p>Let’s imagine you finally rolled up your sleeves and presented your ideas to the team. Suddenly, your colleagues come up with many counter-arguments as to why your proposals wouldn’t work in practice:</p><ul><li>We’ll do it when there’s more time.</li><li>Code review? What for? You can program, right?</li><li>You know this is a very specific project…</li><li>We can’t have a project plan because of the frequently changing requirements.</li><li>What do you mean? Everything is working fine!</li><li>It has been broken for a long time.</li><li>You can’t do anything with it.</li><li>We’ve always been doing it that way here.</li><li>What are you reading those thick books for? You won’t use it here anyway.</li><li>OOP would be nice, but there are already so many IFs in the code that it has no meaning.</li><li>We do not have checklists. Everyone does it their own way here</li><li>The integration department is to blame.</li></ul><p>Do these points sound familiar? Such never-adopters can be recognized by their excuses. On the other hand, we don’t want to throw the baby out with the bathwater. Some counter-proposals can be perfectly reasonable. Take an example. The formal code review will be widely adopted once we have migrated to Git and feature branches will be the only way to integrate incremental changes into the main development branch. Fortunately, MANTA cherishes an open and friendly culture where feedback and striving for improvement are always welcome.</p><h3>Antipatterns</h3><p>On the other hand, I recall several moments when I myself was overly enthusiastic about innovation, which made my colleagues mad. See some of the anti-patterns I followed by mistake:</p><ul><li><em>“Improvements are what is most important. Development can wait.”</em> I did not know or respect the priorities of other people.</li><li><em>“I understand that bugs get fixed before the release, but I also want you to start doing continuous code reviews.”</em> Ouch, bad timing.</li><li><em>“Why are you still on Java 8? We’re upgrading to Java 11” </em>I had no clue about the actual reason for Java 8.</li></ul><h3>To Be Continued…</h3><p>In this part of our two-part blog series, I talked about why I care about software process improvement. In part two, I’ll talk about the “how” — that is, the steps you can take to actually accomplish that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=748b5dbacd4c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manta-engineering-blog/why-i-care-about-software-process-improvement-748b5dbacd4c">Why I Care About Software Process Improvement</a> was originally published in <a href="https://medium.com/manta-engineering-blog">MANTA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating to Git — A Piece of Cake Or a Full-blown Engineering Project?]]></title>
            <link>https://medium.com/manta-engineering-blog/migrating-to-git-a-piece-of-cake-or-a-full-blown-engineering-project-cc85b4bd5cfe?source=rss-36832e81eeb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc85b4bd5cfe</guid>
            <dc:creator><![CDATA[Pavel Mrázek]]></dc:creator>
            <pubDate>Mon, 06 Jun 2022 14:02:34 GMT</pubDate>
            <atom:updated>2022-06-06T14:04:07.716Z</atom:updated>
            <content:encoded><![CDATA[<h3>Migrating to Git — A Piece of Cake Or a Full-blown Engineering Project?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rescDMuBgen92QB6C3apqw.jpeg" /></figure><h3>How It All Began. Do We Really Need Git?</h3><p>Before migrating to Git, we had been using Subversion to store all our source code for more than a decade. SVN did its job well and accompanied us on the path from a very small experimental tool to a commercially successful automated Data Lineage Solution. However, discussions about having all our code versioned in Git were becoming more frequent. Our developers had been asking for it (well, some of them). Git is popular among the majority of our potential colleague developers out there and we should stay competitive. Google trends advocates Git, and Git also takes the biggest share of the market. Most importantly, MANTA is a leading technology in the data lineage space, so we are continuously looking at which tools to adopt and which to abandon.</p><p>Personally, I am not a fan of heedlessly experimenting with the shiny allurements of the latest trends. While it’s important to adapt and innovate, the “latest and greatest” trend can sometimes overshadow the importance of established principles and practices, such as good SW architecture, design patterns based on OOP, CI systems, static code analysis, a well-defined software process regardless of the selected methodology, automated tests, etc.</p><p>That said, there are many things I love about Git, such as easy branching, fast commands execution, usage of pull requests, integration with numerous tools and systems, powerful scripting possibilities, and the ability to scale to support our growing team of developers.</p><p>These benefits are the low-hanging fruit. Besides migrating to another source code management system, there is another dimension — the overall tuning of our development pipeline.</p><h3>Planning Of the Migration To Git</h3><p>Once we started thinking about the migration we revealed more and more complexities. We found out that it will affect the structure of the repository, branching model, build system, dependency management, continuous integration, release process, daily development routine, code review, etc. It was already clear that we were not talking about any “one script does it all” task which can be finished over a weekend. We needed a plan to make sure we all understood the what, who, when, and how. The plan contained a work breakdown list of approximately 100 tasks comprising 200 MD of work across several teams.</p><p>Do not underestimate planning. Your boss’s first questions will probably be: “How much will it cost? How many people will you need? When will you be done?” I admit I was thankful that the VP of Engineering was absolutely on the same page and we approved the plan while understanding it was a real project with all its phases, prerequisites, schedule, and dependencies. Artificially pushing down the costs to get people’s vote usually leads to even bigger disenchantment in the later phases of the project.</p><p>Here we were, a little team of three engineers and a plan for the migration of the whole codebase from SVN to Git. We worked out a list of activities needed to finish the whole migration:</p><ul><li>Craft a migration plan to streamline the final migration</li><li>Rollback plan or plan B if things go wrong</li><li>Communication with the development team about the planned changes that will affect their work</li><li>Analysis is divided into several parts (we will get to it in the next chapter) describing the impacts and the technical solution</li><li>Acceptance criteria to make sure we did not miss anything important and the final migrated codebase is complete, not corrupted, consistent</li><li>Training to ensure everyone was familiar with even advanced Git commands and techniques</li><li>Configuration and administration (authorization, authentication, access rights, user groups, establishing repositories, branch settings, disabled force push, protected branches, notifications)</li><li>Infrastructure (HW, SW, hosting, RAM, HDD, CPU, monitoring)</li><li>Test environment (Git, Jenkins, SVN, everything backed up)</li><li>Implementation (migration scripts, impacts on the build tools, utilities, CI, release scripts, Git hooks)</li><li>Be prepared to run the migration procedure repeatedly before everything is perfect. The migration will probably require manual steps.</li><li>Install the local development environment (all developers will need to install Git, configure it, generate SSH keys, clone the repo, set up IDE, and migrate work in progress manually).</li><li>Baby-sitting: after-migration support, solving issues, helping others, fixing broken builds, new requests, configuration, performance tuning, monitoring</li><li>Initial lower efficiency — the developers will need some time to get to full speed</li><li>Time for some overhead activities<br>- Meetings with the migration team and the stakeholders (architects, DevOps team, managers)<br>- Consultations</li><li>Documentation of the development routines (specific branching strategy, release procedure, changes in CI, changes in code review, impacts on existing utilities, installation instructions)</li><li>Buffer for the unknown. I’d recommend a 20% to 50% risk buffer, depending on how brave you are and how established you are with your team and environment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2nkkzTBNA0vxqen-eIlvg.png" /><figcaption>A simple Gantt chart of these activities showed the possibilities of parallel activities, their dependencies, and the expected schedule.</figcaption></figure><h3>Analysis</h3><p>Knowing there would be so many important topics to consider well, we decided to proceed as if we were developing a piece of software. Before jumping into implementation, we needed to make sure all relevant solutions and their impacts, pros, and cons were considered as well. Each topic was analyzed, and all assumptions were approved by the group of architects and selected team leaders of the development teams.</p><h4>Monorepo Versus Many Repos</h4><p>Our codebase resided in hundreds of independent SVN repositories and a few existing smaller Git repositories with their own commits history. Additionally, each repo was structured into the trunk, branches, and tags. Our task was to decide if we shall merge all product-related repos into one monorepo or go with many little repos independently.</p><p>Initially, we gathered all possible requirements on the repository structure and how we wanted to work with it. Some questions we asked ourselves included:</p><ul><li>Can new projects be added with ease?</li><li>How do we restrict user rights to grant access to certain user groups to certain repositories or parts of a repository?</li><li>How fast will the clone be?</li><li>How could the checkout of a branch in a monorepo influence the IDE? Could the reindexing of the whole project in the IDE cause its crash?</li><li>How will we refactor a code spanning several modules?</li><li>How to merge the history of the individual SVN repositories into a single history log?</li><li>How to merge the existing Git repositories into the monorepo while persisting their history log?</li><li>Shall we retain individual module-related release branches or merge them into one big single branch?</li><li>What do we do with the historical module-related tags?</li><li>How would the code review be structured in a monorepo / multiple repos?</li><li>How to merge the changes atomically?</li><li>How shall the experimental projects be treated?</li><li>How shall the folders be restructured?</li><li>How to migrate SVN commit ID to Git commit log?</li><li>How to transform svnignore files to gitignore files?</li><li>What to do with the deleted files and folders?</li><li>What to do with the files stored aside from the standard trunk-branches-tags structure?</li></ul><p>The conclusion of the analysis was: “The fewer repos we maintain, the fewer problems we will have. Especially releasing and inter-module development will be easier.” It showed that scaling of one single Git repo is not a problem. Git clone speed was 20 minutes compared to several hours in SVN. The final monorepo hosted all application modules that are to be released together and their versions should be kept in sync. We also created additional twenty small Git repositories dedicated to the utilities and experimental projects.</p><h4>Selecting a Proper Branching Model</h4><p>Now that the structure of the repository was defined, the next step was to lay out a branching strategy. To start with, we adopted the widely used Git-flow model. It was a huge step to move away from the trunk-based development to more complex but powerful feature branches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*eaHo-_rCJO0LfhFBeD-tXw.jpeg" /><figcaption>Inspired by <a href="https://nvie.com/posts/a-successful-git-branching-model">https://nvie.com/posts/a-successful-git-branching-model</a></figcaption></figure><p>However, this model does not follow up with a software product like MANTA which is deployed and maintained in several parallel production versions. Also, tagging must be done differently to address a different release process. We also practice a so-called “minor release” which utilizes a slightly different branching and merging procedure.</p><p>We reshaped the given model to respond better to our needs:</p><ol><li>Only 1 permanent integration branch (develop)</li><li>No master branch.</li><li>There can be several parallel release branches. All tags are placed on the release branches.</li><li>A feature branch is merged to develop using a switch — no-ff to create a merge commit for evidence and easier cherry-picking of a single merge commit.</li><li>All development (features and bugfixes) is done via feature branches (as short-lived as possible).</li><li>Feature branches branch off from the integration branches (develop, release branch).</li><li>Regular pushing to the origin, voluntary commits squashing, forbidden force-push, and regular rebasing of feature branches on the head of the integration branch.</li><li>We use pull requests and all code must pass a code review.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*0uQxEw1ocoMR25rCi0aTZg.png" /><figcaption>Branching model</figcaption></figure><h3>Continuous Integration</h3><p>Our branching model was completely redefined. The workflow consists of the feature branches merged as pull requests. In contrast to the past, we wanted to incorporate the release branches into our CI to be able to package and test the new version heading to production.</p><p>We used Jenkins Maven jobs to continuously integrate the changes in the main development branches such as develop and release branches. The chain of the dependencies is represented by upstream and downstream Jenkins projects based on the maven configuration (pom.xml files). The build of one subsidiary module triggers the superordinate jobs in Jenkins.</p><p>Some types of branches, such as feature branches, are created on the fly and are automatically tracked in Jenkins. We used the “Multibranch Pipeline” and Git plugin to notify Jenkins about recent changes. The pipeline is defined in the “Jenkinsfile” describing the build behavior for different types of branches and separate stages of the build. In fact, we build only the pull requests, not the underlying feature branches.</p><p>Having more than 150 required a certain level of automation. We call it Jenkins utility and it simplifies bulk operation on the Jenkins jobs such as creating, pausing, cloning, etc.</p><p>If you are curious about the insidious details, check my <a href="https://medium.com/@pavel.mrazek/65cab4df80c">blog post about CI and a multi-modular monorepo</a>. It elaborates on the more advanced topics of:</p><ul><li>When to use regular polling</li><li>How to detect and build only the changed modules in a single PR</li><li>Managing direct and transitive dependencies</li><li>Avoiding conflicting versions of artifacts in different branches</li><li>When to deploy the artifacts into the Nexus repository</li><li>Git sparse-checkout to fetch only sources in the affected parts of the repo</li></ul><h3>Build and Release Process</h3><p>This was a real challenge. Remember, our code-base is fragmented into one thousand modules which form a complicated net of dependencies. These, however, are assembled as a few end products (applications) and distributed via an installer wizard and docker images in a public Nexus repository. The release process is done centrally for all modules. We support three kinds of releases: major, minor, and hotfix. Each has its unique characteristics but also some common features that were automated and simplified. And we were not allowed to break anything.</p><p>In the middle of it all stands the Maven release plugin wrapped in our automated script and extended with a custom Maven release strategy. The tricky part was to do the versioning schema right. More specifically, assigning the right (SNAPSHOT and release) versions to all modules (automatically) in different Git branches while the branches are born and die and are merged and artifacts and their aggregate modules are released and re-released when something goes wrong.</p><p>Our release process is far from perfect. We knew that before the migration and we know it now. However, we had to descope several improvement ideas from the migration backlog to finish in a reasonable time.</p><h3>Migration Script</h3><p>To be able to streamline the migration itself required a maximum amount of automation. The following flow chart depicts distinct parts of the migration shell script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-B61vW74S8ZMj2n_DXfNTw.png" /><figcaption>Migration script flow chart</figcaption></figure><ul><li>The flow diagram depicts the steps of the automated migration script.</li><li>The left branch represents the original SVN repo on the input.</li><li>The right branch shows how the original smaller Git repos were transformed into their raw (bare) format to enable fast processing. Additionally, tags and release branches were given a unique name to reflect the module name.</li><li>The original SVN repository was cloned (checked out).</li><li>Mapping of SVN commits authors to new Git user accounts.</li><li>The rules for the migration were captured in the *.rules files. They consisted of<br>- List of all SVN modules<br>- Mapping of original SVN addresses to new addresses in Git.<br>- Mapping of tags and branches and folders.</li><li><em>“*.rules</em>” were used by <a href="https://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git">svn2git utility</a> to transform the SVN repo into a single Git repo. We slightly customized svn2git to use our merge strategy and enrich the commit message with the original SVN revision ID.</li><li>The history of commits has been cleaned.</li><li>The first test suite was triggered to verify the correctness and completeness<br>- The content of develop branch corresponds to the trunks of the individual modules<br>- No files or directories can be missing or be redundant<br>- All feature branches from the original repositories should be present</li><li>Then, the main monorepo was merged with the existing smaller Git repos via the <a href="https://medium.com/swlh/migrating-to-monorepo-ee641dd6502a">ToMono utility</a>.</li><li>The individual modules’ release branches were post-processed (merged) so that every MANTA major release was represented by a single release branch common for all modules.</li><li>The final post-processing consisted of several steps<br>- Maven aggregation pom.xml was added to the root of every branch.<br>- Jenkinsfile was added to the root of every branch.<br>- CRLF settings in <em>.gitattributes<br>- </em>One central <em>.gitignore</em> constructed</li><li>Additional test suite T2 is triggered.</li></ul><h3>The Final Migration Weekend</h3><p>After months of preparations, and several dry run migrations, we approached the “D Day” of the final weekend migration. I can’t stress enough the importance of the dry run migrations we undertook to fine-tune the migration script and the manual steps of the process. We tried the usual development scenarios with full integration with other systems such as Jenkins. We had to be sure everything would go smoothly and on time to ensure migration would be completed over a single weekend.</p><p>Finally, everything was in place and ready to start:</p><ul><li>Detailed migration plan with clearly described steps and responsibilities including a time schedule</li><li>Production environment with sufficient resources (HDD, CPU, RAM)</li><li>All systems and tools installed and pre-configured (Git, Gitea, SonarQube, Jenkins, Nexus, etc.)</li><li>Repositories prepared and user groups configured</li><li>Migration scripts ready and tested</li><li>Documentation ready to be published in Confluence</li><li>SVN backed up</li><li>Acceptance criteria written down</li><li>Jenkinsfile implemented and pom.xml changes prepared</li><li>Jenkins utility ready to ramp up all 150 Jenkins jobs pointing on Git repo using sparse checkout</li><li>Informative emails and user instructions written in advance</li><li>Developers trained in advance</li></ul><p>Despite all the preparations, we were under time pressure and made a few mistakes. The migration was successful on the third try and we finished literally 5 minutes before midnight.</p><h3>Avoiding Trouble</h3><p>Let’s briefly mention the challenges and risks we were facing or those we were afraid of. Some of them were completely (luckily) avoided, and some of them were only mitigated partly. Some of the issues were clear from the very beginning, some pits were hidden deeper under the surface waiting for us to fall into them. When undertaking a migration, consider the following risks and possible mitigations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*wnfq6o1PPOUlv6TC_0cw5A.png" /></figure><h3>Final Words</h3><p>Migrating to Git was preceded by careful consideration of whether we really needed it and if we actually wanted it. We wanted to take advantage of the powerful scripting possibilities and Git’s ability to scale to support our fast-growing team. Apart from switching the versioning systems, we substantially changed our development process which proved even more challenging and time-consuming.</p><p>However, what helped us to manage it, was taking an approach to a regular software project with all its requirements, phases, risks, people, resources, milestones, schedule, issue tracking, and testing.</p><p>In the end, the project took one year (twice as long as expected) to finish and we spent 300 MD in total.</p><p>Finally, hundreds of SVN modules were transformed into twenty-five independent Git repositories dominated by a multimodule monorepo hosting eight million lines of source code. We were able to preserve the history of commits and we utilized an enhanced branching model. The developers can enjoy more granular units of work in the form of pull requests undergoing a code review and a regular continuous integration. The release process was centralized and access rights to the repositories were simplified.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc85b4bd5cfe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manta-engineering-blog/migrating-to-git-a-piece-of-cake-or-a-full-blown-engineering-project-cc85b4bd5cfe">Migrating to Git — A Piece of Cake Or a Full-blown Engineering Project?</a> was originally published in <a href="https://medium.com/manta-engineering-blog">MANTA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SW Development Process At MANTA]]></title>
            <link>https://medium.com/manta-engineering-blog/sw-development-process-at-manta-5dc6bd224942?source=rss-36832e81eeb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dc6bd224942</guid>
            <dc:creator><![CDATA[Pavel Mrázek]]></dc:creator>
            <pubDate>Mon, 06 Jun 2022 07:32:59 GMT</pubDate>
            <atom:updated>2022-06-09T08:28:43.095Z</atom:updated>
            <content:encoded><![CDATA[<p>Can our development team scale from 4 to 70 developers in a few years while using a source code mono-repo? We will talk about the core principles of the source code modularity in connection with a branching model and continuous integration peculiarities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qbnbmQuZtqV1QYth6ik2dQ.png" /></figure><h3>Understanding the Context — What Is MANTA</h3><p>MANTA is a software product company developing a product for data lineage analysis used by dozens of big customers from all over the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ey6Oo7pNtEX3g-8rW7_6wg.png" /><figcaption>Examples of MANTA customers</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8qe1F_vGUoNm_l_o9jvYQ.png" /><figcaption>Examples of MANTA partners</figcaption></figure><p>The truth is, we have grown out of our child shoes quite fast. If you visited our office 4 years ago, you would see only 4 enthusiasts. Nowadays, 40 developers make the magic possible and we expect to grow to 70 or 80 within two years. We maintain eight million lines of code, mainly written in Java, often nontrivial code parsing and analyzing SQL code or integrating with 3rd party systems through various APIs. Speaking about the volume of work, we usually develop 1000 features and generate an effort of 6.500 man-days per year.</p><p>I somehow feel we do not fit in the box labeled as “Startup” anymore, although it would sound attractive, right? I know, the startup could be also seen as a kind of mindset, not limited by the company size. No doubt. My point here is that the growth sooner or later brings the need for a certain level of organization and actionable processes we are going to talk about. These facts have particular implications for the software development process we use today. We will especially focus on the core development part of our SW development process, deliberately omitting other important parts such as analysis, design, and testing.</p><h3>Code Repository and Modularity</h3><h4>Mono-repo</h4><p>Yes, we have a monolithic repository. Sounds almost reactionary when I say it loud. Indeed, mono-repo seems to lose its popularity in our modern times of microservice architecture and code structured in many independent repositories giving so much flexibility.</p><p>When migrating from Subversion to Git, we had a good chance to evaluate the current source code versioning model and change to another model that would suit our needs better. The migration itself is another topic, so long story short. What was our rationale behind the mono-repo? We made a requirements analysis similar to what you would usually do as part of a standard development iteration. Our customers were the developers, our continuous system, our release process, and whoever might use the Git repository now or soon. We elaborated on all the relevant usage scenarios and simply said, “For us, the fewer repos, the fewer complications with releasing, daily coding and building, inter-module dependencies, etc.” Impaired scaling caused by a growing codebase was assessed as low risk. So, we stayed with one main mono-repo. The decision could be very different in another organization or team.</p><h4>Modularity</h4><p>After this decision, we were immediately facing the question: “How are you going to organize the code and build for dozens of programmers using a single codebase to avoid the mess?” Complicated things are easier to understand and handle when they are broken into smaller pieces. We borrowed the idea of modularity from the SW architecture and applied it to the modularization on the level of the codebase to enable parallel, independent, efficient work. Secondly, the software system architecture is often influenced by Conway’s Law: “Organizations design systems that mirror their communication structure“. This means that the organization chart is reflected in the SW architecture. And it could go even further by saying “The architecture gets reflected in the code-base structure.” Of course, there were many more factors and triggers that made us gradually decompose the repository into approximately 1000 modules. These modules are grouped into logical components such as Oracle scanner, export to Informatica Data Catalog, or one of our deliverables such as the web interface for the visualization of the data lineage or the administration console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tmjpwhYcrT0_tK7Ga9qAw.png" /><figcaption>When the Maven dependency tree is not helpful anymore. This is just a fraction.</figcaption></figure><p>From the project perspective, dividing the big repository into smaller cohesive components allows us to organize work effectively. Each team or developer is responsible for one or more components. If this division was missing, proper team scaling could be problematic. However, adding a new developer to the team often means assigning tasks connected to some existing modules or creating new, separate modules. This way we were witnessing the team growing from four to forty developers in four years and there will probably be seventy plus developers very soon.</p><p>From the design perspective, cohesive also means an encapsulated, well-defined functionality that can be grasped and used on its own. It can even be reused in multiple places, such as utility artifacts or platform code. Although the modules and components are cohesive, it doesn’t mean they are always independent. They form a complex system of direct and indirect compile-time dependencies which are checked and linked during the build phase.</p><p>Technically, the whole dependency management is managed by Maven. The high modularity of compilation units frees us from building and running the whole project consisting of one thousand artifacts to debug only one small increment of code. Each modification is located and can be addressed independently. Usually, the developer runs his code (one module) against the dedicated suite of unit or integration tests. If practiced consciously this workflow encourages test-driven approach and improves code quality and code coverage. Besides that, running only the related part of the code-base saves an incredible amount of time for the build and deployment to the application server in case of server-side code. If the integration testing and end-to-end testing are delayed for too long, the bugs will stack one on the other and the cause of the bugs can become hard to track.</p><h4>Modularity challenges</h4><p>Nothing in the world is perfect. We have been facing several issues regarding the modularity which I would like to sketch here because it could save you a few wrinkles.</p><p>Each developer should stay in sync with other developers to work with up-to-date snapshots of their modules. These snapshots must compile, function well, and should pass the required quality standards before they are made available to other developers. We will describe the implementation side of this in the chapter dedicated to continuous integration.</p><p>There is another challenge connected to codebase fragmentation and complicated dependency management. By working on individual modules separately, you inevitably happen to break APIs between the modules or you can corrupt functional contracts between your service and its client residing in a different module. This happens simply because you run and test only your own code without considering what is behind the wall and who else consumes your API or relies on your particular way of behaving. Luckily, you and your consumer should find out by the next day thanks to the continuous integration which we will describe further.</p><p>Apart from the functional dependencies, the compile-time dependencies must be tackled as well. The main theme here is “What depends on what.” Direct dependencies are easy to track and find out. However, transitive dependencies are not so obvious and you as a developer need to make sure the right version of the dependencies propagate into the final assembly (the end product). The Maven dependency tree command is helpful but can give very chaotic results for big projects with many dependencies. Also, cyclic dependencies should not occur but they do. They will ruin your build and are tricky to resolve.</p><p>Handling all those versions of all your artifacts and versions of their dependencies needs a systematic approach. Otherwise, you will drown in the sea of dependencies. Every time you branch off to start working on a new version of the application, you need to upgrade the versions. Maven will help you with its <em>mvn versions:set</em></p><p>But this can get even more complicated if you do minor (partial) releases as we do. We regularly release a patch containing only a small subset of modules. In such a case, we need to reliably identify only the modified modules and release only them.</p><p>Another moment to be aware of the versions is during release when you need to turn the snapshot versions into release versions. The Maven release plugin can handle this.</p><p>Remember we maintain 1000 modules in several teams. Such an amount of build configuration can get messy and inconsistent quite quickly. To avoid that, we built our own automation to control the build configuration. This helps us eliminate misconfiguration or usage of one library in two different versions. On top of this automated process, there is also a dedicated architect watching for duplicate or similar external libraries on the classpath.</p><p>In the next chapters, we will show that dividing the project into many smaller modules can make the CI more complicated, especially if your code resides in a mono-repo and you want to build several branches in parallel.</p><h3>Branching Model</h3><p>As a branching model, we started using Git-flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*eaHo-_rCJO0LfhFBeD-tXw.jpeg" /><figcaption>Git-flow model inspired by <a href="https://nvie.com/posts/a-successful-git-branching-model">https://nvie.com/posts/a-successful-git-branching-model</a></figcaption></figure><p>Very soon, it became obvious we needed to adjust the flow to cover all relevant scenarios at MANTA.</p><ol><li>There is only one permanent integration branch called develop.</li><li>Considering we support up to six major production versions in parallel, we completely dropped the master branch as a holder for the tags. We tag release revision on the dedicated release branches.</li><li>We use several parallel release branches — each hosts the code of a particular production version. The major release branch branches off from develop branch while the minor release branch starts from the last major release branch (minor contains only patches).</li><li>A hotfix, similar to the minor release branch, branches off from the major release branch. After the hotfix branch was tagged, the branch was merged to the major release branch.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*7yU5vX8Jw6s6ikyKJvxxyg.png" /></figure><p>The rest of the principles are similar to the standard Git-flow such as using short-lived feature branches to merge the little increments to the integration branch (develop or release branch). There are also a few additional rules we stick to.</p><ol><li>Feature branches should be merged to the integration branch using a switch — no-ff. This always creates a merge commit that is more explicit than just a silent fast-forward merge.</li><li>Feature branches should be kept in sync with the integration branch using rebase (instead of merge). This keeps the history clean without unnecessary crossroads.</li><li>Squashing of groups of related commits is recommended.</li><li>The only way to merge there is to use a pull request and pass a code review.</li><li>We set the integration branches as protected. Nobody can force-push and over-write the history with their local changes.</li></ol><p>The branching model must go hand in hand with the proper configuration management we talked about earlier. We must be able to support different production versions in parallel. Thus, in every type of branch (develop, feature, release, hotfix) you must correctly set the versions of the projects that are to be released in the target version. If you are not strict with the configuration management, you will likely deploy your artifacts into the central repository with the wrong version and others will use them unknowingly.</p><h3>Continuous Integration</h3><p>I have described code modularization as one of the ways of streamlining the development pipeline. You as a developer just change your source file, rebuild the increment and verify the test results, right? Yes, but…. It means the whole application is divided into pieces. If we leave it like that for too long those pieces will be very difficult to glue back together and make them work together. That is why we strongly focus on continuous integration and regularly integrate the modules to ensure consistency and correctness.</p><p>How does it work? Every merge to develop branch triggers Jenkins build which pulls the code, compiles and tests the source code, and checks the source code quality. This static code analysis phase is implemented via SonarQube quality gates integration. A quality gate in SonarQube is a set of selected quality metrics guarding your code quality. We usually relate to the level of blocker or critical issues, bugs, duplicated lines, code coverage, and vulnerabilities. If any of these thresholds are not met, the build fails. If all stages pass, the artifact is deployed to Nexus central repository. Consequently, a cascade of related downstream jobs is triggered and the whole build pipeline is executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-XjnVhxKFKPe70oDSxGqg.png" /><figcaption>Jenkins build jobs</figcaption></figure><p>This is a more or less standard process. But what are the specifics and challenges of the multimodule mono-repo used in a rapidly growing team?</p><p>We build all types of branches (pull requests, develop, release, hotfix) to spot the problems as early as possible. Building pull requests before merging helps us keep the integration branches stable.</p><h4>Maven jobs in Jenkins</h4><p>The CI of integration branches is implemented by Maven jobs in Jenkins. Each job contains several Maven projects bundled together. Each job has Upstream and Downstream projects, which allows Jenkins to run the jobs in the right order. They are automatically computed from dependencies in POMs of the individual projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*_Oo9mPuDrZbTM-0eTuAoPQ.png" /><figcaption>Jenkins upstream and downstream projects</figcaption></figure><p>Concerning multi-module mono-repo, it is practical to bind Jenkins jobs with individual folders of the mono-repo that host the individual modules. The solution is called sparse checkout. The workspace folder is restricted to only one sub-folder.</p><h4>Multibranch Pipeline</h4><p>Some types of branches such as feature branches are created on the fly and are automatically tracked in Jenkins. We use the “Multibranch Pipeline” and the Git plugin to notify Jenkins about recent changes. The pipeline is defined in the “Jenkinsfile” describing the build behavior for different types of branches and separate stages of the build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3ixPnZ-Y3UhNODrij9Ymw.png" /><figcaption>Multipipeline Job — Hotfix branches and feature branches</figcaption></figure><p>Different branches may point to different versions of the developed application. E.g., develop refers to <em>MANTA 34.0.0-SNAPSHOT</em> while in the release branch we test and fix <em>MANTA 33.0.0-SNAPSHOT</em>. Both branches should deploy their artifacts to the Nexus repository to provide SNAPSHOT JARs. Thanks to the different versions these artifacts won’t interfere in Nexus. However, work in the feature branches is in progress and could be unstable. That is why the feature branches’ build should not be deployed to the artifacts repository because it would overwrite the stable version deployed from the integration branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*3sP9qkLb3oHETVeYqW4jQg.png" /></figure><h4>Checking Vulnerabilities in External Libraries</h4><p>Now about security. Generally, the majority of the source code in large software systems was not written by the authors of the system, but it comes from the external libraries linked to the final executable. And often the developed system must comply with some security guidelines or you simply want to focus on the security concerns of your application. Then, mitigating the vulnerabilities in the external libraries makes sense. We, at MANTA, have automated the whole process of checking the external libraries against the database of known security vulnerabilities from the wide rank of OWASP. We use a Maven plugin called “OWASP Dependency-Check” scanning Java JARs and JavaScript libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n858XWztVAYqcyfh-olEAQ.png" /><figcaption>Vulnerabilities in Spring REST (<a href="https://cve.mitre.org/index.html">https://cve.mitre.org/index.html</a>)</figcaption></figure><h4>CI Catches</h4><p>Again, there are some common catches connected with the continuous integration and multi-module repository. Each of them caused us a headache, so make sure you’ve thought it out well. Here are our recommendations:</p><ol><li>Creating a new release branch means creating all the Jenkins jobs as they exist in the develop branch. We have 160 of them. Doing this manually would be quite a frustrating experience. We have automated this step via Jenkins CLI and is a matter of minutes.</li><li>One PR can change several modules and you need to build all the affected modules. Detecting only the changed modules within the pull request can be complicated.</li><li>Map the logical dependencies between your modules. Upstream and downstream projects are not everything.</li><li>Make sure the merged PR doesn’t trigger all other build jobs or build of all PRs.</li><li>Define and enforce the naming conventions for the different types of branches. CI will use the name patterns to detect new branches and to decide how to build each of them.</li><li>Build artifacts are cached in the local Maven repo. Use force update (switch <em>-U</em>) to get the fresh snapshot from the central repository.</li><li>Be ready for CI performance scaling. CI can be flooded when everybody starts pushing their features.</li></ol><h3>Summary</h3><p>I wish you the smoothest possible development experience. Let me share our top 10 tips.</p><ol><li>For smooth scaling of your team consider proper source code versioning model, branching model, and continuous integration strategy</li><li>Think carefully about project structure — project perspective, design perspective, implementation perspective</li><li>The branching model should support your development and release cycle</li><li>Use CI</li><li>Know the build dependencies and triggers</li><li>Define the build process for different types of branches</li><li>Define release process</li><li>Automate release process</li><li>Watch for build misconfiguration and build path</li><li>Deploy artifacts regarding the related application version</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dc6bd224942" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manta-engineering-blog/sw-development-process-at-manta-5dc6bd224942">SW Development Process At MANTA</a> was originally published in <a href="https://medium.com/manta-engineering-blog">MANTA Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>