<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[netcracker - Medium]]></title>
        <description><![CDATA[We are a forward-looking software company, offering mission-critical solutions to service providers around the globe. - Medium]]></description>
        <link>https://medium.com/netcracker?source=rss----ff491b436b53---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>netcracker - Medium</title>
            <link>https://medium.com/netcracker?source=rss----ff491b436b53---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:12:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/netcracker" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Join meetup - Implementing the micro front-end pattern: 2 years in 60 minutes]]></title>
            <link>https://medium.com/netcracker/join-meetup-implementing-the-micro-front-end-pattern-2-years-in-60-minutes-3d2759712c2d?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d2759712c2d</guid>
            <category><![CDATA[front-end-developer]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[micro-frontends]]></category>
            <category><![CDATA[meetup]]></category>
            <category><![CDATA[front-end-frameworks]]></category>
            <dc:creator><![CDATA[netcracker_team]]></dc:creator>
            <pubDate>Tue, 04 Oct 2022 13:16:34 GMT</pubDate>
            <atom:updated>2022-10-11T11:31:54.821Z</atom:updated>
            <content:encoded><![CDATA[<h3>Join meetup | Implementing the micro front-end pattern: 2 years in 60 minutes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7nGT54icFv0VO0bsmbk8fA.png" /></figure><p>There is a second front-end meetup hosted by Netcracker Tech Community🔥😊 Two years we were actively implementing micro frontends to a few telecommunications projects, this experience was accelerated into a framework that allows:</p><ul><li>different teams to create their own UIs independently</li><li>use both Angular and React</li><li>multiple teams to ship features independently</li></ul><p>Join the meetup to learn more and level up your expertise in micro frontends!</p><p>When: October 27th, 2022; 17.00–18.00 IST. Location: online</p><p>📝 <strong>Meetup agenda:</strong></p><ul><li>Quick framework overview</li><li>Live coding session</li><li>Q&amp;A</li></ul><p><a href="https://www.meetup.com/netcracker-technology-meetups/events/288897015/">Register</a> to the meetup and <a href="https://www.meetup.com/netcracker-technology-meetups/">stay tuned for more events with Netcracker tech community</a> ! 🤩</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d2759712c2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/join-meetup-implementing-the-micro-front-end-pattern-2-years-in-60-minutes-3d2759712c2d">Join meetup - Implementing the micro front-end pattern: 2 years in 60 minutes</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubemarine overview: open source tool for Kubernetes cluster deployment and maintenance]]></title>
            <link>https://medium.com/netcracker/kubemarine-overview-open-source-tool-for-kubernetes-cluster-deployment-and-maintenance-8c260493ad9b?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/8c260493ad9b</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[devops-tool]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Julia Murashova]]></dc:creator>
            <pubDate>Thu, 29 Sep 2022 14:47:08 GMT</pubDate>
            <atom:updated>2022-09-29T14:46:50.098Z</atom:updated>
            <content:encoded><![CDATA[<p>This article provides an overview and of Kubemarine — open source tool for Kubernetes designed by Netcracker Cloud core team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*HcI4aHu1dICnDsqv0cZ_Mg.png" /></figure><p>In Netcracker, we build and deliver enterprise-grade cloud ecosystems based on Kubernetes to our customers. Such solutions consist of both enterprise Kubernetes, which we prepare and maintain using the appropriate tools and many managed infrastructure cloud services, such as PostgreSQL, Cloud Streaming, Cloud Queues, etc. Kubemarine is one of these tools we created to solve engineering challenges, and now we are making it Opensource.</p><h3>What is Kubemarine?</h3><p><a href="https://github.com/Netcracker/KubeMarine">Kubemarine</a> is an open source, lightweight and powerful management tool built for end-to-end Kubernetes cluster deployment and maintenance.</p><p>It’s applicable for many purposes like <strong>simple and quick onboarding Kubernetes</strong> on local and production environments in different HA schemes depending on your aims, budget and capabilities. Together with simplicity Kubemarine can be very flexible and customizable tool <strong>covering specific configurability cases on both deployment and maintenance stages</strong>. This library provides powerful CLI commands, as well as can be customized via Python extension API.</p><p>The tool should be helpful for Cloud and DevOps engineers.</p><h3>Highlights and what can be done with Kubemarine</h3><ul><li>Easy to use</li><li>Many procedures supported: install, add_node, remove_node, upgrade, backup, restore, check_iaas, check_paas, manage_psp.</li><li>Single cluster inventory for all operations, highly customizable</li><li>Default values of all parameters in configurations with a minimum of required parameters</li><li>Control planes balancing with external balancers and VRRP</li><li>Ability to resume or skip specific task without re-running entire pipeline</li><li>Pre-built plugins out of the box and custom plugins support</li><li>Support for executing in closed environments with private registries</li><li>Extended logging, configs dumping</li><li>Build supported as a package, container and binary</li><li>Package extension with open extension API</li><li>Support different deployment schemes (all-in-one, mini-HA, HA, etc.)</li></ul><h3>Community and contributors</h3><ul><li>Join our <a href="https://communityinviter.com/apps/netcracker-tech/community">Slack Community</a></li><li>Follow us on <a href="https://github.com/Netcracker/KubeMarine">github</a>, use Kubemarine and make it better</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c260493ad9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/kubemarine-overview-open-source-tool-for-kubernetes-cluster-deployment-and-maintenance-8c260493ad9b">Kubemarine overview: open source tool for Kubernetes cluster deployment and maintenance</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystify AI]]></title>
            <link>https://medium.com/netcracker/demystify-ai-614c9f984be2?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/614c9f984be2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Rishabh Kumar]]></dc:creator>
            <pubDate>Mon, 22 Aug 2022 13:44:16 GMT</pubDate>
            <atom:updated>2022-08-22T13:44:16.330Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we begin, let’s imagine all technologies that we have today without AI. Every day we get hundreds of mail or even more and chances are that some of them will turn out to be spam. Google uses large-scale classification to filter those spam, and thus we can just sit back and do our work.</p><p>Not only that, we all are very sensitive towards the environment we are living in, we always want to know about the weather conditions. Guess how anchors on tv tell us that!? The answer is Machine Learning.</p><p>To make it more relatable, look at the e-commerce giants such as Amazon, Walmart, etc, the use of Prescriptive and descriptive Analytics has resulted in exponential growth.</p><p>Today we will be discussing how this AI appeared, the fundamentals to get started with AI and we will conclude with the impact and scope of AI.</p><h3>What is Artificial Intelligence?</h3><p>In simple words, it is the capability of machines to mimic human intelligence through various learning processes. The fundamental question is “How do we make the machine learn?” The answer is “Algorithms”. The illustration below depicts the functional part of Artificial intelligence and their relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*ZPyYETOwSPAOAYRRJvUW-Q.png" /><figcaption>A Broad View</figcaption></figure><p><strong>What is Machine Learning?</strong></p><p>Machine Learning is a subset of Artificial Intelligence that’s responsible for imparting learning capabilities to machines through Algorithms. Based on what kind of intelligence we want the machine to have, AI Engineers devise algorithms taking feature engineering into account.</p><p>Let’s say you brought a puppy which is to say some weeks old and now you want to train your puppy. What do you do? You use gestures and at the end of every correct response, you reward your puppy.</p><p>You supervised the puppy’s brain to perform certain actions linked to a specific command. Not only that, you imparted a permanent behavioral change. In the context of a computer program, when a program starts to improve its performance P in relation to some task T from the past experience E, It is said to be learning.</p><p>The way of equipping a computer program to learn can be different based on the problem we are dealing with and that’s why these algorithms are kept under broad classification.</p><p>Machine learning is classified into:</p><ol><li><strong>Supervised Learning: </strong>Let’s say our mom had cooked something and asked you to taste it but first she tells you that this is a dish “A”. The next time you eat the same dish or anything resembling it you quickly map and guess whether it’s the same or not. In technical language, this is called labels. In a nutshell, initially, we supervise the program and then we evaluate how much it could learn. A most common example would be predicting the price of houses in an area or weather forecasting.</li><li><strong>Unsupervised Learning: </strong>Here, we don’t tell the machine about the data features we are feeding and we let the machine understand the patterns on its own. Hence, the pattern can be a wider connotation such as dimension, colors, etc. depending upon the problem. For example, customer segmentation based on their choice and preference for better sales is done via such algorithms.</li><li><strong>Reinforcement Learning: </strong>This category of learning is built upon the dynamics of the environment, action, and reward system. In other words, suppose we have a robot and while walking, it steps over an obstacle and falls down. Now we want the robot to get up automatically and at the same time learn to avoid such obstacles in the future. This learning is rewarded with some positive tags and the robot starts differentiating between positive and negative actions.</li></ol><p>Supervised Learning is broadly classified into:</p><p><strong>A. Regression: </strong>Regression: Briefly, regression means prediction. That means it tries to learn the relationship using the labeled and based on the relation it predicts output on new data.</p><p>Consider Temperature as the input variable and humidity as our output. So here humidity is dependent upon temperature thus we can mathematically say</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*1GiNkztXzQcv5TUu4_kduw.png" /><figcaption>Weather Forecasting</figcaption></figure><p>H= m*t + error ,Here, h=humidity ,T=temperature ,M=slope, Error=difference between real and predicted value.</p><p><strong>B. Classification: </strong>Classification is about categorizing given data into distinct classes. In other words, it is a “Yes” or “No” classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*pMJ1-ML512sGu0yvZ7stow.png" /><figcaption>Spam Classification</figcaption></figure><p>The most common example is Spam Classification. Firstly, we train the algorithm to learn about spam features, and later algorithm uses this knowledge to filter spam. This requires a lot of training data to make the model more accurate.</p><p>Here, I would like to emphasize that classification is all about predicting categorical value that is “yes” or “no” while regression is about continuous value prediction.</p><p><strong>Unsupervised Learning</strong>: In this type of Learning we don’t provide an algorithm labeled data instead we ask it to learn the patterns and label them on their own. Based on the past learnings algorithm starts to provide a response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*9F3g4pfLle0pOMrcXg-0ug.png" /><figcaption>Unsupervised Learning</figcaption></figure><p>Today we have discussed the fundamentals of AI and how it evolved and also we touched upon how ML concepts are influencing the global market. I have tried my best to demonstrate the working logic with examples of my own.</p><p>I hope you enjoyed the article. What do you think about the future of Artificial Intelligence? Let me know in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=614c9f984be2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/demystify-ai-614c9f984be2">Demystify AI</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Pricing in Telecom]]></title>
            <link>https://medium.com/netcracker/dynamic-pricing-in-telecom-a8aaf8125e31?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8aaf8125e31</guid>
            <category><![CDATA[dynamic-pricing]]></category>
            <category><![CDATA[business-analysis]]></category>
            <dc:creator><![CDATA[netcracker_team]]></dc:creator>
            <pubDate>Mon, 14 Mar 2022 13:03:10 GMT</pubDate>
            <atom:updated>2022-03-14T13:39:22.902Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7BGlRSTjhF7gqjhi.png" /></figure><h3>1. Introduction</h3><p>When the price of an offering can be determined at the time of sale, based on the unique conditions prevailing at the time of sale for that particular customer, we can term it dynamic pricing. Some of the other terms used are surge pricing, demand pricing and time-based pricing.</p><p>Until now, this sort of pricing was predominantly used in Airline, Hotel and consumer shopping industries. But there is a good case to be made for Telecom sector too and we will explore this in more detail.</p><p><strong>Different ways of achieving Dynamic Pricing</strong></p><p>1. One of the simplest ways to achieve dynamic pricing is to configure all possible permutations of the price impacting factors and configuring a price for them in advance. But this approach severely restricts a marketing team’s ability to quickly launch new price based on new factors.</p><p>2. A step forward from configuring all permutations is to have rules and programmatic logic to determine the price using the prevailing factors as inputs. Special attention has to be paid to the system performance in such an architecture as executing the programs at run time can get expensive.</p><p>3. <strong>The best possible method</strong> however would be to analyze the past behavior of the customer and performance of the offering to arrive at the price that is most optimal. Getting price right every time is everyone’s concern. But with proper planning and a tool that supports flexible solution, it is definitely achievable. We expect this to be the method that will be adopted by majority of the service providers in future.</p><h3>2. Need for dynamic offers and pricing</h3><p><em>“Why use dynamic pricing when obviously there is an element of customer dissatisfaction?” </em>— This is a question that might cross every marketeer’s mind. The reasons are multitude and it is important to remember that dynamic pricing is mainly driven by the need for dynamic offerings and bundles.</p><p><strong>Hyper competitive markets</strong></p><p>In a hyper competitive market, customer retention means catering to their personal needs. Creating offers tailored to the needs of each customer and pricing it dynamically has become essential to avoid customer churn.</p><p><strong>Fast changing product portfolios</strong></p><p>Gone are the days when voice used to be the primary service and data rules now. This could change with time as technology advances and the software should be able to adapt to the ever changing world.</p><p><strong>Newer technologies and services enabling a multitude of new services to be launched</strong></p><p>Who would have imagined a world ruled by streaming services so much that the traditional ways of viewing content would be sidelined and forgotten. Advent of 5G will bring in more dynamism and services providers should be prepared for the challenges and opportunities.</p><p><strong>Customer intimacy and personalization</strong></p><p>Offering the same set of products to a wide customer base with wide array of requirements is like offering the same treatment to all kinds of medical problems. One size doesn’t fit all, and bespoke is the need of the hour.</p><p><strong>New marketing strategies and offer bundles</strong></p><p>While static offer bundles also offer some tools to enhance sales, ability to create dynamic bundles of atomic products offers the best flexibility to marketing to stay ahead of the competition.</p><h3>3. Factors impacting dynamism</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*mNcsSbiVwcauSceXfgPeEA.png" /></figure><p><strong>Customer Information</strong></p><p>· Location — Customer needs and affordability varies by geography</p><p>· Loyalty — Retention is as important as gaining customers to grow customer base and attractive pricing offers are the key to retain loyal customers</p><p>· Category — Different categories of customers are willing to pay different amounts for the same product</p><p>· Demography — Demography study helps offer the right products to customers</p><p>· Past Purchases — Having a robust database of customer behavior can yield good insights to develop future portfolio of offerings</p><p><strong>Business factors</strong></p><p>· Demand — Service providers can price higher when demand is high and vice versa</p><p>· Competition — It is imperative to be able to quickly change pricing in line with competition to retain customers</p><p>· Promotional period — Low seasons or festive moods can both be used as reasons to offer promotional prices</p><p>· Channel — Promotions can be run to drive sales through Online or App</p><p>· Revenue target — Prices can be lowered to drive volumes or raised to meet revenue target</p><p><strong>Offering Parameters</strong></p><p>· Volume — High volumes can be driven by offering great prices for bulk purchases</p><p>· Contract period — Locking customers for a longer period can yield profitability even with lower recurring charges</p><p>· Value — Plans with high cost can also have higher discounts while still being profitable</p><h3>4. Use cases</h3><p>Here are some examples of real life situations when dynamic prices can be used and how it can a source of customer delight as well as profitable to the service provider when used with the right strategy. Such frequent changes in pricing is only possible with a flexible and easy to use pricing engine.</p><p><strong>Consumer</strong></p><p>A consumer who wants a great discount waits for a Black Friday deal that was advertised to start his subscription. He already has a subscription with a rival competitor and doesn’t mind waiting. When the day comes, he bags a great deal and ports his number to the service provider.</p><p>A few days later, the same customer wants to buy an additional subscription with same allowances for his daughter who is turning 14. He doesn’t mind paying a little extra for the additional subscription since he wants it to a birthday gift.</p><p>Few days later he receives an offer to buy the latest device at a deep discounted price based on his loyalty and multiple subscriptions. He is delighted and appreciates that the service provider was able to offer him a personalized offer.</p><p><strong>Service Provider</strong></p><p>A Service Provider wants to gain more customer base in the market and hence launches offers with aggressive pricing in the market. Due to the great competitive price, it manages to gain a huge volume of consumers.</p><p>After a few days when the target consumer count is reached, the price of the same product is raised in some markets which are identified as price agnostic, with a higher QoS. This keeps the sales intact while obtaining a higher revenue per user and helps them meet the revenue target.</p><p>To retain recently acquired and long term customers, the service provider designs bundled offerings with attractively priced accessories aimed at existing customers. The rental of the base plan is raised slightly so that the promotional discount is recovered in the long run.</p><h3>5. Pros and Cons of dynamism</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*0Rj6ivXtXKlabruBuRL9IA.png" /></figure><p>Dynamic pricing is a strong tool in the business team’s arsenal to drive up sales, while still maintaining the revenue target. From a customer point of view, they can avail great discounts if purchasing offerings at the right time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/1*pEclx2cwSk8KPJeMpATgqw.png" /></figure><p>Customers can be antagonized if the right strategy is not followed and they notice a huge discrepancy in pricing compared to other customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/1*48hiZCj00VcrWEjaEv98kQ.png" /></figure><p>Some form of dynamism already exists in telecom services pricing in a rudimentary form. The best results can be achieved by forming the right strategy and having the right tools to achieve the goals identified.</p><h3>6. Proposed architecture for dynamic pricing tool</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*I1Lnet_pZ7SM7_vsB0oiJQ.png" /></figure><p>Proposed solution for the dynamic pricing system works on a two-step architecture.</p><p>First, dynamic offerings and bundles are formed utilizing direct inputs from customer as well as personas formed by data analytic inputs.</p><p>This can then be priced not just on the offering/bundle composition, but also based on myriad of other factors like demand, retention, location, promotion etc.</p><p>The architecture intends to leverage the robust product capabilities and years of industry experience to deliver a dynamic pricing solution that is flexible and scalable. It is expected to support a wide variety of factors which can determine the dynamism of the price.</p><p>The proposed system also intends to allow the business users to enter the inputs easily and help reducing the go-to-market time for any new product launch.</p><h3>7. Summary</h3><p>Even though dynamic pricing is not used as widely in telecom product pricing as it is in some other industries, it offers some exciting possibilities with both customers and service providers. Adopting it in the right manner can result in win-win situation for all involved.</p><p>In future, this could become the preferred way of pricing products and it is imperative to get a head start on implementing automation in applying dynamic prices.</p><p>by Apoorva Manmohan and Krishan Dhall</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8aaf8125e31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/dynamic-pricing-in-telecom-a8aaf8125e31">Dynamic Pricing in Telecom</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exec Probes: A Story about Experiment and Relevant Problems]]></title>
            <link>https://medium.com/netcracker/exec-probes-a-story-about-experiment-and-relevant-problems-12de616c0a76?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/12de616c0a76</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[netcracker_team]]></dc:creator>
            <pubDate>Wed, 02 Feb 2022 13:41:42 GMT</pubDate>
            <atom:updated>2022-02-02T13:41:42.468Z</atom:updated>
            <content:encoded><![CDATA[<p>Netcracker teams actively deploy their applications to Kubernetes and use <a href="https://kubernetes.io/ru/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#%D0%BE%D0%BF%D1%80%D0%B5%D0%B4%D0%B5%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-liveness-http-%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81%D0%B0">HTTP probes</a> for monitoring health of the services. However, we decided to experiment with the exec probes and… Ended up with the cluster down!</p><p>How so? It is the exec probes that are <em>first to be</em> <em>covered</em> in all Kubernetes manuals and guidelines, <a href="https://kubernetes.io/ru/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#%D0%BE%D0%BF%D1%80%D0%B5%D0%B4%D0%B5%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-liveness-%D0%BA%D0%BE%D0%BC%D0%B0%D0%BD%D0%B4%D1%8B">including the official ones</a>! So what is wrong?</p><p>Here you will find some relevant information that can save you from potential problems and rash decisions. At the very end, we will explain what is wrong with HTTP probes and why did we experiment with exec probes at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*9z1NmpsgjV5Moc6HtiJpdg.png" /></figure><h3><strong>Experiment parameters</strong></h3><p>Our cluster:</p><p>· Dev cluster used by the development and testing teams. No production load, so we can try something new there;</p><p>· About 120 vCPU, 1Tb RAM on worker nodes;</p><p>· About 1,800 running pods. Each uses HTTP probes for liveness/readiness/startup.</p><p>A set of probes is a required standard for all of our services. If there are no probes, the service will not be accepted even for internal testing.</p><p>Average cluster load before the experiment: CPU load is 45%, RAM load is about 50%. System processes (including systemd) consume about 10% of one CPU core.</p><p>We have updated about 490 services to use exec probes after which the CPU load reached 68% (+23%) and continued to increase as the probes were installed. The CPU consumption on systemd increased to almost 100% making the cluster virtually unresponsive.</p><p>So what went wrong? And why did we have to roll back everything?</p><h3>How does the Kubernetes cluster work?</h3><p>To understand the problem, I will briefly refer to the roles of processes in the cluster. <em>For sure, this is a large topic worthy of a series of articles. If you want to know more about it, please write in the comments.</em></p><p>We will be interested in the major players:</p><p>· <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/">kube-scheduler</a></p><p>· <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/">kubelet</a></p><p>· <a href="https://containerd.io/">containerd</a>/<a href="https://github.com/opencontainers/runc">runc</a></p><p>· <a href="https://systemd.io/">systemd</a></p><p>Let’s assume that you have already <a href="https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/">deleted Docker from your cluster</a> and switched to the CRI implementation from <a href="https://containerd.io/">containerd</a>/<a href="https://cri-o.io/">cri-o</a>. Therefore, we will not talk about Docker.</p><p>Simply put, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmYuDVYjCM70FIUQUo2t4A.png" /></figure><p>· <strong>kube-scheduler assigns a pod to the node.</strong> This is some sort of a cluster team lead. If you imagine it as your team lead, that’s a person who assigns pods as bugs and stories to team members.</p><p>· <strong>kubelet is an executor.</strong> It takes tasks assigned by kube-scheduler and executes them. However, it does not actually do them itself. It just passes high-level commands such as “execute the container” to the CRI layer, which is containerd/cri-o.</p><p>· <strong>containerd/runc</strong> deal with the physical load of the container layers, mounting them on top of each other and other system tasks. However, they also do not do these tasks themselves.</p><p>· <strong>systemd</strong> executes all <strong>system calls </strong>and low-level commands.</p><h3>How the HTTP probe works</h3><p>The HTTP probe works as simple as that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koK_oxgELxOsmyHnsZqorQ.png" /></figure><p>HTTP probe code is present in <a href="https://github.com/kubernetes/kubernetes/blob/092fbfbf53427de67cac1e9fa54aaa09a28371d7/pkg/probe/http/http.go#L87-L139">pkg/probe/http/http.go:91</a> and does the following:</p><p>1. Connects to the pod using its internal address;</p><p>2. Requests the specified page;</p><p>3. Defines the result based on the HTTP code of the response.</p><p>Shortly, it looks as follows:</p><pre><strong>func</strong> <strong>DoHTTPProbe</strong>(url *url.URL, headers http.Header, client GetHTTPInterface) (probe.Result, <strong>string</strong>, error) {</pre><pre>req, err := http.NewRequest(&quot;GET&quot;, url.String(), nil)</pre><pre>res, err := client.Do(req)</pre><pre>b, err := utilio.ReadAtMost(res.Body, maxRespBodyLength)</pre><pre>body := <strong>string</strong>(b)</pre><pre><strong>if</strong> res.StatusCode &gt;= http.StatusOK &amp;&amp; res.StatusCode &lt; http.StatusBadRequest {</pre><pre><strong>if</strong> res.StatusCode &gt;= http.StatusMultipleChoices { // Redirect</pre><pre><strong>return</strong> probe.Warning, body, nil</pre><pre>)</pre><pre><strong>return</strong> probe.Success, body, nil</pre><pre>)</pre><pre><strong>return</strong> probe.Failure, fmt.Sprintf(&quot;HTTP probe failed with statuscode: %d&quot;, res.StatusCode), nil</pre><pre>)</pre><p>As you can see, everything is really <em>very </em>simple and clear.</p><p>There is a little problem with the last line, but we will talk about it later. Now we are focused on the fact that this is a simple, clear, and lightweight request that causes no significant overheads and can be used in any volume.</p><h3>How the exec probe works</h3><p>This is a much more complicated, interesting, and multi-layered process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z1NnXGY5ApR3EQDQ9KCitA.png" /></figure><p>1. Kubelet gives a task to execute a command within the existing <em>pod</em> <em>sandbox </em>to the underlying CRI layer — <a href="https://github.com/kubernetes/kubernetes/blob/092fbfbf53427de67cac1e9fa54aaa09a28371d7/pkg/kubelet/prober/prober.go#L160">pkg/kubelet/prober/prober.go#L160</a>.</p><p>2. Containerd executes a series of actions with underlying layers, which include searching for the active sandbox, attaching to that sandbox, allocating the command and input-output buffers, actual execution, and subsequent cleanup. All this results in about 5–7 calls to the underlying layers and is described here: <a href="https://github.com/containerd/containerd/blob/36cc874494a56a253cd181a1a685b44b58a2e34a/pkg/cri/server/container_execsync.go#L196-L211">pkg/cri/server/container_execsync.go#L196-L211</a></p><p>3. Systemd executes all this jazz, passing the calls to the Linux kernel.</p><p>4. The kernel executes the commands and returns the results.</p><p>Thus, compared to a plain HTTP probe executing only one request, exec probe performs a set of actions at both systemd and kernel level.</p><h3>What it led to</h3><p>Simple math tells us:</p><p>· 490 probes that are executed every three seconds. On average, each includes 7 systemd calls =&gt; 1,143 systemd calls per second to the cluster;</p><p>· For the cluster with 11 nodes, it is about 100 systems calls per second on each node.</p><p>And here is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1QGXSv_hY0qss2myGlxlA.png" /></figure><p>systemd occupies about 70% of one CPU. And metrics on this screenshot were not even our worst! Unfortunately, I have misplaced the one showing beautiful 100% consumption of CPU.</p><p>Important: systemd is a single-threaded process. And when it consumes 100%, it does not just consume the cluster resources. It also makes the cluster unresponsive even for such simple routine tasks as deploying new containers.</p><p><em>As a result, almost entire workload has stopped. It became impossible to use the cluster as it allocated almost all systemd resources for servicing probes.</em></p><h3>Why did we even bother doing this</h3><p>Now you may ask why have we even tried using exec probes if we already had working HTTP.</p><p>The answer is in this part of the Kubernetes code — pkg/probe/http/http.go#L138.</p><pre>body := <strong>string</strong>(b)</pre><pre><strong>if</strong> res.StatusCode &gt;= http.StatusOK &amp;&amp; res.StatusCode &lt; http.StatusBadRequest {</pre><pre><strong>if</strong> res.StatusCode &gt;= http.StatusMultipleChoices { // Redirect</pre><pre><strong>return</strong> probe.Warning, body, nil</pre><pre>)</pre><pre><strong>return</strong> probe.Success, body, nil</pre><pre>)</pre><pre><strong>return</strong> probe.Failure, fmt.Sprintf(&quot;HTTP probe failed with statuscode: %d&quot;, res.StatusCode), nil</pre><p>As you can see, when the probe fails, it completely removes the read body and returns a non-informative message “HTTP probe failed with statuscode: 500”.</p><p>And for troubleshooting, we want to see clear information on why this particular probe failed. We can easily display this information in the probe body, but we need Kubernetes to read it from there.</p><p>Whereas, exec probes display complete information that is further provided to stderr. This looked more convenient for us.</p><p>Now it seems, that a simple pull request to Kubernetes would be a more appropriate solution.</p><h3>Conclusions</h3><p>Exec probes are a well-documented Kubernetes tool. Even the official “<a href="https://kubernetes.io/ru/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#%D0%BE%D0%BF%D1%80%D0%B5%D0%B4%D0%B5%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5-liveness-%D0%BA%D0%BE%D0%BC%D0%B0%D0%BD%D0%B4%D1%8B">Configure Liveness, Readiness and Startup Probes</a>” article uses exec probes as examples. Naively relying on them, we got very unexpected results, which I shared with you. And there is no information about the possible consequences, such as a non-working cluster. But now you know the situation. Forewarned is forearmed!</p><p>After rolling back to HTTP probes, the cluster returned to operating state with an average systemd load of about 10%.</p><p>Have you had a similar experience? Share in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12de616c0a76" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/exec-probes-a-story-about-experiment-and-relevant-problems-12de616c0a76">Exec Probes: A Story about Experiment and Relevant Problems</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Version Control of Configuration Files Using Kubernetes]]></title>
            <link>https://medium.com/netcracker/version-control-of-configuration-files-using-kubernetes-21673766203?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/21673766203</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[netcracker_team]]></dc:creator>
            <pubDate>Wed, 02 Feb 2022 13:41:13 GMT</pubDate>
            <atom:updated>2022-02-02T13:41:13.441Z</atom:updated>
            <content:encoded><![CDATA[<p>The Billing API SDK team in Netcracker develops tools for the Cloud Billing product. We deal with version control, API lifecycle, REST/gRPC server configuration and start, authentication and authorization, audit, tracing, and many other things in Go language.</p><p>Let’s talk about the configuration files!</p><p>If your applications have configuration files, following situation must be familiar to you: you develop an application, and then you create a configuration file and document it. After a while, you need to add some settings as the old ones do not meet all the requirements and, in general, it is better to change the structure.</p><p>What to do? If you do not change the configuration format, over time, it will turn into a bunch of things that “we need to deal with for historical reasons”. And if you change it… In this case, you always need to check if the configuration files are compatible with the product version you are installing for the customer. The operation team, customers, and many others will not really like this.</p><p>These problems can be solved by multi-version configurations. Borrowing them from the Kubernetes, we have developed and applied them. Now let’s discuss how it works.</p><h4>A few words about the hub-spoke model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IRBAUlmB-YHJTbbqInUKmw.png" /></figure><p>The hub-spoke model was initially an architectural pattern of the network topology, which referred not only to a computer network but also, for example, to a transport one. However, the pattern has been also put to good use in objects versioning. Thus, Kubernetes uses it for versioning of its APIs and resources.</p><p>The hub-spoke model assumes that there is a “hub” version of some object, which cannot be directly used by users, but it is related to other spoke versions. The hub version is the current version used in our code. When the configuration structure is changed, a new “spoke” version is separated from the hub. This version is identical to the hub version. To ensure the compatibility of the versions, we must be able to convert any Spoke version to the Hub version and vice versa.</p><p>Thus, there is no more worrying about configuration version that users of our service utilize. All you need to do is:</p><p>· define config version;</p><p>· convert config from “spoke” to “hub” version;</p><p>· use “hub” version of config in our code.</p><p>Thus, our Cloud Billing components can seamlessly upgrade config versions inside without updating all customer configuration files to new versions.</p><p>A nice bonus: all fixes made in new version are automatically added for users of the old config versions.</p><h4>Adding a touch of versioning to config</h4><p>Let’s say that our application has a configuration file describing roles and resources available for each of them:</p><pre>roles:</pre><pre>- name: &quot;account_ro&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;customer&quot;, &quot;account&quot;, &quot;address&quot; ]</pre><pre>- name: &quot;account_rw&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot;, &quot;Create&quot;, &quot;Update&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;customer&quot;, &quot;account&quot;, &quot;address&quot; ]</pre><pre>- name: &quot;dev&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;*&quot; ]</pre><p>To generate conversion functions between versions of this config, we will use the Kubernetes <a href="https://github.com/kubernetes/apimachinery">apimachinery</a> library and conversion-gen, deepcopy-gen and defaulter-gen Go code generators from <a href="https://github.com/kubernetes/code-generator/tree/master/cmd">kubernetes/code-generator</a>.</p><p>In order for the Kubernetes tools to identify our config, convert it to the Kubernetes-like view:</p><pre>apiVersion: config.billing.netcracker.com/v1alpha1</pre><pre>kind: RolesConfig</pre><pre>spec:</pre><pre>roles:</pre><pre>- name: &quot;account_ro&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;customer&quot;, &quot;account&quot;, &quot;address&quot; ]</pre><pre>- name: &quot;account_rw&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot;, &quot;Create&quot;, &quot;Update&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;customer&quot;, &quot;account&quot;, &quot;address&quot; ]</pre><pre>- name: &quot;dev&quot;</pre><pre>rules:</pre><pre>- verbs: [ &quot;Get&quot;, &quot;List&quot; ]</pre><pre>apiGroups: [ &quot;billing.netcracker.com&quot; ]</pre><pre>resources: [ &quot;*&quot; ]</pre><p>Where</p><p>· <strong><em>apiVersion</em></strong> identifies the config version. It consists of the API group (`config.billing.netcracker.com`) and spoke version name (`v1alpha1`);</p><p>· <strong><em>kind</em></strong> is a config identifier;</p><p>· <strong><em>spec</em></strong> contains config content itself.</p><p>To work with this configuration in code, let’s add Go structures that will store config data. For this, we’ll create two packages for hub and spoke versions in our project. It looks like this:</p><p>├── hub</p><p>├── v1alpha1</p><p>Note: some files will be identical (except for the package name) in both hub and spoke packages. In this case, I will give example of only one file.</p><p>Add <em>types.go</em> file to the <em>hub</em> package. The file should contain:</p><pre><strong>package</strong> hub</pre><pre><strong>import</strong> meta &quot;k8s.io/apimachinery/pkg/apis/meta/v1&quot;</pre><pre>// RolesConfig contains a list of roles for authorization</pre><pre><strong>type</strong> RolesConfig <strong>struct</strong> {</pre><pre>// TypeMeta describes an individual object in an API response or request</pre><pre>// with strings representing the type of the object and its API schema version.</pre><pre>// Structures that are versioned or persisted should inline TypeMeta.</pre><pre>meta.TypeMeta `json:&quot;,inline&quot;`</pre><pre>// ObjectMeta is metadata that all persisted resources must have,</pre><pre>meta.ObjectMeta `json:&quot;metadata,omitempty&quot;`</pre><pre>// RolesSpec is a list of existing RBAC Roles.</pre><pre>RolesSpec RolesSpec `json:&quot;spec,omitempty&quot;`</pre><pre>)</pre><pre><strong>type</strong> RolesSpec <strong>struct</strong> {</pre><pre>Roles []Role `json:&quot;roles,omitempty&quot;`</pre><pre>)</pre><pre>// Role contains rules that represent a set of permissions.</pre><pre><strong>type</strong> Role <strong>struct</strong> {</pre><pre>// Name is unique role name.</pre><pre>// +kubebuilder:validation:Required</pre><pre>Name <strong>string</strong> `json:&quot;name,omitempty&quot;`</pre><pre>// IsAnonymous identify that role is allowed for anonymous user - user.</pre><pre>// +kubebuilder:validation:Optional</pre><pre>IsAnonymous <strong>bool</strong> `json:&quot;anonymous&quot;`</pre><pre>// Rules is set of rules available for the role.</pre><pre>// +kubebuilder:validation:Required</pre><pre>Rules RuleSet `json:&quot;rules&quot;`</pre><pre>)</pre><pre>// RuleSet contains set of rules for a role</pre><pre><strong>type</strong> RuleSet []Rule</pre><pre>// Rule is the list of actions the subject is allowed to perform on resources.</pre><pre><strong>type</strong> Rule <strong>struct</strong> {</pre><pre>// +kubebuilder:validation:Required</pre><pre>Verbs []<strong>string</strong> `json:&quot;verbs,omitempty&quot;`</pre><pre>// +kubebuilder:validation:Required</pre><pre>Groups []<strong>string</strong> `json:&quot;apiGroups,omitempty&quot;`</pre><pre>// +kubebuilder:validation:Required</pre><pre>Kinds []<strong>string</strong> `json:&quot;resources,omitempty&quot;`</pre><pre>)</pre><p>Similarly, add <em>types.go</em> to the v1alpha1 package. Since we have only one configuration version so far, hub and v1alpha1 are identical.</p><p>Add <em>doc.go</em> with config metadata to the hub package:</p><pre>// +k8s:deepcopy-gen=package,register</pre><pre>// +groupName=config.billing.netcracker.com</pre><pre><strong>package</strong> hub</pre><pre><strong>const</strong> (</pre><pre>Version = &quot;hub&quot;</pre><pre>Group   = &quot;config.billing.netcracker.com&quot;</pre><pre>Kind    = &quot;RolesConfig&quot;</pre><pre>)</pre><p>The `<em>+k8s:deepcopy-gen`</em> comment specifies settings for the deepcopy-gen plugin.</p><p>· <strong><em>package</em></strong> — this plugin generates the DeepCopy() method for all types in <em>types.go</em></p><p>· <strong><em>register</em></strong> — registers generated methods in the scheme (more on that below).</p><p>Same file is required in the v1alpha1 package:</p><pre>// +k8s:deepcopy-gen=package,register</pre><pre>// +k8s:conversion-gen=nrm.netcracker.cloud//billing-api-sdk/pkg/sdk/security/authorization/hub</pre><pre>// +groupName=config.billing.netcracker.com</pre><pre><strong>package</strong> v1alpha1</pre><pre><strong>const</strong> (</pre><pre>Version = &quot;v1alpha1&quot;</pre><pre>Group   = &quot;config.billing.netcracker.com&quot;</pre><pre>Kind    = &quot;RolesConfig&quot;</pre><pre>)</pre><p>In<strong><em> `</em></strong><em>+k8s:conversion-gen</em><strong><em>`</em></strong>, specify path to the hub package.</p><p>Kubernetes uses concept of a scheme to generate conversion functions between different versions. Go structures of each version as well as the SetDefault and DeepCopy features are registered in it. Add <em>register.go</em> file to the hub and v1alpha1 to register newly added versions:</p><pre>// +k8s:deepcopy-gen=package</pre><pre>// +groupName=config.billing.netcracker.com</pre><pre><strong>package</strong> v1alpha1</pre><pre><strong>import</strong> (</pre><pre>meta &quot;k8s.io/apimachinery/pkg/apis/meta/v1&quot;</pre><pre>&quot;k8s.io/apimachinery/pkg/runtime&quot;</pre><pre>&quot;k8s.io/apimachinery/pkg/runtime/schema&quot;</pre><pre>&quot;sigs.k8s.io/controller-runtime/pkg/scheme&quot;</pre><pre>)</pre><pre><strong>var</strong> (</pre><pre>// SchemeGroupVersion is group version used to register these objects</pre><pre>SchemeGroupVersion = schema.GroupVersion{Group: Group, Version: Version}</pre><pre>// SchemeBuilder is used to add go types to the GroupVersionKind scheme</pre><pre>SchemeBuilder = &amp;scheme.Builder{GroupVersion: SchemeGroupVersion}</pre><pre>localSchemeBuilder = runtime.NewSchemeBuilder()</pre><pre>)</pre><pre><strong>func init</strong>() {</pre><pre>SchemeBuilder.SchemeBuilder.Register(<strong>func</strong>(s *runtime.Scheme) <strong>error</strong> {</pre><pre>s.AddKnownTypeWithName(SchemeBuilder.GroupVersion.WithKind(Kind), &amp;RolesConfig{})</pre><pre>meta.AddToGroupVersion(s, SchemeBuilder.GroupVersion)</pre><pre><strong>return</strong> RegisterConversions(s)</pre><pre>})</pre><pre>)</pre><p>After all previous steps, you must get following file structure:</p><pre>├── hub</pre><pre>│   ├── doc.<strong>go</strong></pre><pre>│   ├── register.<strong>go</strong></pre><pre>│   ├── types.<strong>go</strong></pre><pre>├── v1alpha1</pre><pre>│   ├── doc.<strong>go</strong></pre><pre>│   ├── register.<strong>go</strong></pre><pre>│   ├── types.<strong>go</strong></pre><p>Now, let’s use Kubernetes plugins and generate the conversion functions between the versions. First, install the following plugins:</p><p>· “k8s.io/code-generator/cmd/conversion-gen”</p><p>· “k8s.io/code-generator/cmd/deepcopy-gen”</p><p>· “k8s.io/code-generator/cmd/defaulter-gen”</p><p>You can install each plugin using the <em>go</em> <em>install</em> command. Then, call each plugin passing paths to the hub and v1alpha1 versions at the input:</p><pre>conversion-gen --input-dirs $(PATH_TO_SPOKE_CONFIG) --output-<strong>package</strong> $(ROOT_PKG) --output-file-base zz_generated.conversion --output-base $(CURDIR) --<strong>go</strong>-header-file header.<strong>go</strong>.txt -v 1</pre><pre>deepcopy-gen --input-dirs $(PATH_TO_SPOKE_CONFIGS),$(PATH_TO_HUB_CONFIG) --output-<strong>package</strong> $(ROOT_PKG) --output-file-base zz_generated.deepcopy --output-base $(CURDIR)       --<strong>go</strong>-header-file header.<strong>go</strong>.txt -v 1</pre><pre>defaulter-gen --input-dirs $(PATH_TO_SPOKE_CONFIGS) --output-<strong>package</strong> $(ROOT_PKG) --output-file-base zz_generated.<strong>default</strong> --output-base $(CURDIR) --<strong>go</strong>-header-file  header.<strong>go</strong>.txt -v 1</pre><p>Plugin work should result in following file structure:</p><pre>├── hub</pre><pre>│   ├── doc.<strong>go</strong></pre><pre>│   ├── register.<strong>go</strong></pre><pre>│   ├── types.<strong>go</strong></pre><pre>│   ├── zz_generated.deepcopy.<strong>go</strong></pre><pre>│   ├── zz_generated.<strong>default</strong>.<strong>go</strong></pre><pre>├── v1alpha1</pre><pre>│   ├── doc.<strong>go</strong></pre><pre>│   ├── register.<strong>go</strong></pre><pre>│   ├── types.<strong>go</strong></pre><pre>│   ├── zz_generated.conversion.<strong>go</strong></pre><pre>│   ├── zz_generated.deepcopy.<strong>go</strong></pre><pre>│   ├── zz_generated.<strong>default</strong>.<strong>go</strong></pre><p>Great! Now we can convert our configuration from v1alpha1 to the hub version.</p><h4><strong>Providing an API to load configurations from file</strong></h4><p>Let’s provide our users with the possibility to load configurations from files into the RolesConfig object from hub. Name the file <em>config.go</em> and place it next to hub and v1alpha1 packages. It is not required to do a separate loading function for each type. Since all configuration types are registered in the global scheme and implement runtime.Object interface, you can write only one loading function that will accept a config object and fill it from a file for all config types.</p><pre>// getConfigScheme returns a new instance of runtime.Schema with registered authorization config.</pre><pre><strong>func</strong> <strong>getConfigScheme</strong>() (*runtime.Scheme, error) {</pre><pre>scheme := runtime.NewScheme()</pre><pre>err := hub.SchemeBuilder.AddToScheme(scheme)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>err = v1alpha1.SchemeBuilder.AddToScheme(scheme)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre><strong>return</strong> scheme, nil</pre><pre>)</pre><pre>// LoadFromFile reads roles configuration from provided config file and converts it to hub representation.</pre><pre><strong>func</strong> <strong>LoadFromFile</strong>(filePath <strong>string</strong>) (*hub.RolesConfig, error) {</pre><pre>// get conversion schema</pre><pre>scheme, err := getConfigScheme()</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>// read file content</pre><pre>data, err := os.ReadFile(filePath)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, fmt.Errorf(&quot;unable to read roles configuration: %w&quot;, err)</pre><pre>)</pre><pre>// unmarshal it to meta.TypeMeta to get metadata</pre><pre>typeMeta := &amp;meta.TypeMeta{}</pre><pre>err = yaml.Unmarshal(data, typeMeta)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>// validate that configuration in file has the same kind</pre><pre><strong>if</strong> typeMeta.Kind != hub.Kind {</pre><pre><strong>return</strong> nil, fmt.Errorf(&quot;unable to read roles configuration: invalid config kind &#39;%s&#39;, only &#39;%s&#39; supported&quot;, typeMeta.Kind, hub.Kind)</pre><pre>)</pre><pre>// create new spoke with same version as in config</pre><pre>storedObject, err := scheme.New(typeMeta.GroupVersionKind())</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>// unmarshall config to spoke version</pre><pre>err = yaml.Unmarshal(data, storedObject)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>// set defaults for spoke. If you added defaults.</pre><pre>scheme.Default(storedObject)</pre><pre>// convert spoke to hub</pre><pre>hubObj := &amp;hub.RolesConfig{}</pre><pre>err = scheme.Convert(storedObject, hubObj, nil)</pre><pre><strong>if</strong> err != nil {</pre><pre><strong>return</strong> nil, err</pre><pre>)</pre><pre>// set default for hub, if required</pre><pre>scheme.Default(hubObj)</pre><pre><strong>return</strong> hubObj, nil</pre><pre>)</pre><p>We need getConfigScheme() method to get Kubernetes scheme, in which all configuration versions are registered. Having received this scheme, we can determine version from the file passed to LoadFromFile(). Then, create an instance of the RolesConfig object of necessary version and unmarshal the file content to this object. Using the scheme, convert this object to the object hub version, set default values and return as an output.</p><p>It is not required to create a new scheme to load one type of config. You can have a global scheme where you will register all application configs, or you can pass the scheme directly to the loading method.</p><h4>Adding new spoke version</h4><p>If you need to change the configuration structure, do it in the hub version. After making changes, you need to:</p><p>· Create a package for the new spoke version (for example, v1alpha2) and copy updated types.go from the hub to it;</p><p>· Add doc.go, register.go and default.go, similarly to v1alpha1;</p><p>Call the conversion-gen, deepcopy-gen, and defaulter-gen plugins for all three versions (hub, v1alpha1, v1alpha2).</p><h4>Major changes: what to do?</h4><p>What if some major changes were made to the configuration and the conversion-gen plugin cannot automatically convert that configuration between the versions? You may actually face a situation like this… In this case, you need to implement the conversion function and place it in package of the spoke version, with which you are having problems.</p><p>Name the file <em>zz_generated.manual.go</em> to distinguish it from the rest of the generated files. The code will be as follows:</p><pre><strong>package</strong> v1alpha1</pre><pre><strong>import</strong> (</pre><pre>hub &quot;billing-api-sdk.git/pkg/sdk/db/hub&quot;</pre><pre>conversion &quot;k8s.io/apimachinery/pkg/conversion&quot;</pre><pre>)</pre><pre><strong>func</strong> <strong>Convert_hub_RolesSpec_To_v1alpha1_RolesSpec</strong>(in *hub.RolesSpec, out *RolesSpec, s conversion.Scope) <strong>error</strong> {</pre><pre>// some custom conversion logic here...</pre><pre>)</pre><h4><strong>Other possible improvements</strong></h4><p>To simplify our work with versioned configs, we can do the following:</p><p><strong><em>Adding ability to simultaneously load several configurations from a directory and merge them into one</em></strong></p><p>Configurations often have different sources of settings. Some of them come with the product, others are customized, some are always defaulted and others need to be changed for every environment. If you load several configuration fragments at once and merge them, configuration parts can be divided into default, environment-specific, and other types, depending on the selected criteria.</p><p>For this, we need to clearly define the criteria based on which we can merge the configuration parts. Kubernetes uses <a href="https://kubernetes.io/docs/reference/using-api/server-side-apply/#merge-strategy">this merge strategy</a>. It is important to have same merge rules for all configs, as people who fill the configuration often do not know all merge aspects. Therefore, we simply recommend merging object lists by the <em>Name</em> field and returning an error if there are conflicts.</p><p>Next, you can find all files with the yaml/yml extension in the directory, find all files of the appropriate kind among them, unmarshal them to their spoke versions, merge them, and then convert to hub version.</p><p><strong><em>Adding ability to load multiple configurations with conditions</em></strong></p><p>Kubernetes provides us with ability to use selectors for working with objects, which improves loading from a directory. Using a selector, you can add different labels to configuration files and load all configurations with provided label. Thus, you can store multiple configurations for different environments in one directory and quickly change the profiles.</p><p><strong><em>Adding configuration validation through CRD</em></strong></p><p>Since structure of versioned configurations corresponds to CustomResource from Kubernetes, we can generate CustomResourceDefinition and validate configuration content using it. See the detailed information about the validations using CRD <a href="https://book.kubebuilder.io/reference/markers/crd-validation.html">here</a>.</p><p><strong><em>Making a centralized service for configuration</em></strong></p><p>Since a configuration is in fact a public API, it is reasonable to make it centralized. By registering all configuration types and versions in one scheme, you can also generate documentation as well as find delta.</p><h4>Conclusions</h4><p>It is impossible to build an ideal system that requires no improvements. And configs are no exception.</p><p>Using the hub-spoke versioning model allows you to seamlessly make changes to configuration files, ensuring backward and forward compatibility. This promotes quick adaptation to environment requirements without confusing different versions and making the configuration body too complicated.</p><p>Backward and forward compatibility of configuration makes life easier!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21673766203" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/version-control-of-configuration-files-using-kubernetes-21673766203">Version Control of Configuration Files Using Kubernetes</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cybersecurity]]></title>
            <link>https://medium.com/netcracker/cybersecurity-c0ba0011fcbb?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/c0ba0011fcbb</guid>
            <category><![CDATA[cybersecurity-awareness]]></category>
            <dc:creator><![CDATA[Dhaval Naidu]]></dc:creator>
            <pubDate>Fri, 07 Jan 2022 08:51:00 GMT</pubDate>
            <atom:updated>2022-01-07T08:51:00.192Z</atom:updated>
            <content:encoded><![CDATA[<p>Cybersecurity helps secure our sensitive data on premise and on cloud, but the change in work culture puts the data at considerable risk. Vulnerabilities are spread across open source applications, servers, and unaudited OS, and the company’s data is at risk. To ensure efficient business continuity the protection of company data is necessary.</p><p>Applications and solutions providing security and compliance help to keep end-to-end data safe. They provide on premise and cloud hosted support for small, mid-size, and large organizations. Their technologies are globally accepted as they have benchmarks and standard remediation tools. This provides real-time infrastructure status and quick solutions. The technology’s focus is on security assessment, vulnerability assessment, and remediation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*oNKEpnkO2ILrgrcQE-i-WA.png" /></figure><p>Security As A Service helps to provide end to end infrastructure solutions by:</p><p>· Scanning and assessing the infrastructure for vulnerabilities, compliance issues, and missing patches in near time.</p><p>· Grouping and targeting the remediation strategy, identifying patch issues, and configuration issues</p><p>· Remediating the Operating System Patches, any 3rd Party Application patches and configuration fixes, and providing remote access for manual actions</p><p>· Reporting the real-time data for configurations, vulnerability and compliance issues, and generating compliance reports after successful remediation.</p><p>Security and Compliance solutions help in vulnerability assessment and remediation methods by scanning the infrastructure in real-time. The solution provides reports that you can disintegrate to understand the minutest of issues, accordingly a corrective action can be taken. The solutions are provided by applying patches, recent built in commands, and custom remediation policies.</p><p>Vulnerability assessment helps to identify, evaluate, and report the threat found in devices and software. The vulnerability assessment solution’s core functions are:</p><p>· Real-time monitoring and assessment</p><p>· Multi-tenant solution over cloud</p><p>· Repositories and patches</p><p>· Complete automated scanning</p><p>· Military grade inscription</p><p>· Global acceptance of scanners</p><p>· Single and secure connection</p><p>· Customized commands and remediation’s</p><p>· Immediate remedy</p><p>· Detailed report</p><p>· Reduced time and cost</p><p>All the managed devices need to be continuously assessed and kept away from vulnerability scripts being run by hackers. Security assessment does an intrinsic evaluation of any security loopholes, logs the vulnerabilities, creates severity levels for them and provides a remedy. They follow a global list of guidelines specific to industry type, local security regulations, region, and the business model. Some of the well-known entities are Health Information Trust Alliance (HIPA), National Institute of Standards and Technology (NIST), and Health Insurance Portability and Accountability Act 1996 (HIPAA).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0ba0011fcbb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/cybersecurity-c0ba0011fcbb">Cybersecurity</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does migrating from Oracle to PostgreSQL in the cloud impact storage?]]></title>
            <link>https://medium.com/netcracker/how-does-migrating-from-oracle-to-postgresql-in-the-cloud-impact-storage-422fb30f3d21?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/422fb30f3d21</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Duncanwalker]]></dc:creator>
            <pubDate>Mon, 20 Dec 2021 09:42:42 GMT</pubDate>
            <atom:updated>2021-12-20T09:42:41.906Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever considered moving a large scale software solution from Oracle to PostgreSQL and then deploying it into a public cloud? We did, and we ran some tests to see how it performed. If you are considering undertaking a similar exercise you might find this account of our experience enlightening. This article focuses specifically on our experience of configuring the cloud storage used by PostgreSQL to optimise performance.</p><p>OK so storage isn’t the most exciting topic but it is something you will need to consider to get the best out of a large PostgreSQL instance. Firstly let’s start with some high-level differences between how Oracle and PostgreSQL store data on disk.</p><p>1. Oracle stores lots of tables in the same tablespace and each tablespace can be split over several data files. Whereas PostgreSQL usually stores one table in one data file.</p><p>2. Oracle also has automatic Storage Management (ASM), which manages groups of disks to store data files and can do clever things like distribute the data files evenly over disks to avoid hotspots.</p><p>In short, Oracle tries to spread the records for a given table over several storage devices, while PostgreSQL concentrates the records for a table into a single data file, which under load can create hotspots if not managed.</p><p>Now let’s consider the impact of moving into the public cloud. We are no longer buying disks and storage arrays, rather buying commodity services from our cloud provider. So all we need to do is select the amount and speed of storage from a dropdown list, right? Well not quite. It turns out a single storage block is not able to provide the required throughput and latency required by our applications under test.</p><p>Regardless of the IOPS selected latency increases with higher traffic volume. Every storage system will have a point where it cannot keep up with demand, and latency will increase; using faster storage just moves the point at which this will happen. The real question becomes can we raise the bar high enough that PostgreSQL gets consistently good response times from the storage medium while processing the required workload?</p><p>Finally, before we start to look at test results let’s take a look at the points where PostgreSQL touches the storage and consider what impact poor storage performance might impact PostgreSQL performance in each case.</p><p>Like all RDBMS PostgreSQL employs caching of data in memory, this is further backed up by Linux using available free memory to buffer file system metadata and cache the contents of data files, but ultimately if the information we want to read or modify isn’t in any of these places then we need to read it from storage. The longer the read takes, the longer it takes PostgreSQL to return the data to the application. This can be improved by using more memory to increase the buffer hit ratio but ultimately unless the entire database is going to fit in memory we are going to have to read some things from storage at some point.</p><h3>Updates/Inserts/Deletes WAL log writes</h3><p>DML that changes the data stored in the database does so by manipulating the data in the shared buffer. However for durability, at the point the data is committed PostgreSQL writes the details of the changes to the WAL log on disk. This is a synchronous write and so forms part of the commit transaction time, if the storage holding the WAL has excessive write latency then the commit takes longer than it should.</p><h3>Writes impacting reads from storage</h3><p>In general writes to storage from PostgreSQL (apart from the WAL log) are asynchronous and are separated from the time it takes to perform a query. However poor write performance and high volumes of writes can saturate the storage IOPS capacity leading to slower read times which will impact PostgreSQL’s query times.</p><h3>Checkpoints</h3><p>We have similar issues around checkpoints (this is where PostgreSQL wants to mark a point in the WAL log where all transactions to this point have been written down to disk). This can require large amounts of data to be written to storage before the checkpoint can be written, poor storage latency can slow this process down.</p><p>The problems above can have multiple causes; bad design, poor queries, under-resourcing, poor configuration…, which I am not going to address in this article. Instead, I’m going to fast forward to the point where we are satisfied that the application and database are generally healthy and we want to improve the response time of getting data from storage to maximise PostgreSQL performance.</p><h3>Testing</h3><p>We executed a series of tests to determine the maximum throughput of one of our applications against a fixed hardware setup, our expectation was that we would keep adding more parallel instances of the application until we saturated the CPU and/or memory capacity of the database server and performance of the application dropped off. What we found was that we ended up IO-bound way before we reached CPU saturation.</p><p>The application in question was originally only built for Oracle but over the last few years has been converted to run on PostgreSQL. We understand the load profile this application will place on the database in the configuration used for this test. The application will read very heavily from one single table, which we will call Table A (750m rows in this test set). It will also write large amounts of data to two other tables, we will call them B and C to avoid getting bogged down in the specifics of the application. It’s enough to say that when tested the application reads from several tables including table A, does some work, then outputs the results of its work several other tables including tables B and C, generating a large number of IOPS.</p><p>The test was conducted using PostgreSQL 11 residing on a 16-vCPU host with 117GB of RAM on the Oracle cloud (VM.Standard2.8 two threads per core each thread represented as a vCPU). We generated the load from applications running on a Kubernetes cluster consisting of 4 nodes with the same specification as the DB server allowing more than enough resources to ensure this would not be the bottleneck.</p><p>Initial benchmarking</p><p>Pg_profile showed the read from table A as ~50% of the run time, the inserts into B and C as 5–7% each. Given the profile of the product, we don’t expect much of the data in Table A to be in the shared buffer at the time of the test as it&#39;s very large and written through the month and then read back once per month, creating a large amount of IO for testing disk configurations.</p><p>We performed a series of tests, each time modifying the layout of the storage configuration to optimise performance. Initially with the thought that since this is a cloud environment, we didn’t need to specify a storage layout as we might on an on-premises deployment We started with a single storage volume and the results failed to live up to expectations. Realising the storage was a problem we reorganised the storage to a more sensible configuration over 4 storage blocks using separate tablespaces to direct specific tables to specific storage:-</p><p>sda — contained the boot disk</p><p>sdb — contained most of the data files for the DB</p><p>sdc — contained the WAL log</p><p>sdd — contained table A</p><p>Each storage block was Oracle cloud balanced storage with a volume size of over 600Gb, which should give a throughput of 200 MB/s</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*WWbZzmMULp2rU-dKuKFQXA.png" /><figcaption>Initial testing</figcaption></figure><p>This gave us a realistic starting point to try and investigate the latency issues. As shown in the chart above, moving table A to a separate disk allowed us to monitor the storage latency of just this table. Needless to say, a read time of 15ms to retrieve data from disk for the tables on the shared storage and 6ms to read table A which will be served almost exclusively from disk, fell well below our hopes. It should be noted that because this was a saturation test we increased the load until we hit a limitation of the system; at lower volumes the storage performed in the 1ms range. We tried using a higher performance disk and this improved things to a point, as shown by the graph below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Nb8F453Q8TqdBuvHMeUzLQ.png" /><figcaption>WAL and table A given own storage block</figcaption></figure><p>This left us with a problem: using the highest quality storage available, the IOPS available simply was not capable of supporting the requests being made by the database. The only solution available was to add more storage and split the workload to see if that improved the latency.</p><p>We experimented with adding new storage volumes to the system and moving the hottest tables (A, B, C) to the new disks through the use of tablespaces pointing to the mounts of the new disks. This separated out the hot tables from the rest of the data and allowed the rest of the tables to enjoy better latency (1–2ms). This also allowed us to measure the latency of each disk holding hot tables. As we improved the storage performance, we added further parallel instances of the application under test to try to reach higher throughput rates. The throughput increased, however, we again reached a point where the storage holding A had a latency of 20ms. We had a table that required more IOPS than anyone storage block could provide and we still hadn’t run out of compute or memory.</p><p>We decided to split table A over a number of storage blocks using a hash partition* which greatly improved the read times for table A to the point where the writes of table B and C became relatively much larger in the pg_profile report so we decided to hash partition tables B and C as well over the same disks.</p><p>*It should be noted that using a logical volume striped across several disks is the traditional approach to distributing data evenly across the file system (and the one we use today). However part of the scope of the work was to investigate new ways of working with PostgreSQL and using a logical volume didn’t allow us to test out a new PostgreSQL feature we haven’t used before. For the record, creating an lvm vs hash partitioning delivered a seemingly equivalent performance in our tests.</p><p>sda — contained the boot disk</p><p>sdb — contained most of the data files for the DB</p><p>sdc — contained the WAL log</p><p>sdd ,sde,sdf,sdg,sdh contained table A B and C hash partitioned over 5 storage blocks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*_wlxuxCmJ4gAmoIOqEFzjA.png" /><figcaption>Partitioning of hot tables over 5 storage blocks</figcaption></figure><p>After making these changes, The WAL log writes were consistently below 1ms. The reads from all tables take around 1ms. We have the occasional spike in writes but they are not on the critical path.</p><p><strong>Key takeaways:-</strong></p><p>Coming from Oracle to PostgreSQL, one of the ‘hidden’ costs is that we need to consider the storage layout proactively; it is not something that is handled auto-magically as in Oracle and moving to the cloud isn’t a silver bullet. While cloud operators want us to think about buying storage services rather than physical hardware, we still need to consider what to do if the best storage device isn’t good enough on its own. We need to consider ways of using several storage blocks in parallel, through disk striping or hash partitioning J</p><p>The WAL log should always be on separate storage and it should be the fastest available. This may need to be a logical volume consisting of more than one storage device to provide the IOPS required.</p><p>That’s all for now, our work is still ongoing, next time I’ll cover the final results and general guidelines for organising PostgreSQL storage in the cloud.</p><p>Duncan Walker</p><p>Netcracker Technology</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=422fb30f3d21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/how-does-migrating-from-oracle-to-postgresql-in-the-cloud-impact-storage-422fb30f3d21">How does migrating from Oracle to PostgreSQL in the cloud impact storage?</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing a stuck HotSpot C2 compilation]]></title>
            <link>https://medium.com/netcracker/analyzing-a-stuck-hotspot-c2-compilation-85e0ca230744?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/85e0ca230744</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Vladimir Sitnikov]]></dc:creator>
            <pubDate>Fri, 17 Dec 2021 14:19:31 GMT</pubDate>
            <atom:updated>2021-12-17T14:19:31.671Z</atom:updated>
            <content:encoded><![CDATA[<p>Once upon a time there was a microservice-based application that ended up consuming CPU for no apparent reason. The system was idle, yet it consumed CPU. The problem was not specific to a particular service, and it reproduced quite often.</p><p>Configuration: CentOS 7.4.1708, Oracle JDK 1.8.0_152-b16 running inside Docker inside OpenShift.</p><p>Note: this might be a known issue, and it might be solved by just upgrading to 1.8.0_172, however the purpose of the article is to show a way to approach similar issues.</p><p>I would like to thank <a href="https://twitter.com/nitsanw">Nitsan Wakart</a> for helpful comments and corrections.</p><p>The first step is to ensure there’s a problem, so I just runtop:</p><pre>top - 09:09:57 up 134 days, 3:09, 0 users, load average: 3.36, ...<br>Threads:  80 total, 4 running, 76 sleeping, 0 stopped, 0 zombie<br>%Cpu(s): 19.5 us, 3.8 sy, 0.1 ni, 67.0 id, 0.0 wa<br>PID USER PR    VIRT    RES   SHR S  %CPU %MEM   TIME+ COMMAND <br> 46 root 20  441124 345740 10512 S 198.0 1.1  5294:27 java<br>  1 root 20   11796   1612  1212 S   0.0 0.0  0:00.03 bash</pre><p>As top refreshes it becomes apparent that java process 46 is consuming a couple of cores. How do we know which action is consuming the CPU? It has to be caused by one thread or another, howevertop does not split CPU consumption per thread by default.</p><p>One can use <a href="http://blog.ragozin.info/2018/05/sjk-is-learning-new-tricks.html">SJK ttop</a> by <a href="http://blog.ragozin.info/">Alexey Ragozin</a> to identify which threads consume CPU, allocate memory, but in this case top is good enough.</p><p>Per thread reporting can be enabled via top -H (that might depend on the OS):</p><pre>-H : Threads toggle<br>          Starts  top  with  the  last  remembered ’H’ state<br>          reversed.  When this toggle is On, all individual threads<br>          will be displayed.  Otherwise, top displays a summation of<br>          all threads in a process.</pre><p>The new display is as follows:</p><pre>PID USER PR    VIRT    RES   SHR S %CPU %MEM   TIME+  COMMAND <br> 68 root 20  441124 345740 10512 R 88.7  1.1  2328:34 java<br> 71 root 20  441124 345740 10512 R 85.0  1.1  2708:41 java<br> 66 root 20  441124 345740 10512 R  7.6  1.1 38:21.01 java</pre><p>As we see (the results are quite consistent as top refreshes), there are two threads consuming high CPU: 68 and 61. The next step is to identify what are the threads doing. Luckily a thread dump contains thread ids, so we just need to grab one: jstack 46 :</p><pre>...<br>&quot;C2 CompilerThread3&quot; #8 daemon prio=9 os_prio=0 tid=0xeb0f5c00 nid=0x44 runnable [0x00000000]                                                                                                                                    <br>   java.lang.Thread.State: RUNNABLE</pre><pre>&quot;C2 CompilerThread6&quot; #11 daemon prio=9 os_prio=0 tid=0xeb0fa800 nid=0x47 runnable [0x00000000]                                                                                                                                   <br>   java.lang.Thread.State: RUNNABLE</pre><p>Note: top shows thread ids in decimal, and thread dump shows thread ids in hex (0x44 == 68, 0x47 == 71)</p><p>Unfortunately the stacktrace is empty, so it is not a regular “badly looping Java code”, but it looks like an issue with a JVM itself as “C2 CompilerThread” is related to a JIT compiler.</p><h3>gdb</h3><p>Let’s go deeper. In order to get native stacktrace we can use either jstack -m 46 or pstack utility. It turns out pstack is not installed, so I install gdb first:</p><pre>yum install gdb</pre><p>pstack 46 produces the following (the output is pretty much consistent across pstack invocations):</p><pre>...<br>Thread 1 (process 68):                                                                                                                                                                                                           <br>#0  0xf6abda90 in MemBarCPUOrderNode::Opcode() const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                <br>#1  0xf6e8462c in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                       <br>#2  0xf6e84afb in LoadNode::Value(PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                            <br>#3  0xf6f5368a in PhaseIterGVN::transform_old(Node*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                <br>#4  0xf6f51084 in PhaseIterGVN::optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                          <br>#5  0xf6aece24 in Compile::Optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                               <br>#6  0xf6aeea0a in Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                           <br>#7  0xf6a43a4d in C2Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                <br>#8  0xf6af92b1 in CompileBroker::invoke_compiler_on_method(CompileTask*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                            <br>#9  0xf6afa0c7 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                             <br>#10 0xf7031f48 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                       <br>#11 0xf703cadf in JavaThread::thread_main_inner() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                   <br>#12 0xf703cc5b in JavaThread::run() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                                 <br>#13 0xf6f0c179 in java_start(Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so                                                                                                                               <br>#14 0xf77b8b2c in start_thread () from /lib/libpthread.so.0                                                                                                                                                                      <br>#15 0xf76cf17e in clone () from /lib/libc.so.6</pre><p>What does it tell us? JVM was trying to compile a method viaC2Compiler::compile_method and it got stuck there.</p><p>That gives us the first workaround: disable C2 compiler completely. We can do that by supplying -XX:TieredStopAtLevel=1 Java option. This option heals CPU consumption in idle state, so it validates our hypothesis that C2 causes the issue. However this workaround might not be valid and/or desirable for the production system as it disables the sophisticated C2 compiler, thus it might result in much lower application performance.</p><p>It turns out there are ways to exclude certain methods from being JIT compiled (see-XX:CompileCommand), so we can try to identify the bad method in question and add mark it for exclusion.</p><p>If we look closer, frame #7 has ciMethod* argument that looks like a method:</p><pre>#7  0xf6a43a4d in C2Compiler::compile_method(ciEnv*, ciMethod*, int)</pre><p>Let’s figure it out. In order to do that we can attach to the java process with a gdb.</p><h3>Core dump</h3><p>While GDB can be used on a live process, it makes sense to investigate in offline mode since OpenShift performs health-checks of the application and it can easily restart Docker container right in the middle of the debugging session. To go offline, we need to collect a core dump (that is the snapshot of the application memory)</p><pre># gcore 46<br>...<br>Saved corefile core.46</pre><p>Then we export core.46 to a safe location. Core dump is not self-sufficient for the analysis, but original binaries are required as well. In this case we have Docker, so we can just pull the same image and use it for the investigation:</p><pre>$ docker pull docker_image_name<br>$ docker exec -it --rm --mount type=bind,source=&quot;/folder/with/core&quot;,target=/u42 docker_image_name bash</pre><p>It gets into the bash shell to the same environment, so we can play with GDB:</p><pre># gdb /usr/java/jdk1.8.0_152/jre/bin/java /app/core.dump<br>...<br>[New LWP 231]<br>[New LWP 46]<br>[Thread debugging using libthread_db enabled]<br>Using host libthread_db library &quot;/lib64/libthread_db.so.1&quot;.<br>Core was generated by `/etc/alternatives/java&#39;.<br>#0  0xf7737430 in __kernel_vsyscall ()<br>Missing separate debuginfos, use: debuginfo-install jdk1.8-1.8.0_152-fcs.i586<br>(gdb) </pre><p>We need to find a thread and check its variables:</p><pre>(gdb) info threads<br>...<br>  20   Thread 0xeee7ab40 (LWP 66) 0xf7737430 in __kernel_vsyscall ()<br>  19   Thread 0xeeefbb40 (LWP 65) 0xf6dc767b in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>  18   Thread 0xeef7cb40 (LWP 64) 0xf7737430 in __kernel_vsyscall ()<br>  17   Thread 0xeeffdb40 (LWP 63) 0xf7737430 in __kernel_vsyscall ()</pre><p>Ah-ha! Here we go:</p><pre>(gdb) thread 19<br>[Switching to thread 19 (Thread 0xeeefbb40 (LWP 65))]<br>#0  0xf6dc767b in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>(gdb) bt<br>#0  0xf6dc767b in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#1  0xf6dc7afb in LoadNode::Value(PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#2  0xf6e9668a in PhaseIterGVN::transform_old(Node*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#3  0xf6e94084 in PhaseIterGVN::optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#4  0xf6a2fe24 in Compile::Optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#5  0xf6a31a0a in Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#6  0xf6986a4d in C2Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#7  0xf6a3c2b1 in CompileBroker::invoke_compiler_on_method(CompileTask*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#8  0xf6a3d0c7 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#9  0xf6f74f48 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#10 0xf6f7fadf in JavaThread::thread_main_inner() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#11 0xf6f7fc5b in JavaThread::run() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#12 0xf6e4f179 in java_start(Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so<br>#13 0xf76fbb2c in start_thread () from /lib/libpthread.so.0<br>#14 0xf761217e in clone () from /lib/libc.so.6</pre><p>Then we just need to select a thread frame and check its arguments. Let’s go with frame #6:</p><pre>(gdb) frame 6<br>#6  0xf6986a4d in C2Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so</pre><pre>(gdb) info args<br>No symbol table info available.</pre><p>What? It looks like gdb can’t get the debug information, so it cannot tell where (in which CPU registers) the method arguments are stored.</p><p>What do we do with “no symbol table info available”? Well, do you remember the message that gdb printed at the very beginning?</p><pre>Missing separate debuginfos, use: debuginfo-install jdk1.8-1.8.0_152-fcs.i586</pre><p>Let’s do it!</p><pre>Loaded plugins: fastestmirror, ovl<br>enabling base-debuginfo<br>Loading mirror speeds from cached hostfile<br> * base: mirror.yandex.ru<br> * extras: ftp.nsc.ru<br> * updates: mirror.yandex.ru<br>Could not find debuginfo for main pkg: 2000:jdk1.8-1.8.0_152-fcs.i586<br>No debuginfo packages available to install</pre><p>Unfortunately, I was not able to make that fly, so we should do something without debug information.</p><h3>VMStructs</h3><p>One of the parts of debug information is struct layout (the size of the structures, the types of the fields and its offsets). It turns out HotSpot JVM has that information by default even in regular release builds. It is called VMStructs and it is used by the tools like jstack You can find more on VMStructs in <a href="https://twitter.com/AndreiPangin">Andrei Pangin</a>’s <a href="https://jpoint.ru/en/talks/7lh7kkir1ye6uamik4w4cs/">VMStructs: how an application may benefit from knowing JVM internals talk</a> and his <a href="https://github.com/apangin/helfy/blob/ee390ae05f217e2cdca85fa6804a0a4a5791069f/test/one/helfy/HeapInfo.java#L9">wonderful</a> <a href="https://github.com/apangin/helfy">helfy</a> tool.</p><p>Let’s get our hands dirty. As you remember, we do not know argument values, however we can assume the arguments are stored either in registers or at the stack. We can just dump both and try to find something looking like a ciMethod</p><pre>(gdb) info registers<br>eax            0xe7e4a57c -404445828<br>ecx            0x0 0<br>edx            0xe7fcae98 -402870632<br>ebx            0xf70d2f40 -150130880<br>esp            0xeeefa4f0 0xeeefa4f0<br>ebp            0xeeefab08 0xeeefab08<br>esi            0xffffffff -1<br>edi            0xf70cdf08 -150151416<br>eip            0xf6986a4d 0xf6986a4d &lt;C2Compiler::compile_method(ciEnv*, ciMethod*, int)+541&gt;<br>eflags         0x293 [ CF AF SF IF ]<br>cs             0x23 35<br>ss             0x2b 43<br>ds             0x2b 43<br>es             0x2b 43<br>fs             0x0 0<br>gs             0x63 99</pre><pre>(gdb) x/10x $esp<br>0xeeefa4f0: 0xeeefa55c 0xeeefacf4 0xef2ab878 0x096fe388<br>0xeeefa500: 0xffffffff 0x00000001 0x00000001 0x00000001<br>0xeeefa510: 0x0120ba85 0xeeefab20</pre><p>How do we know which is which? Of course we start HSDB debugger (or jhsdb in Java 9). For more information on that, you might like <a href="https://twitter.com/volker_simonis">Volker Simonis</a>’ <a href="https://www.youtube.com/watch?v=xC8fEeo7izI">Analyzing HotSpot Crashes</a> talk.</p><p>This time I’ll use command line version of the tool:</p><pre># /usr/java/jdk1.8.0_152/bin/java -classpath /usr/java/jdk1.8.0_152/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB<br>hsdb&gt;</pre><p>Then we tell it to attach to our core file:</p><pre>attach /usr/java/jdk1.8.0_152/jre/bin/java /app/core.dump<br>Opening core file, please wait...<br>Error: java.lang.RuntimeException: can&#39;t load JavaScript engine</pre><p>Great! We just miss some JavaScript there. It is not clear why JavaScript was missing in my Docker image, however I was able to get through by just copying relevant (the one from 1.8u152)nashorn.jar into /usr/java/jdk1.8.0_162/jre/lib/ext folder:</p><pre># /usr/java/jdk1.8.0_152/bin/java -classpath /usr/java/jdk1.8.0_152/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB<br>hsdb&gt; attach /usr/java/jdk1.8.0_152/jre/bin/java /app/core.dump<br>Opening core file, please wait...<br>hsdb&gt;</pre><h3>Inspecting the core via CLHSDB</h3><p>It does not take much effort to just brute-force all the values stored in the registers and on the stack:</p><pre>hsdb&gt; inspect 0xe7e4a57c<br>Type is ConNode (size of 40)<br>Node** Node::_in: Node* @ 0xe7e4a5a4<br>Node** Node::_out: Node* @ null<br>node_idx_t Node::_cnt: 1<br>node_idx_t Node::_max: 1<br>node_idx_t Node::_outcnt: 0<br>node_idx_t Node::_outmax: 0<br>node_idx_t Node::_idx: 1<br>jushort Node::_class_id: 4<br>jushort Node::_flags: 16<br>hsdb&gt; inspect 0xe7fcae98<br>hsdb&gt; inspect 0xf70d2f40<br>hsdb&gt; inspect 0xeeefa4f0<br>hsdb&gt; inspect 0xeeefab08<br>hsdb&gt; inspect 0xf70cdf08<br>hsdb&gt; inspect 0xeeefa55c<br>hsdb&gt; inspect 0xeeefacf4<br>hsdb&gt; inspect 0xef2ab878</pre><pre>hsdb&gt; inspect 0x096fe388<br>Type is ciMethod (size of 100)<br>uint ciBaseObject::_ident: 1670<br>Metadata* ciMetadata::_metadata: Metadata @ 0xedaab218<br>int ciMethod::_interpreter_invocation_count: 31298<br>int ciMethod::_interpreter_throwout_count: 0<br>int ciMethod::_instructions_size: -1</pre><p>Ah-ha! We have a ciMethod, however it does not look like anything to us. Where is the name?</p><p>Let’s check if ciMethod has relevant fields to get the name. Of course, we can clone the JDK sources or we can just Google for ciMethod.hpp and <a href="https://github.com/JetBrains/jdk8u_hotspot/blob/a051c3d5f12d12252624b9130eca11739adfe89a/src/share/vm/ci/ciMethod.hpp#L47-L61">check a random repository</a>:</p><pre>class ciMethod : public ciMetadata {<br>...<br>private:  // General method information.<br>  ciFlags          _flags;<br>  ciSymbol*        _name;<br>  ciInstanceKlass* _holder;<br>  ciSignature*     _signature;</pre><p>It looks like those fields are not in VMStructs so let’s just brute force them. As we know, 0x096fe388 points to a ciMethod instance, so we should check its contents:</p><pre>hsdb&gt; examine 0x096fe388/40<br>0x096fe388: 0xf70bf108 0x00000686 0xedaab218 0x81000009 0x096fe3f0 0x096fe328<br>0x096fe3a0: 0x096fe410 0x096fe550 0x096ff248 0x00000031 0x00000005 0x00000003<br>0x096fe3b8: 0x00000000 0x00000000 0x00007a42 0x00000000 0xffffffff 0x00000002<br>0x096fe3d0: 0x01010100 0x00000001 0x096ff378 0x096ff3b0 0x09700320 0x096ff200</pre><p>Then we check each value one by one:</p><pre>hsdb&gt; inspect 0xf70bf108<br>hsdb&gt; inspect 0x00000686<br>hsdb&gt; inspect 0xedaab218<br>Type is Method (size of 48)<br>ConstMethod* Method::_constMethod: ConstMethod @ 0xedaab188<br>MethodData* Method::_method_data: MethodData @ 0xeaa9cef8<br>MethodCounters* Method::_method_counters: MethodCounters @ 0xed6e5f50<br>AccessFlags Method::_access_flags: -2130706423<br>int Method::_vtable_index: -2<br>u2 Method::_method_size: 12<br>u1 Method::_intrinsic_id: 0<br>nmethod* Method::_code: nmethod @ null<br>address Method::_i2i_entry: address @ 0xedaab234<br>AdapterHandlerEntry* Method::_adapter: AdapterHandlerEntry @ 0xf656d660<br>address Method::_from_compiled_entry: address @ 0xedaab23c<br>address Method::_from_interpreted_entry: address @ 0xedaab244<br>hsdb&gt; inspect 0x81000009</pre><pre>hsdb&gt; inspect 0x096fe3f0<br>Type is ciSymbol (size of 16)<br>Symbol* ciSymbol::_symbol: Symbol @ 0xee1ef060<br>hsdb&gt; inspect 0x096fe328<br>Type is ciInstanceKlass (size of 80)<br>uint ciBaseObject::_ident: 1664<br>Metadata* ciMetadata::_metadata: Metadata @ 0xedaabb40<br>BasicType ciType::_basic_type: 12<br>ciSymbol* ciKlass::_name: ciSymbol @ 0x096fe378<br>InstanceKlass::ClassState ciInstanceKlass::_init_state: 4<br>bool ciInstanceKlass::_is_shared: 0</pre><p>Well, it looks like 0x096fe3f0 points to the method name and 0x096fe328 points to the class. Let’s dump those as well:</p><pre>hsdb&gt; inspect 0x096fe3f0<br>Type is ciSymbol (size of 16)<br>Symbol* ciSymbol::_symbol: Symbol @ 0xee1ef060<br>hsdb&gt; inspect 0xee1ef060</pre><p>Bad luck. hsdb does not show meaningful information here. It turns out we need to use symbol command to inspect symbols:</p><pre>hsdb&gt; symbol 0xee1ef060<br>#forMethodParameter<br>hsdb&gt; inspect 0x096fe378<br>Type is ciSymbol (size of 16)<br>Symbol* ciSymbol::_symbol: Symbol @ 0xee0a0e80<br>hsdb&gt; symbol 0xee0a0e80<br>#org/springframework/core/ResolvableType</pre><p>Great. Now we know that the issue was caused by C2 trying to compile org.springframework.core.ResolvableType#forMethodParameter method. We can add the method to exclusion list via the following Java option</p><pre>-XX:CompileCommand=exclude,org.springframework.core.ResolvableType::forMethodParameter</pre><p>Note: CompileCommand=exclude excludes the method completely, so the method will be interpreted. It might affect the performance of the application, thus CompileCommand should be treated as a workaround.</p><p>There are lots of methods forMethodParameter, and we can identify the “bad” overload for the fun’s sake. Method signature comes right after ciInstanceClass*, so 0x96fe410 is the signature:</p><pre>hsdb&gt; inspect 0x096fe410<br>hsdb&gt;<br>hsdb&gt; whatis 0x096fe410<br>Address 0x096fe410: In unknown location</pre><p>Here’s <a href="https://github.com/JetBrains/jdk8u_hotspot/blob/a051c3d5f12d12252624b9130eca11739adfe89a/src/share/vm/ci/ciSignature.hpp#L36-L43">the source</a> for ciSignature:</p><pre>class ciSignature : public ResourceObj {<br>  ciSymbol* _symbol;<br>  ciKlass*  _accessing_klass;<br>  GrowableArray&lt;ciType*&gt;* _types;<br>  int _size;   // number of stack slots required for arguments<br>  int _count;  // number of parameter types in the signature</pre><p>The dump of 0x96fe410 reveals that the method has 2 arguments (_size and _count are both equal to 2):</p><pre>hsdb&gt; examine 0x96fe410/5<br>0x096fe410: 0x096fe400 0x096fe328 0x096fe428 0x00000002 0x00000002<br>hsdb&gt; inspect 0x096fe400<br>Type is ciSymbol (size of 16)<br>Symbol* ciSymbol::_symbol: Symbol @ 0xee1ef2b0<br>hsdb&gt; symbol 0xee1ef2b0<br>#(Lorg/springframework/core/MethodParameter;Ljava/lang/reflect/Type;)Lorg/springframework/core/ResolvableType;</pre><h3>Conclusion</h3><p>Command line debugging is not that fun while you debug, however it is just amazing which capabilities you get with things like VMStructs and HSDB.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85e0ca230744" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/analyzing-a-stuck-hotspot-c2-compilation-85e0ca230744">Analyzing a stuck HotSpot C2 compilation</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dependency verification: checksum vs PGP]]></title>
            <link>https://medium.com/netcracker/dependency-verification-checksum-vs-pgp-582e76207019?source=rss----ff491b436b53---4</link>
            <guid isPermaLink="false">https://medium.com/p/582e76207019</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[gradle]]></category>
            <category><![CDATA[pgp]]></category>
            <category><![CDATA[dependency-management]]></category>
            <dc:creator><![CDATA[Vladimir Sitnikov]]></dc:creator>
            <pubDate>Fri, 17 Dec 2021 14:18:54 GMT</pubDate>
            <atom:updated>2022-05-09T14:17:48.770Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m the author of <a href="https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin">checksum-dependency-plugin</a>. The purpose of this article is to convince you to use checksums and/or PGP signatures in your build scripts.</p><p>The article focuses on Java ecosystem (<a href="https://www.sonatype.com/nexus-repository-oss">Sonatype OSS repository</a>, Apache Maven, Gradle); however, very similar principles are used with great success in other dependency managers.</p><p>Even though checksums speedup build execution, I treat it as out of scope for the article (unless someone proves it should be in scope).</p><p>Note: I’m not a security expert, however, I do my best to represent things in a proper light.</p><p>Many thanks go to <a href="https://twitter.com/dave_cramer">Dave Cramer</a> and <a href="https://twitter.com/jbaruch/">Baruch Sadogursky</a> for the review.</p><h3>2022–05–09: Sigstore is the next standard for signing and verification</h3><p>With PGP, you can’t trust the identity (see “malicious PGP identity” below), and the lifetime of a PGP key is quite long, so the key compromise is a disaster. <a href="https://www.sigstore.dev/">Sigstore</a> solves both issues by verifying the identity at <strong>the time of signing</strong>, and it issues short-time certificates, so the danger of the compromised certificate is way less. On top of that, the open ledger <a href="https://github.com/sigstore/rekor">Rekor</a> enables detection of all the signatures, so malicious signatures can easily be spotted.</p><p>Maven Central is going to integrate Sigstore: <a href="https://central.sonatype.org/news/20220310_sigstore/">https://central.sonatype.org/news/20220310_sigstore/</a></p><p>Sigstore Java is getting into shape: <a href="https://github.com/sigstore/sigstore-java">https://github.com/sigstore/sigstore-java</a></p><p>Gradle issue “sigstore: support signing and verification artifacts via transparency log”: <a href="https://github.com/gradle/gradle/issues/19748">https://github.com/gradle/gradle/issues/19748</a></p><h3>2019–12–23: Gradle 6.2 brings integrated verification</h3><p>Gradle 6.2 is going to introduce integrated <a href="https://github.com/gradle/gradle/issues/10443#issuecomment-568741472">checksum and signature verification</a>. It supersedes checksum-dependency-plugin. So if you can upgrade to Gradle 6.2 when it is released, you probably don’t need checksum-dependency-plugin.</p><p>The contents of the post is relevant for all Gradle versions up to 6.2.</p><h3>Prior art</h3><p>I don’t provide an extensive list here, however below are the articles that are relevant. It does not mean I fully agree with all the contents, however, I think they are worth reading. Please take into account publication dates as certain bits might be just out of date.</p><ul><li><a href="http://branchandbound.net/blog/security/2012/03/crossbuild-injection-how-safe-is-your-build/">http://branchandbound.net/blog/security/2012/03/crossbuild-injection-how-safe-is-your-build/</a></li><li><a href="https://www.netbsd.org/developers/pgp.html">https://www.netbsd.org/developers/pgp.html</a></li><li><a href="https://jfrog.com/blog/feel-secure-with-ssl-think-again/">https://jfrog.com/blog/feel-secure-with-ssl-think-again/</a></li><li><a href="https://github.com/gary-rowe/BitcoinjEnforcerRules">https://github.com/gary-rowe/BitcoinjEnforcerRules</a></li><li><a href="https://blog.filippo.io/giving-up-on-long-term-pgp/">https://blog.filippo.io/giving-up-on-long-term-pgp/</a></li><li><a href="https://medium.com/@jonathan.leitschuh/many-of-these-gpg-signatures-are-signed-with-sha-1-which-is-vulnerable-to-a-second-preimage-attack-67104d827930">https://medium.com/@jonathan.leitschuh/many-of-these-gpg-signatures-are-signed-with-sha-1-which-is-vulnerable-to-a-second-preimage-attack-67104d827930</a></li></ul><h3>The need for verification</h3><p>Virtually all the software is using dependencies in one way or another.</p><p>For instance, let us consider the following:</p><pre>&lt;dependency&gt;<br>  &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;<br>  &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;<br>  &lt;version&gt;2.11.1&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>You declare a couple of lines, and the build system downloads the required dependency transparently for you. There’s a catch: what if somebody has replaced the file at the Maven Central repository or intercepts the network request?</p><p>If the log4j-core.jar is replaced, then you will be executing arbitrary code that you have just downloaded from the Internet.</p><p>You don’t want to execute arbitrary code on your machine, so you want to verify if the downloaded dependencies are “what you think they are”</p><p>Note: there’s no “ultimate security level”. It might turn out your computer was rooted at the factory, and all your keystrokes are logged and silently sent over Bluetooth. You never know unless you completely disassemble your hardware.</p><p>On the other hand, you don’t want to grab random jars from the Internet and execute them. So you need to figure out “good enough” security level that suits your workflow.</p><p>For instance, I don’t disassemble my digital piano to check if it is trying to steal data from me. I just assume that risk is too low. Someone might scream aloud, but I’m willing to trust the official release managers of Apache Log4j. That is I’m ok to execute a jar if I know it was produced by Apache Log4j project. Your preferences might vary, however, you still need to identify a set of identities you trust, and you need to establish a way to verify that the artifacts were produced by the trusted parties.</p><h3>Maven Central, JCenter and others are secure. It is just enough security</h3><p>You should trust neither Maven Central nor JCenter.</p><p>Note: even if you inspect GitHub sources for the library in question that does not mean the published jar files were built from the source files you’ve inspected.</p><p>Here’s a recent case, when a malicious jar file was published while the sources looked OK: <a href="https://blog.autsoft.hu/a-confusing-dependency/">https://blog.autsoft.hu/a-confusing-dependency/</a></p><p>Here’s a case for NPM: <a href="https://news.ycombinator.com/item?id=14901566">https://news.ycombinator.com/item?id=14901566</a></p><p>Note: it does not mean JCenter is bad. It just means “source code available” is not enough to trust the jar. The case could have been prevented if PGP was used to verify the jar.</p><h3>Level 0: HTTPS</h3><p>Rule #0: please never use http:// URLs for repositories. The connections over plain HTTP are not that hard to intercept, so just use https://</p><p>JCenter will deny HTTP requests</p><h3>Level 1: checksums</h3><p>Isn’t https enough? When you use https://repository.acme.com/… URL it typically means the server has to provide a valid certificate for repository.acme.com domain. This is good provided you never add “root certificates” to the keystore.</p><p>You might notice that Maven Central has <a href="https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.11.1/">checksums</a> (see <a href="https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.11.1/log4j-core-2.11.1.jar.sha1">sha1</a> and <a href="https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.11.1/log4j-core-2.11.1.jar.md5">md5</a>). In fact, the checksums are mandatory, and one can’t release a package without those checksums.</p><p>Here’s how you verify SHA1:</p><pre>$ openssl dgst -sha1 log4j-core-2.11.1.jar<br>SHA1(log4j-core-2.11.1.jar)=592a48674c926b01a9a747c7831bcd82a9e6d6e4</pre><p>My computation seems to agree with the value that Maven Central shows to me. Does that mean the file is good to use? Unfortunately, it does not.</p><p>If someone intercepts my requests to <a href="https://repo1.maven.org/">https://repo1.maven.org</a>, then they can intercept requests to .sha1 file as well and compute the checksum on the fly.</p><p>Does that mean checksums are completely useless? It does not. Checksums enable one to verify if the file was downloaded properly.</p><p>As you might guess, manual verification of the checksums is not practical.</p><p>Certain build systems (e.g. <a href="https://bazel.build/">Bazel</a>) require every dependency declaration to be accompanied by its checksum:</p><p><a href="https://docs.bazel.build/versions/master/be/workspace.html#maven_jar_examples">https://docs.bazel.build/versions/master/be/workspace.html#maven_jar_examples</a></p><pre>maven_jar(<br>    name = &quot;com_google_guava_guava&quot;,<br>    artifact = &quot;com.google.guava:guava:18.0&quot;,<br>    sha1 = &quot;cce0823396aa693798f8882e64213b1772032b09&quot;,<br>    sha1_src = &quot;ad97fe8faaf01a3d3faacecd58e8fa6e78a973ca&quot;,<br>)</pre><p>In other words, the expected checksums need be declared somehow. Unfortunately, <a href="https://stackoverflow.com/a/34795359/1261287">neither Maven</a> (as of 3.6.2) nor <a href="https://github.com/gradle/gradle/issues/10443">Gradle</a> (as of 5.6.2) allows one to declare checksums that will be verified during dependency resolution.</p><h4>Implementing checksums via plugins</h4><p>If you are using Maven, you can try <a href="https://github.com/gary-rowe/BitcoinjEnforcerRules">BitcoinjEnforcerRules</a> or <a href="https://github.com/s4u/pgpverify-maven-plugin">pgpverify-maven-plugin</a>.</p><p>If you are using Gradle (prior to 6.2), you can use <a href="https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin">checksum-dependency-plugin</a>.</p><p>However, there’s a question: where do you get “the expected checksum” from? It should be taken from a place you trust.</p><p>For instance, Apache Maven download page has links to the expected SHA-512 of the source and binary archives. However, it would be very surprising to see that every library author publishes the expected SHA-512 of all the library versions. It would be cool, but it is currently not complete.</p><p>A semi-solution would be to download the file somehow (e.g. from the server you trust using the machine you trust), compute the checksum, and bake it to the build script. Then it would warn you if someone else gets another checksum for exactly the same artifact.</p><h3>Level 2: PGP</h3><p>Checksums enable you to recognize when the file differs, however they do not link artifacts to the trusted identities.</p><p>Apache Software Foundation provides <a href="https://checker.apache.org/">https://checker.apache.org/</a> that enables you to type a checksum and it tells you if the checksum is a part of the official release. Note: Apache focuses on source code releases, so it is likely you won’t find a match for a jar file.</p><p>Luckily, PGP helps to solve that issue.</p><p>TL;DR: PGP allows to create cryptographic signatures, so everybody can verify that the signature was created by the owner of the private key. In other words, log4j-core-2.12.0.jar.asc signature is “impossible” to forge (see “malicious PGP identity” below).</p><p>Note: PGP signs hash digest of a file, so it is important that strong digest is used. Here’s a case when <a href="https://medium.com/@jonathan.leitschuh/many-of-these-gpg-signatures-are-signed-with-sha-1-which-is-vulnerable-to-a-second-preimage-attack-67104d827930">PGP signature uses SHA1 checksum</a> which is not that secure. Here’s a <a href="https://github.com/gradle/gradle/pull/10543">PR to Gradle to use SHA256</a> when signing artifacts.</p><p>If you use <a href="https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin">checksum-dependency-plugin</a>, you can declare:</p><pre>&lt;trust-requirement pgp=&#39;GROUP&#39; checksum=&#39;MODULE&#39; /&gt;<br>&lt;trusted-keys&gt;<br>  &lt;trusted-key id=&#39;3595395eb3d8e1ba&#39;<br>            group=&#39;org.apache.logging.log4j&#39; /&gt;</pre><p>That would mean the plugin would ensure that artifacts from org.apache.logging.log4j group must be signed with PGP key of 3595395eb3d8e1ba.</p><p>Note: it is the default mode for checksum-dependency-plugin. It tries to use PGP for dependency verification, and if the dependency is not signed (for instance, <a href="https://github.com/gradle/gradle/issues/10182">Gradle Plugin Portal does not allow to publish PGP signatures</a> as of 2019–09–09)</p><h4>Malicious PGP identity</h4><p>New PGP keys are very easy to create, so you shouldn’t trust each and every PGP key.</p><p>However, keep in mind that Maven/Gradle never verify PGP identities during artifact resolution, so if you add a requirement for PGP keys, then you make your build more secure even in the case where you have never met the key owner in person. If you require a valid PGP signature, it makes it much harder for an attacker to forge a malicious artifact, on the other hand, you don’t really need to update the set of PGP ids often. You can update library versions, and existing PGP keys would likely match.</p><h4>Verification of PGP identities</h4><p><a href="https://www.netbsd.org/developers/pgp.html#sign-recommendations">https://www.netbsd.org/developers/pgp.html#sign-recommendation </a>suggests you need to meet the key owner in person, and verify their ID. Unfortunately, that is not always possible.</p><p>For Apache projects you can find PGP keys that are used by release managers in the relevant KEYS files (see <a href="https://www.apache.org/dist/logging/KEYS">https://www.apache.org/dist/logging/KEYS</a> ).</p><p>If the project page fails to list the PGP key, you can meet project lead in question and/or file an issue to publish project signing key in the README.</p><p>The release manager might happen to sign a Git tag. For instance, you can see that <a href="https://github.com/jacoco/jacoco/releases/tag/v0.8.4">JaCoCo 0.8.4 is signed</a> with key CB43338E060CF9FA.</p><p><a href="https://keybase.io/godin">https://keybase.io/godin</a> shows that that the owner of that key controls GitHub account <a href="https://github.com/Godin">https://github.com/Godin</a>, Twitter account <a href="https://twitter.com/_Godin_">https://twitter.com/_Godin_</a>, and so on. After investigation of the listed accounts, it might look ok to just trust that key. However, beware that all those accounts might be fake.</p><p>Sample requests to publish the official release PGP keys: <a href="https://gitlab.ow2.org/asm/asm/issues/317878">ObjectWeb ASM</a>, <a href="https://github.com/spring-projects/spring-framework/issues/23434#issuecomment-523882229">Spring Framework</a>, <a href="https://github.com/raphw/byte-buddy/issues/721">ByteBuddy</a>, <a href="https://github.com/jacoco/jacoco/issues/937">JaCoCo</a>, <a href="https://github.com/GPars/GPars/issues/62">GPars</a></p><h3>Level 3: PGP+checksum</h3><p>If a simple PGP is not enough, you might want to use both PGP and checksums.</p><p>The declaration for checksum-dependency-plugin would be:</p><pre>&lt;trust-requirement pgp=&#39;MODULE&#39; checksum=&#39;MODULE&#39; /&gt;<br>&lt;dependency<br>    group=&#39;org.apache.logging.log4j&#39;<br>    module=&#39;log4j-core&#39;<br>    version=&#39;2.11.1&#39;&gt;<br>  &lt;pgp&gt;3595395eb3d8e1ba&lt;/pgp&gt;<br>  &lt;sha512&gt;6D23ECAF0AC18A5...C7DA3DDFC3FFB96B261&lt;/sha512&gt;<br>&lt;/dependency&gt;</pre><p>It would be a bit more complicated to support (every dependency update would require to add new checksum), however, SHA-512 would prevent unexpected changes, and PGP would simplify the review (you see that old and new library are signed with the same PGP key).</p><h3>Level 4: who verifies the verifier?</h3><p>I’m not sure about Maven, however, I don’t think Maven (as of 3.6.2) allows you to block the execution of a plugin if its checksum does not match expectations.</p><p>Gradle allows one to resolve the classpath of the build script and compute checksums before the plugins are activated/applied. That enables you to download checksum-dependency-plugin from the repository, and you can still verify that the file matches your expectations.</p><p>See full sample <a href="https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin#installation">plugin README</a>.</p><p>The important bit here is that the plugin is added via settings.gradle script (or settings.gradle.kts if Kotlin DSL is used). That enables the plugin to intercept all the further resolutions.</p><p>The expected checksums are published in the README file. On top of that, you can check out plugin sources, build it in your environment and crosscheck the resulting jar file. Travis CI <a href="https://travis-ci.org/vlsi/vlsi-release-plugins/jobs/581575674#L334">prints the expected checksums as well</a>.</p><h3>Level 5: sandbox repository</h3><p>Your build is never secure if you allow it to download random code from the Internet. It does not matter if you verify checksums or not.</p><p>For instance, Gradle build script is written in Apache Groovy / Kotlin, and the script can issue a plain old HTTP(s) request to download extra dependency. That won’t be visible for “dependency verification plugins”, so you are exposed to security issues.</p><p>The ultimate solution would be to setup a proxy repository and allow your build machine to access that proxy repository only. That would intercept all the possible resolution requests, and the proxy repository can decide if a certain dependency is allowed or not.</p><p>That is great if you host Artifactory and/or Nexus Pro repository. However proxy approach is not very feasible for open-source development.</p><p>Just in case, Artifactory configuration can be found at “<a href="https://jfrog.com/blog/feel-secure-with-ssl-think-again/">But Maven just goes out there and brings stuff</a>”</p><p>Nexus Pro can work as a <a href="https://dzone.com/articles/why-nexus-and-not-artifactory">proxy repository that transparently verifies PGP signatures for all the resolved dependencies</a></p><h3>Does PGP really help?</h3><p>TL;DR: it does.</p><p>Consider a case from <a href="https://blog.autsoft.hu/a-confusing-dependency/">https://blog.autsoft.hu/a-confusing-dependency/</a></p><p>(Un)fortunately the dependency in question (com.github.adrielcafe:AndroidAudioRecorder:0.3.0) has been removed from JCenter.</p><p>However, the article mentions</p><blockquote>Created by the obviously fake <em>jakewhaarton</em></blockquote><p>Let us try adding a <a href="https://github.com/JakeWharton/picasso2-okhttp3-downloader">random JakeWharton / picasso dependency</a>:</p><pre>dependencies {<br>  compile &#39;com.jakewharton.picasso:picasso2-okhttp3-downloader:1.1.0&#39;<br>}</pre><pre>$ ./gradlew help</pre><p>Here’s how checksum-dependency-plugin identifies the issue and fails the build:</p><pre>FAILURE: Build failed with an exception.</pre><pre>* What went wrong:<br>A problem occurred configuring project &#39;:bitcoinj-wallettemplate&#39;.<br>&gt; Checksum/PGP violations detected on resolving configuration :bitcoinj-wallettemplate:runtimeClasspath<br>    <strong>No trusted PGP keys</strong> are configured for group com.jakewharton.picasso:<br>      com.jakewharton.picasso:<strong>picasso2-okhttp3-downloader</strong>:1.1.0 (pgp=[<strong>80c08b1c29100955</strong>], sha512=[computation skipped])<br>    No trusted PGP keys are configured for group com.squareup.picasso:<br>      com.squareup.picasso:<strong>picasso</strong>:2.5.2 (pgp=[<strong>80c08b1c29100955</strong>], sha512=[computation skipped])</pre><pre>You might want to add -PchecksumFailOn=build_finish if you are brave enough<br>  It will collect all the violations, however untrusted code might be executed (e.g. from a plugin)</pre><p>We see that two unknown dependencies appear in the build script (picasso2-okhttp3-downloader and picasso). We see that both dependencies are signed with the same PGP key: 80c08b1c29100955</p><p>How can we tell if the key belongs to a true person or a fake one?</p><p>Let’s try <a href="https://keybase.io/">Keybase</a>: we put key id there, and it suggests <a href="https://keybase.io/jakewharton">https://keybase.io/jakewharton</a></p><p>Luckily for us, Keybase reads that the owner of key 80c08b1c29100955 controls quite a few accounts, and the accounts look decent.</p><p>Even though I have never met Jake in person yet, the above is quite convincing for me to trust key 80c08b1c29100955 for com.jakewharton.picasso and com.squareup.picasso packages. Well, it is not clear why Jake releases under com.squareup.picasso, so it needs extra investigation, however, <a href="https://github.com/square/picasso/graphs/contributors">https://github.com/square/picasso/graphs/contributors</a> shows that Jake contributes to that library a lot, so it looks safe.</p><p>Note: checksum-dependency-plugin creates an updated $rootDir/build/checksum/checksum.xml file so you can compare it with the current one. If you are brave enough, you might add -PchecksumUpdate so the root checksum.xml would be updated automatically.</p><p>This case is quite trivial: a single Keybase search results in a rich identity. Many thanks to Jake for verifying the identity via Keybase.</p><h3><strong>Frequently Asked Questions</strong></h3><h4>Which security level should I use?</h4><p>If you can afford a proxy repository, go for it, and configure the set of trusted dependencies (~ Level 5).</p><p>If you work on an open-source project/library/application, I would suggest you use “Level 2: PGP” for verification of your builds.</p><h4>I’m using https and it is just fine</h4><p>It is good if you use https for all the repositories, however it does not protect you from a consuming a jar file released by a fake identity.</p><h4>PGP is useless without “web of trust”</h4><p>That depends. At some point, you need to have a way to identify the release manager you trust. That can come from your <a href="https://en.wikipedia.org/wiki/Web_of_trust">web of trust</a>. However, sometimes keybase.io verified identity is good enough.</p><p>It might look like “web of trust” would enable you to verify artifacts, however, it looks like no-one ever uses PGP web of trust. See <a href="https://blog.filippo.io/giving-up-on-long-term-pgp/">https://blog.filippo.io/giving-up-on-long-term-pgp/</a></p><p>PGP seems to be the less evil way to verify artifacts for projects where you can’t afford to have a proxy repository. PGP makes your builds more secure even in case you never participate in a <a href="https://en.wikipedia.org/wiki/Key_signing_party">Key signing party</a>.</p><h4>I implement a library, so I do not need dependency verification</h4><p>This is false. Even if your project is just a library, you still do not want your computer to execute arbitrary code from the Internet. You need to verify dependencies to keep your Bitcoins and passwords safe.</p><h4>PGP is useless: you still suggest to just trust Twitter/GitHub/… accounts</h4><p>I do not suggest that you trust a Twitter account. In fact, there’s no way to verify if the jar in question was published by <a href="https://twitter.com/JakeWharton">https://twitter.com/JakeWharton</a> or not. Jake does not publish checksum for each and every released jar. So how should I verify if the jar was released by true Jake or fake Jake? PGP allows me to verify if the jar was signed by the owner of key 80c08b1c29100955, and it is hard to provide a malicious signature with the same key id.</p><p>Note: <a href="https://www.netbsd.org/developers/pgp.html#sign-recommendations">https://www.netbsd.org/developers/pgp.html#sign-recommendations</a> says that you should verify not just key id, but other key properties like “key length”.</p><h4>Should I verify POM files?</h4><p>Gradle (as of 5.6.2) does not treat .pom files as artifacts, so it does not fire the “dependency resolution” listener for them when resolving regular dependencies.<br>Gradle 6.2 will be able to verify checksums and signatures of the .pom files.</p><p>However malicious pom file can’t harm much: it does not contain executable code, so it can’t result in remote code execution. Of course, it might result in a different set of build dependencies, however, <a href="https://reproducible-builds.org/">https://reproducible-builds.org/</a> could help you to verify if builds on different machines result in the same binaries.</p><p>Note: io.spring.gradle:dependency-management-plugin resolves and processes .pom files as regular artifacts. That resolution is can be intercepted and verified. So you can validate POM files, however, it requires non-trivial effort and I do not think it adds much.</p><h4>I do not find .asc signature for a dependency. What should I do?</h4><p>It happens that even very popular libraries fail to ship PGP signatures. When you see that please ask the project to sign the releases.</p><p>For instance, here’s an issue I’ve filed for ObjectWeb ASM: <a href="https://gitlab.ow2.org/asm/asm/issues/317878">https://gitlab.ow2.org/asm/asm/issues/317878</a>. It turns out the lack of PGP signatures was not expected by the developers.</p><h4>A newer dependency version is signed with a completely different key. What should I do?</h4><p>You need to figure out if the new key is the official release signing key for the artifact in question. Note: the same binary can be signed with multiple keys, however all of the keys might belong to fake identities.</p><p>So you should ask the project team to clarify which PGP keys are mandatory to sign official releases, and you should not blindly trust any new keys you meet.</p><p>Sample case: <a href="https://github.com/walleth/walleth/pull/396#issuecomment-529602524">https://github.com/walleth/walleth/pull/396#issuecomment-529602524</a></p><p>Clarification issue to Kotlin team: <a href="https://youtrack.jetbrains.com/issue/KT-33781">https://youtrack.jetbrains.com/issue/KT-33781</a></p><h4>How would I know if PGP key can be trusted?</h4><p>See Verification of PGP identities section above.</p><h4>The artifact is signed, and PGP key is listed on the project website as the official one. Does that mean the artifact is safe for use?</h4><p>Unfortunately, valid signature does not mean the artifact is safe to use.</p><p>Valid signature means that the artifact was released by the right person, however keep in mind that <a href="https://xkcd.com/538/">a wrench costs just $5</a></p><h4>I’m library author. Should I use PGP?</h4><p>If you release a library, please ensure you sign your releases, and please publish PGP key ids at the project webpage. That would enable consumers to verify if they use the official releases or not.</p><p>You might want to verify dependencies of your library as well, especially in case you shade/bundle artifacts. For instance, it is common that ObjectWeb ASM is repackaged and shaded. However you do not want to ship malicious code, so you need to verify if the version in question is the official one</p><h4>I’m using a version range. Should I ignore the verification then?</h4><p>Please consider <a href="https://reproducible-builds.org/">making your builds reproducible</a>. That is version range is just fine provided the resolution ends up with a predictable version.</p><p>On top of that, even if you want to use ranges like 1.+ you might still get some security out of PGP verification. PGP keys change not that often, so you might just list all the possible keys, and automatic updates would play well with the verification.</p><h4>Ok. I want to implement dependency verification in my Gradle project. What should I do?</h4><p>Try a pre-release Gradle version that has built-in verification.</p><p>For older versions consider using <a href="https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin">https://github.com/vlsi/vlsi-release-plugins/tree/master/plugins/checksum-dependency-plugin</a></p><p>The steps to integrate the plugin are:</p><ol><li>Add a relevant section to settings.gradle (and to buildSrc/settings.gradle if you have one)</li><li>Collect current dependencies. You can use ./gradlew allDependencies -PchecksumUpdate. allDependencies task comes with the plugin</li><li>Add -PchecksumFailOn=build_finish -PprintChecksum to your CI configuration. That would collect all the violations and print the updated checksum.xml contents to the <a href="https://travis-ci.org/spotbugs/spotbugs/jobs/583073027#L2583">build output</a>.</li></ol><p>Note: by default, the plugin fails on the first violation. That ensures your builds are secure (you don’t accidentally pull untrusted artifacts), however, that is painful for CI (you don’t want to add new dependencies one by one). However, you might want to refrain from using -PchecksumFailOn=build_finish for “deploy CI jobs” because you do not want to execute untrusted code in an environment that is capable to publish artifacts.</p><h4>Is checksum-dependency-plugin signed?</h4><p>Nice catch. Unfortunately, Gradle Plugin Portal <a href="https://github.com/gradle/gradle/issues/10182">rejects PGP signatures</a>. Please vote for the issue so Gradle plugins could be signed.</p><p>As of now, the builds of checksums-dependency-plugin are reproducible, so you can build your own jar and end up with exactly the same binary.</p><h4>I develop an Android application and I see .aar files lack PGP signatures. Is that intentional?</h4><p>.aar files could be signed as well, so please contact artifact authors so they provide PGP signatures.</p><p>If you are curious, here’s an <a href="https://github.com/trezor/trezor-android/pull/53/files#diff-e50b6b9c02782cb6adb9dc5723e30c9aR51">example how aar dependency looks like</a>.</p><h3>Does anybody use checksum/PGP for dependency verification?</h3><p>Debian has enforced PGP for releases for <a href="https://wiki.debian.org/SecureApt">quite a while</a>. CentOS <a href="https://www.centos.org/keys/">requires PGP as well</a>. Apache Software Foundation <a href="http://www.apache.org/legal/release-policy.html#release-signing">enforces that all official releases must be signed</a>, and they <a href="https://www.apache.org/info/verification.html">recommend to use PGP for verification</a>. PGP does not completely prevent malicious code execution, however, PGP verification is better than nothing.</p><p>Relevant Gradle issues are <a href="https://github.com/gradle/gradle/issues/10443">https://github.com/gradle/gradle/issues/10443</a> and <a href="https://github.com/gradle/gradle/issues/10182">https://github.com/gradle/gradle/issues/10182</a></p><p>Here’s a set of PRs for Java applications and libraries.</p><p>Accepted:</p><ul><li><a href="https://github.com/walleth/walleth/pull/396">https://github.com/walleth/walleth/pull/396</a></li></ul><p>Declined:</p><ul><li><a href="https://github.com/mvmike/min-cal-widget/pull/44">https://github.com/mvmike/min-cal-widget/pull/44</a> “HTTPS is enough”</li><li><a href="https://github.com/openhab/openhab-android/pull/1532">https://github.com/openhab/openhab-android/pull/1532</a> “Dependabot would stop working”</li></ul><p>Open (not commented by maintainers):</p><ul><li><a href="https://github.com/apache/jmeter/pull/488">https://github.com/apache/jmeter/pull/488</a></li><li><a href="https://github.com/ben-manes/caffeine/pull/342">https://github.com/ben-manes/caffeine/pull/342</a></li><li><a href="https://github.com/bisq-network/bisq/pull/3051">https://github.com/bisq-network/bisq/pull/3051</a></li><li><a href="https://github.com/signalapp/Signal-Android/pull/9022">https://github.com/signalapp/Signal-Android/pull/9022</a></li><li><a href="https://github.com/trezor/trezor-android/pull/53">https://github.com/trezor/trezor-android/pull/53</a></li><li><a href="https://gitlab.com/fdroid/fdroidclient/merge_requests/837">https://gitlab.com/fdroid/fdroidclient/merge_requests/837</a></li><li><a href="https://github.com/bitcoinj/bitcoinj/pull/1931">https://github.com/bitcoinj/bitcoinj/pull/1931</a></li><li><a href="https://github.com/open-keychain/open-keychain/pull/2488">https://github.com/open-keychain/open-keychain/pull/2488</a></li><li><a href="https://github.com/apereo/cas/pull/4256">https://github.com/apereo/cas/pull/4256</a></li><li><a href="https://github.com/spring-projects/spring-framework/pull/23615">https://github.com/spring-projects/spring-framework/pull/23615</a></li><li><a href="https://github.com/spotbugs/spotbugs/pull/1010">https://github.com/spotbugs/spotbugs/pull/1010</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=582e76207019" width="1" height="1" alt=""><hr><p><a href="https://medium.com/netcracker/dependency-verification-checksum-vs-pgp-582e76207019">Dependency verification: checksum vs PGP</a> was originally published in <a href="https://medium.com/netcracker">netcracker</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>