<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hristofor Pamyatnih on Medium]]></title>
        <description><![CDATA[Stories by Hristofor Pamyatnih on Medium]]></description>
        <link>https://medium.com/@pamyatnih?source=rss-baa1a0aff10e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*I_510tIkg2gPeUwi.</url>
            <title>Stories by Hristofor Pamyatnih on Medium</title>
            <link>https://medium.com/@pamyatnih?source=rss-baa1a0aff10e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:06:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pamyatnih/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Yet another tutorial for Jenkins in Kubernetes.]]></title>
            <link>https://pamyatnih.medium.com/yet-another-tutorial-for-jenkins-in-kubernetes-4d0d7c3cc89?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d0d7c3cc89</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Wed, 31 May 2023 12:22:24 GMT</pubDate>
            <atom:updated>2023-05-31T12:22:24.857Z</atom:updated>
            <content:encoded><![CDATA[<p>A couple of weeks ago I decided to make a POC for the Jenkins server, running in k8s just to clear some concepts on our automation project. I found several tutorials on different sites, which were almost copy-paste from the official documentation of the Jenkins plugin and I was not quite surprised when I found that it is simply not working. Well, the part with the installation of k8s, Jenkins configuration, and so on was correct, but when I tried to configure the pods, it was simply not working.</p><p>There were two misleading points in all of the tutorials — the pod template name and the command to execute.</p><p>But let us start with the pod templates. There is Jenkins agent template in the Dockerhub, but in my case, I don’t want the default OS, provided with the agent. So here is my Dockerfile for OpenSUSE:</p><pre>FROM opensuse/leap<br><br>ARG user=jenkins<br>ARG group=jenkins<br>ARG uid=1000<br>ARG gid=1000<br><br>ENV LANG=&#39;en_US.UTF-8&#39; LANGUAGE=&#39;en_US:en&#39; LC_ALL=&#39;en_US.UTF-8&#39;<br>ENV TZ=Etc/UTC<br>ENV JENKINS_SRV=&#39;jenkins_host:port&#39;<br><br>RUN zypper ar http://adoptopenjdk.jfrog.io/adoptopenjdk/rpm/opensuse/15.0/x86_64 adoptopenjdk &amp;&amp; \<br>    zypper --gpg-auto-import-keys ar https://releases.jfrog.io/artifactory/jfrog-rpms jfrog &amp;&amp; \<br>    zypper --gpg-auto-import-keys in -y java-11-openjdk java-11-openjdk-devel maven git rpm-build wget openssl gnu_parallel&amp;&amp; \<br>    groupadd -g &quot;${gid}&quot; &quot;${group}&quot; &amp;&amp;\<br>    useradd -m -u &quot;${uid}&quot; -g &quot;${group}&quot; &quot;${user}&quot; &amp;&amp;\<br>    mkdir -p /usr/share/jenkins &amp;&amp;\<br>    wget http://${JENKINS_SRV}/jnlpJars/agent.jar -O /usr/share/jenkins/agent.jar &amp;&amp;\<br>    chown ${user}:${group} /usr/share/jenkins/agent.jar &amp;&amp; \<br>    chmod 0644 /usr/share/jenkins/agent.jar &amp;&amp;\<br>    ln -sf /usr/share/jenkins/agent.jar /usr/share/jenkins/slave.jar<br><br>COPY jenkins-agent /usr/local/bin/jenkins-agent<br><br>RUN chmod +x /usr/local/bin/jenkins-agent &amp;&amp;\<br>    ln -s /usr/local/bin/jenkins-agent /usr/local/bin/jenkins-slave<br><br>USER &quot;${user}&quot;<br>ARG AGENT_WORKDIR=/home/&quot;${user}&quot;/agent<br>ENV AGENT_WORKDIR=&quot;${AGENT_WORKDIR}&quot;<br>RUN mkdir /home/&quot;${user}&quot;/.jenkins &amp;&amp; mkdir -p &quot;${AGENT_WORKDIR}&quot;<br><br>VOLUME /home/&quot;${user}&quot;/.jenkins<br>VOLUME &quot;${AGENT_WORKDIR}&quot;<br>WORKDIR /home/&quot;${user}&quot;<br>ENV user=${user}<br><br>ENTRYPOINT [&quot;/usr/local/bin/jenkins-agent&quot;]</pre><p>This is the way how I produce OpenSUSE image with some dev tools installed, also I removed some company-related steps. The script jenkins-agent can be downloaded from Jenkins GitHub repositories. It is executing</p><pre> exec $JAVA_BIN $JAVA_OPTIONS -cp /usr/share/jenkins/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $WEB_SOCKET $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME &quot;$@&quot;</pre><p>When the entrypoint is added to the Docker image, the default options that are pre-populated in the Jenkins k8s plugin settings and also shown in all of the tutorials should be removed</p><p>Wrong (widely distributed in the tutorials):</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*1s9wNulD6acNgAM2.png" /></figure><p>Correct:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CfZDQvdTdzoUoQzsGijM2g.png" /></figure><p><strong>Important! </strong>If the container name is not set to “jnlp,” it will refuse to start.</p><p>With these small alterations, the setup seemed to work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d0d7c3cc89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Windows ssl error 2146893017]]></title>
            <link>https://pamyatnih.medium.com/windows-ssl-error-2146893017-1afa30877c2b?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1afa30877c2b</guid>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[microsoft-error-code]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 15:20:10 GMT</pubDate>
            <atom:updated>2021-09-20T15:24:44.909Z</atom:updated>
            <content:encoded><![CDATA[<p>Didn’t find any useful information in the Internet, so I’ll leave it here.</p><p>I was running security scan with Appspider and application failed with error code -2146893017 Quick search didn’t find explanation so the reason for the error is that remote server is <strong>rejecting your client certificate</strong>. Same error thrown by curl:</p><pre><strong>➜  ~</strong> curl -k --cert auth.pfx --cert-type p12 --pass pass https://host -vvv</pre><pre>*   Trying host...</pre><pre>* TCP_NODELAY set</pre><pre>* Connected to host (host) port 444 (#0)</pre><pre>* ALPN, offering h2</pre><pre>* ALPN, offering http/1.1</pre><pre>* successfully set certificate verify locations:</pre><pre>*   CAfile: /etc/ssl/cert.pem</pre><pre>CApath: none</pre><pre>* TLSv1.2 (OUT), TLS handshake, Client hello (1):</pre><pre>* TLSv1.2 (IN), TLS handshake, Server hello (2):</pre><pre>* TLSv1.2 (IN), TLS handshake, Certificate (11):</pre><pre>* TLSv1.2 (IN), TLS handshake, Server key exchange (12):</pre><pre>* TLSv1.2 (IN), TLS handshake, Request CERT (13):</pre><pre>* TLSv1.2 (IN), TLS handshake, Server finished (14):</pre><pre>* TLSv1.2 (OUT), TLS handshake, Certificate (11):</pre><pre>* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):</pre><pre>* TLSv1.2 (OUT), TLS handshake, CERT verify (15):</pre><pre>* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):</pre><pre>* TLSv1.2 (OUT), TLS handshake, Finished (20):</pre><pre>* TLSv1.2 (IN), TLS alert, certificate unknown (558):</pre><pre>* error:1401E416:SSL routines:CONNECT_CR_FINISHED:sslv3 alert certificate unknown</pre><pre>* Closing connection 0</pre><pre>curl: (35) error:1401E416:SSL routines:CONNECT_CR_FINISHED:sslv3 alert certificate unknown</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1afa30877c2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Maven release for 2 or more OS]]></title>
            <link>https://pamyatnih.medium.com/building-maven-release-for-2-or-more-os-7b6ebdc464f4?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b6ebdc464f4</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[multiproject-build]]></category>
            <category><![CDATA[maven]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Wed, 21 Apr 2021 17:44:36 GMT</pubDate>
            <atom:updated>2021-04-21T17:56:59.281Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Maven release for two or more OS</h3><p>Maintaining product with a long history is not an easy work. And definitely we have really long history. In the glorious time of concurring the world of software industry, when programming was the job of the toughest man and women, our company had an impressing portfolio of supported operating systems. Browsing the sources of native components, I found the following comment in the Makefile</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8VsjKmrEQQujWG1jeK-ASg.png" /></figure><p>Impressing list, but the only one which is supported 25 years later is Linux. And also we added MS Windows. And… we switched to little bit more recent technologies.</p><p>After all this years only very small pieces of native code was left, the core product was rewritten in Java, but we still had a dozen C files. And the binaries produced by this files was consumed by product build as artifact and everything was great until the day when small change was introduced. And I found that I can’t produce release version of binaries, but only snapshot. The reason — same code was built on 2 Jenkins nodes on different OS and it was producing artifacts with same name and groupId, but different classifier. At my first glance this should not be a problem, but the artifactory where the releases was published was not on the same opinion. Oh, the overwrite of releases is forbidden and we are publishing pom files. So all release plugins was kicked off the list of possible solutions. Another constraint — we have pretty old Jenkins, which can’t be upgraded at this moment, because we are using some corporate plugins and here is the place when I’m recalling the Catch 22 — you can’t upgrade Jenkins because you have outdated plugins, so you can’t update plugins, because you have outdated Jenkins. So forget about Jenkins plugins.</p><p>Damn. Looks like dead end.</p><p>You can split the project into Windows and Linux one. But then you will need to maintain two repos.</p><p>Well, what is the release procedure? In short — you are checking you are not using snapshot dependencies, you are tagging the code in the SCM and than you are building the artifacts from this tag. So how to workaround this? I didn’t found smarter way than tag the SCM with temporary tag, build the snapshot, if the build is successful rename temporary tag to the release one and than publish the artefacts. But we can’t install Jenkins plugins, remember? So here is how it happens.</p><p>Checkout and tag:</p><pre>stage (‘Tag’){<br> node (“centos-7-x64”) {<br>   stage (‘Checkout’) {<br>     cleanWs()<br>     checkout([$class: ‘GitSCM’, <br>     branches: [[name: ‘*/master’]], <br>     userRemoteConfigs: [[url: ‘<a href="https://git-cfius.ecd.axway.int/securetransport/securetransport-native.git&#39;">https://gitserver/project.git&#39;</a>, credentialsId: ‘3d4814fe-bdf1–4e08–909a-cda9d0557461’]]<br> ])<br>     withCredentials([usernamePassword (credentialsId: ‘3d4814fe-bdf1–4e08–909a-cda9d0557461’, usernameVariable: ‘GIT_USERNAME’, passwordVariable: ‘GIT_PASSWORD’)]) { <br>       sh ‘’’if [ ! -d .git ]; then mkdir .git; fi<br>       git config — local http.sslVerify “false”<br>       git config — local credential.username builder<br>       git config — local credential.helper “!f() { echo username=\\$GIT_USERNAME; echo password=\\$GIT_PASSWORD; }; f”<br>       git tag ${GITTAG}-op<br>       git push origin ${GITTAG}-op<br>       git config — local — remove-section credential’’’<br>       } <br>     }<br>  }<br>}</pre><p>Why?</p><ul><li>Jenkins git plugin does not work out fine with tags.</li><li>You should create .git directory before repository checkout if you want to save local settings.</li><li>We don’t use ssh key, because we don’t want permanent credential storage in “plaintext”</li></ul><p>Next step build on Linux and Windows:</p><pre>stage(&#39;Build&#39;) {<br>    parallel (<br>        &#39;Linux&#39;: {<br>            node (&quot;centos&quot;) {<br>                cleanWs()</pre><pre>checkout([$class: &#39;GitSCM&#39;, <br>                branches: [[name: &#39;refs/tags/${GITTAG}-op&#39;]], <br>                userRemoteConfigs: [[url: &#39;<a href="https://git-cfius.ecd.axway.int/securetransport/securetransport-native.git&#39;">https://gitserver/project.git&#39;</a>, credentialsId: &#39;3d4814fe-bdf1-4e08-909a-cda9d0557461&#39;]]<br>                ])<br>            configFileProvider([configFile(fileId: &#39;org.jenkinsci.plugins.configfiles.maven.GlobalMavenSettingsConfig1461312644881&#39;, variable: &#39;MAVEN_SETTINGS_XML&#39;)]) {<br>                    sh &#39;&#39;&#39;mvn --batch-mode -s $MAVEN_SETTINGS_XML versions:set -DnewVersion=${GITTAG}-SNAPSHOT -DprocessAllModules -DgenerateBackupPoms=false<br>                    mvn --batch-mode -s $MAVEN_SETTINGS_XML -U -X clean deploy&#39;&#39;&#39;<br>                }<br>            }<br>        },<br>        <br>        &#39;Windows&#39;: {<br>            node (&quot;windows&quot;) {<br>                cleanWs()</pre><pre>checkout([$class: &#39;GitSCM&#39;, <br>                branches: [[name: &#39;refs/tags/${GITTAG}-op&#39;]], <br>                userRemoteConfigs: [[url: &#39;<a href="https://git-cfius.ecd.axway.int/securetransport/securetransport-native.git&#39;">https://gitserver/project.git&#39;</a>, credentialsId: &#39;3d4814fe-bdf1-4e08-909a-cda9d0557461&#39;]]<br>                ])</pre><pre>configFileProvider([configFile(fileId: &#39;org.jenkinsci.plugins.configfiles.maven.GlobalMavenSettingsConfig1461312644881&#39;, variable: &#39;MAVEN_SETTINGS_XML&#39;)]) {<br>                <br>                        sh &#39;&#39;&#39;mvn --batch-mode -s $MAVEN_SETTINGS_XML versions:set -DnewVersion=${GITTAG}-SNAPSHOT -DprocessAllModules -DgenerateBackupPoms=false<br>                        mvn  --batch-mode -s $MAVEN_SETTINGS_XML -U -X clean deploy&#39;&#39;&#39;<br>                }<br>                <br>            }<br>        }<br>    )   <br>}</pre><p>What happens here?</p><p>Linux build is always faster than Windows one. We have same codebase written in C but slightly different build profiles which are distinguished by OS profile in pom.xml:</p><pre>&lt;profiles&gt;<br>  &lt;profile&gt;<br>  &lt;id&gt;windows&lt;/id&gt;<br>  &lt;activation&gt;<br>    &lt;activeByDefault&gt;true&lt;/activeByDefault&gt;<br>    &lt;os&gt;<br>    &lt;family&gt;Windows&lt;/family&gt;<br>  &lt;/os&gt;<br>  &lt;/activation&gt;<br>  &lt;build&gt;....</pre><p>And here we have splendid legacy structure — all product history in one build procedure — maven is executing ant, ant is executing some ancient shell scripts, shell scripts are executing some configurations steps and than make, and finally make is executing compiler and linker and so on.</p><p>Both builds are producing artifacts with same group and artifact id, but different descriptors. The version of artifacts is ${GITTAG}-SNAPSHOT</p><p>Even if each artifact has unique descriptor, project pom files are uploaded to repository, and overwrite in release repository is forbidden, so first we should publish them as snapshot.</p><p>And final step — the ugly hack to publish the artifacts.</p><pre>stage(&#39;Release&#39;) {<br>    node (&quot;centos&quot;) {<br>        cleanWs()<br>        checkout([$class: &#39;GitSCM&#39;, <br>            branches: [[name: &#39;refs/tags/${GITTAG}-op&#39;]], <br>            userRemoteConfigs: [[url: &#39;<a href="https://git-cfius.ecd.axway.int/securetransport/securetransport-native.git&#39;">https://gitserver/project.git&#39;</a>, credentialsId: &#39;3d4814fe-bdf1-4e08-909a-cda9d0557461&#39;]]<br>            ])<br>        configFileProvider([configFile(fileId: &#39;org.jenkinsci.plugins.configfiles.maven.GlobalMavenSettingsConfig1461312644881&#39;, variable: &#39;MAVEN_SETTINGS_XML&#39;)]) {<br>                <br>            sh &#39;&#39;&#39;cd release-project<br>                mvn --batch-mode -s $MAVEN_SETTINGS_XML -Drel.version=${GITTAG}-SNAPSHOT clean install -U -X -Drelease.number=${GITTAG}-SNAPSHOT<br>                mvn  -B -s $MAVEN_SETTINGS_XML -X -e  deploy:deploy-file -DartifactId=artifact_name \<br>                -Dversion=${GITTAG} \<br>                -Dpackaging=zip \<br>                -Dfile=target/artifacts/artifact-linux-x86.zip \<br>                -Dfiles=target/artifacts/native-utils-windows.zip,target/artifacts/native-utils-other.zip,target/artifacts/native-utils-another.zip \<br>                -Dtypes=zip,zip,zip \<br>                -Dclassifier=linux-x86 \<br>                -Dclassifiers=windows,other,another \<br>                -DrepositoryId=repo-delivery-releases \<br>                -DgroupId=org.company.native \<br>                -Durl=<a href="http://repository11.ecd.axway.int/content/repositories/com.axway.securetransport-release/">http://repository/content/repositories/org.company-release/</a><a href="http://repository11.ecd.axway.int/content/repositories/com.axway.securetransport-release/&#39;&#39;&#39;">&#39;&#39;&#39;</a></pre><pre>                withCredentials([usernamePassword (credentialsId: &#39;3d4814fe-bdf1-4e08-909a-cda9d0557461&#39;, usernameVariable: &#39;GIT_USERNAME&#39;, passwordVariable: &#39;GIT_PASSWORD&#39;)]) {</pre><pre>sh &#39;&#39;&#39;git config --local http.sslVerify &quot;false&quot;<br>                    git config --local credential.username builder<br>                    git config --local credential.helper &quot;!f() { echo username=\\$GIT_USERNAME; echo password=\\$GIT_PASSWORD; }; f&quot;<br>                    git tag ${GITTAG} ${GITTAG}-op<br>                    git tag -d ${GITTAG}-op<br>                    git push origin ${GITTAG} :${GITTAG}-op<br>                    echo &quot;Done&quot;<br>                &#39;&#39;&#39;<br>            }<br>        }<br>    }<br>}</pre><p>What we have here?</p><p>First we checkout the sources tagged with ${GITTAG}-op</p><p>Than we build release project which is only downloading all needed artifacts with version ${GITTAG}-SNAPSHOT</p><p>Than we deploy this artifacts as release version and finally we rename tag from ${GITTAG}-op to ${GITTAG}</p><p>Why this way?</p><p>A long time ago our product was available on 5 different OS platforms. managing 5 different repositories with native code for product, which is 99% written in Java was not a good idea. And the changes in the native code was really rare. So we have just one version of native artifacts and everybody is afraid of touching it.</p><p>Why this is not a good idea?</p><p>Mostly because it is ugly hack and it is stopping you from making any optimisations — it is working, so don’t touch it. In my opinion we may have same codebase, but two different maven projects for two different OS. This is a good beginning of a long way to the bight future i.e. get rid of Cygwin. I had thousands of reasons to do it, but no blessing from the management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b6ebdc464f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Download patterns with zypper]]></title>
            <link>https://pamyatnih.medium.com/download-patterns-with-zypper-6f303c9b2424?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f303c9b2424</guid>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[suse]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[zypper]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Fri, 05 Feb 2021 16:06:03 GMT</pubDate>
            <atom:updated>2021-02-05T16:06:03.546Z</atom:updated>
            <content:encoded><![CDATA[<p>This days I had to play with SuSe Linux a while. Zypper is very cool package manager but of course it can’t do everything you want from it. So I hit the issue with downloading the whole software pattern. Following the logic of install and info options i tried</p><pre>zypper download -t pattern Minimal</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*x6VZdJv1lThMBjzU1mOoMQ.jpeg" /></figure><p>The idea was too good to be true. Let’s play with some bash scripting.</p><pre>zypper info -t pattern</pre><p>returns the list of packages and we need only required ones. So the first guess is</p><pre>zypper info -t pattern Minimal | grep Required | awk -F &#39;|&#39; &#39;{print $2}&#39; | sed &#39;s/ //g&#39; | xargs zypper --pkg-cache-dir /tmp download</pre><p>Unfortunately you can;t run more than one instance of zypper so the single line command turned into shell script:</p><pre>for PKG in $(zypper info -t pattern $1 | grep Required | awk -F &#39;|&#39; &#39;{print $2}&#39; | sed &#39;s/ //g&#39;); do <br>        zypper --pkg-cache-dir /tmp download $PKG <br>done</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f303c9b2424" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use AWS s3 bucket for zypper rpm repository.]]></title>
            <link>https://pamyatnih.medium.com/how-to-use-aws-s3-bucket-for-zypper-rpm-repository-3d8947c7c22d?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d8947c7c22d</guid>
            <category><![CDATA[rpm]]></category>
            <category><![CDATA[zypper]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Wed, 16 Dec 2020 11:00:02 GMT</pubDate>
            <atom:updated>2020-12-16T11:00:02.715Z</atom:updated>
            <content:encoded><![CDATA[<p>Yesterday I was surprised by “Do you remember how we talk about expiration of our contract with CDN provider at the end of the year? You said couple of months ago, you will move the repository with updates to Amazon”</p><p>Of course I forgot. There is no way to remember such thing more than couple of weeks.</p><p>Hopefully, I had s3 bucket with similar, but much smaller repository and I decided to modify the upload script and just to add one more repo. I modified slightly the script, ran Jenkins job, I saw that the files are uploaded and decided — everything is OK. Few hours later, when I was checking article about repository relocation I tried to run updates on test environment… And I was very unpleasantly surprised that update failed ‘coz some files was not found. Closer look showed that the reason is “+” in the file name. Running of</p><pre>aws s3 ls s3://bucket</pre><p>Returned correct filenames.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Vzd4JFDyrmzjpresGJgU0A.png" /></figure><p>Great, I had 100+ files with “+” sign in their names. As result of quick search I found that I’m not the only one who hit this issue, but it was fixed in OpenSuSe as option to use S3 bucket as rpm repository, which was not great, because I needed to stick to http(s) protocol. So I made something which looked really stupid on the first glance, but it worked:</p><pre>#!/bin/bash <br>if [ -z $1 ]; then <br>        echo &quot;Missing repository path&quot; <br>        exit 1 <br>fi <br>removeplus() { <br>        WorkDir=$1 <br>        cd $WorkDir <br>        for FILE in $(ls | grep +); <br>                do <br>                        NewFile=$(echo $FILE | sed -e &#39;s/\+/-plus-/g&#39;) <br>                        mv $FILE $NewFile <br>                done <br>        cd - <br>} <br> <br>for REPO in $@; do <br>        REPODIR=$(realpath $REPO) <br>        KEY_ALIAS=&#39;9DBDD9E8&#39; <br> <br>        cp -a /srv/custom/* ${REPODIR}/ <br>        removeplus ${REPODIR} <br>        createrepo $REPODIR <br> <br>        gpg --no-tty --default-key=$KEY_ALIAS --yes --output ${REPODIR}/repodata/repomd.xml.asc --detach-sign ${REPODIR}/repodata/repomd.xml <br>        cp ~/pub.gpg ${REPODIR}/repodata/repomd.xml.key <br>        cd ${REPODIR}/.. <br>        echo &quot;&lt;html&gt;&lt;/html&gt;&quot; &gt; ${REPODIR}/index.html <br>        echo &quot;&lt;html&gt;&lt;/html&gt;&quot; &gt; ${REPODIR}/repodata/index.html <br>done <br>cd /srv/updates <br>/usr/local/bin/aws s3 sync --delete . s3://bucketname</pre><p>The script should be ran with paths to directories containing rpm packages. Each directory in the parameter list creates repository.</p><p>Simply rename of “+” to “-plus-” did the job. I was really surprised, because usually filenames are taken from rpm metadata, but this hack is working.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d8947c7c22d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multipath mess]]></title>
            <link>https://pamyatnih.medium.com/multipath-mess-6d3c2ab82db4?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d3c2ab82db4</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[systemd]]></category>
            <category><![CDATA[multipath]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 10:11:33 GMT</pubDate>
            <atom:updated>2020-09-09T10:11:33.749Z</atom:updated>
            <content:encoded><![CDATA[<p>A week ago we s-started tests of our product on SLES 12 SP5. Everything went perfect until we decided to setup SLEHA with SAN storage. I followed the instructions, rebooted the OS and than OS refuses to boot. Message</p><blockquote>a start job is running for root.device</blockquote><p>appeared on boot screen and than emergency mode prompt.</p><p>In the previous iterations of the testing we used virtual machines and everything went just fine. So I try to reproduce the issue on VM and I was quite surprised when everything passes as expected. Also it was impossible to mount the device manually.</p><p>Lets dig.</p><p>In the begginning of my research I didn’t establish the relation between multipapth and the accident. I begun to research why boot and swap partitions are not mounted.</p><p>I was really surprised when I understood how filesystem mounts are processed in by systemd. In short — fstab is translated into systemd units by one of the systemd components and filesystems are mounted when this units are executed. If SYSTEMD_READY=0 is set for the device in the udev database you can’t mount it. I found another annoying defect — when you try to execute mount /dev/sda1 /boot, nothing happens. The exit code is 0, no message is printed, but the partition is not mounted. Why? Because when this property is set to zero, systemd is umnounting the partition immediately after mount.</p><p>So what happens? Damn corner case. Due to some defect, multipath daemon is setting SYSTEMD_READY=0 for all disk drives. Since we are not using LVM and we are not accessing disk devices through device mapper, we can’t mount them. Solution — create /etc/multipath/multipath.conf and blacklist the device by wwid or by name. In my case it was:</p><blockquote>blacklist {<br> devnode “^sda[0–9]*”<br>}</blockquote><p>After I found the reason why it happened, I also found <a href="https://www.suse.com/support/kb/doc/?id=000019607">Suse KB article</a> on the subject. Sometimes google result sorting algorithm is playing nice tricks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d3c2ab82db4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Be careful with grep exit codes.]]></title>
            <link>https://pamyatnih.medium.com/be-careful-with-grep-exit-codes-1b49a18324b2?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b49a18324b2</guid>
            <category><![CDATA[grep]]></category>
            <category><![CDATA[exit-code]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[bash]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Thu, 28 May 2020 19:00:20 GMT</pubDate>
            <atom:updated>2020-05-28T19:01:50.897Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*EKdxZmp5MF9ST1tF5AT4MQ.png" /></figure><p>Be careful with grep exit codes.</p><p>By historical reasons, our product is shipped also as virtual appliance, based on SLES. So in order to be compliant to OS, the product is packaged as RPM package. Fine, but this product was never meant to be packaged as RPM, so it comes with Java based installer and some shell scripts. It is pretty standard installer — you are running some script, the script is unpacking product distribution and JRE into temp directory, whatever.</p><p>But when we are installing RPM, we don’t have the benefits of interviewer so we need JRE to make initial configuration. In fact, the product itself and installer are two different products with two different JREs. I decided not to carry third JRE only for initial configuration, so I added to posstinstall section of spec file the following:</p><blockquote><strong>for</strong> F <strong>in</strong> /usr/platform/installer/Java/linux-x86/jre11*.zip; <strong>do</strong> /usr/bin/unzip $F -d /usr/platform/java; <strong>done</strong></blockquote><p>It was working until the moment when somebody packaged Java artifact on his Windows workstation and stripped permission flags from all files in this zip archive. To avoid the issue I added the following line:</p><blockquote><strong>for</strong> f <strong>in</strong> $(find /usr/platform/java -type f); <strong>do</strong> file $f | grep -q ELF &amp;&amp; chmod +x $f; <strong>done</strong></blockquote><p>Looks good, isn’t? Nice and elegant way to add X flag to all ELF files, not just make chmod -R +x jre/bin/*</p><p>And it was working until next java update, where last file returned by find was not ELF. And when the last file is not matching pattern of grep, the exit code of the cycle is 1 and rpm installation fails as:</p><blockquote>warning: %post(platform-common-5.5–20200527.112.i686) scriptlet failed, exit status 1</blockquote><p>And it was just yet another dirty hack to fix the product in 5 minutes.</p><blockquote><strong>for</strong> f <strong>in</strong> $(find /usr/platform/java -type f); <strong>do</strong> file $f | grep -q ELF &amp;&amp; chmod +x $f; <strong>true</strong>; <strong>done</strong></blockquote><p>I’m not happy with this, but sometimes you have to put something in “let’s make it better in the future” list which will never be executed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b49a18324b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Failed FTPs and HTTPS to Java connections to Java services after kernel upgrade.]]></title>
            <link>https://pamyatnih.medium.com/failed-ftps-and-https-to-java-connections-to-java-services-after-kernel-upgrade-78bd9d121ecc?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/78bd9d121ecc</guid>
            <category><![CDATA[cve-2019-11477]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[kernel]]></category>
            <category><![CDATA[tcp-sack]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Thu, 18 Jul 2019 11:48:39 GMT</pubDate>
            <atom:updated>2019-07-18T11:48:39.411Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xiYx7YjiiTPBQSzxk_1z-Q.png" /></figure><p>Just another Friday detective story.</p><p>Couple of weeks ago all major Linux vendors released fix for MDS vulnerability in the kernel — CVE-2018–12126, CVE-2018–12130, CVE-2018–12127 and CVE-2019–11091. Two weeks after release of kernel 3.10.0–957.21.3 by RedHat and CentOS, our customer logged ticket that after the kernel update, FTPs transfers become to fail. In the first ticket the version of the kernel was wrong, I made sanity check on same OS/kernel as reported and closed the ticket as not reproducible. Also I said “There is 90% possibility that the issue is caused by blocking random number generator, because the transfers was stalled, but not failed and it smells like lack of entropy in the system”</p><p>On the next day my colleague from support call me and told me — they have issue with latest version of the kernel, and we have environment with reproducible issue. I installed latest kernel, reboot the machine and voila — I was not able to log in with FTPS client.</p><p>Just to note — our product is Java based MFT solution supporting various protocols and there is Apache FTP server behind our FTPs implementation.</p><p>I made a copy of product instance on another machine with previous version of kernel and tested it. Everything was fine. So the problem was in the kernel. Java application stops working after kernel upgrade? Are you kidding me?</p><p>Comparison between traffics dumps on both machine showed that server is not sending certificate bundle during handshake. The frame was just lost. No error messages, no logs, no exceptions, nothing. I checked the release notes of the kernel — the only changes was:</p><pre>Bugs fixed (<a href="https://bugzilla.redhat.com/">https://bugzilla.redhat.com/</a>):<br><br>1719123 - CVE-2019-11477 Kernel: tcp: integer overflow while processing SACK blocks allows remote denial of service<br>1719128 - CVE-2019-11478 Kernel: tcp: excessive resource consumption while processing SACK blocks allows remote denial of service<br>1719129 - CVE-2019-11479 Kernel: tcp: excessive resource consumption for TCP connections with low MSS allows remote denial of service</pre><p>So I become to dig in the code changes. Nothing suspicious, but I’m definitely not good in kernel development. After some shooting in the dark I found the following counter in the output of netstat -s:</p><blockquote>TCPWqueueTooBig: 776</blockquote><p>Well it does not looks familiar. Why not to dig in this direction? And here is what I found:</p><pre>TCP allows an application to queue up to sk_sndbuf bytes,<br>so we need to give some allowance for non malicious splitting<br>of retransmit queue.<br><br>A new SNMP counter is added to monitor how many times TCP<br>did not allow to split an skb if the allowance was exceeded.</pre><p>And the code change:</p><pre>+	if (unlikely((sk-&gt;sk_wmem_queued &gt;&gt; 1) &gt; sk-&gt;sk_sndbuf)) {<br>+		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPWQUEUETOOBIG);<br>+		return -ENOMEM;<br>+	}</pre><p>Looks like ftp server is trying to put too much data in the socket.</p><p>And let the fun begin.</p><p>I took latest version of Apache ftp server — 1.1.1 copied all the certificates, tried to reproduce the issue, bu I found that the part of the code which is sending certificate bundle to client is our custom code, so no, it can’t be fixed by library upgrade.</p><p>So we have issue with buffer size. Lets check can we play with numbers. After some blind tries I found that increasing of receiveBufferSize from 512 to 4096 solve the problem. Automated test passed without any new issues or even performance degradation, other protocols was not affected, so I forgot about the story for two weeks.</p><p>Two weeks later we received similar bug report from another customer. They upgraded kernel to 2.6.32–754.15.3, it was RHEL 6 in this case and somehow certificate authentication over HTTPS protocol stops to work.</p><p>I was happy, because I received tarball with exact customers setup an just restored it to my VM. First I tested on RHEL 7 with latest kernel. Everything was working as expected — FTPS was hanging, but HTTPS was OK. I built new VM with latest RHEL 6 and copied the instance of the product onto. The big surprise — on RHEL 6 FTP was working, but SSL handshake over HTTPS was hanging after sending of server certificate chain to client side, until session timeout, and than rest of the handshake data was sent. Damn, there was no tunable buffer settings for HTTP daemon exposed. So lets try it the hard way.</p><p><a href="https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt">This page</a> becomes my favorite last days. I knew that the application is trying to put all the data in one TCP frame, but OS was refusing to fragment it. I dig int the settings related to TCP fragmentation, and just when I become desecrated, I had lucky shot:</p><pre>tcp_mtu_probing - INTEGER<br>	Controls TCP Packetization-Layer Path MTU Discovery.  Takes three<br>	values:<br>	  0 - Disabled<br>	  1 - Disabled by default, enabled when an ICMP black hole detected<br>	  2 - Always enabled, use initial MSS of tcp_base_mss.</pre><p>The default setting was zero. Setting it to 2 resolved the issue. The magic command:</p><blockquote>sysctl -w net.ipv4.tcp_mtu_probing=2</blockquote><p>Automation and performance testing lies ahead, but I think we will have just another happy customer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78bd9d121ecc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strange story about Oracle JDBC connector and OpenStack]]></title>
            <link>https://pamyatnih.medium.com/strange-story-about-oracle-jdbc-connector-and-openstack-3332cc1c639f?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3332cc1c639f</guid>
            <category><![CDATA[openstack]]></category>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[jdbc]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Mon, 20 Nov 2017 10:48:20 GMT</pubDate>
            <atom:updated>2017-11-20T10:48:20.627Z</atom:updated>
            <content:encoded><![CDATA[<p>Long time ago we decided to retire our ancient VMware LabManager cloud and migrate to some alternative. Lab Manager was really useful solution, but it was discontinued and the Guys in the High Skies Who Holds the Money says — the new VMware alternatives are too expensive, find something different. That was the short story how one sunny day strange little animal called OpenStack became its life in our servers. It was growing some features and extensions, it was becoming more mature each sprint and finally our little guinea pig begun its youth period — the beta testing by team.</p><p>After some testing we found an issue — it was not able to install our product with Oracle database on RHEL. I decided it was networking issue, realated to DNAT networking and left our DBA to fight the problem. Suddenly I was sent to long-term business trip and I forgot about issue until I returned back.</p><p>When I returned back to my tasks, our DBA told me — “there should not be problem from DB side”, but still, when our product installer was trying to establish test connection to database, the connection was failing with:</p><blockquote>ERROR Unable to connect to database: IO Error: Connection reset<br>java.sql.SQLRecoverableException: IO Error: Connection reset</blockquote><p>The problem with installer was that it was developed by different team, even located in different country, and this installer was discontinued. So it was blackbox for me. Googling shows that this can be caused by blocking /dev/random, but java distributed with the installer was configured to use dev/urandom as source of random seed, so I spent some time digging in wrong direction.</p><p>Finally I found that by some reason /dev/random was hardcoded in JDBC driver as source of random seed. And there was not enough entropy in the virtual machine, so /dev/random was blocking before JDBC driver has enough random bytes to create session’ s shared secret. So why this happens only on OpenStack virtual machines. Short googling gave the answer:</p><blockquote>Linux contains a paravirtualized hardware RNG device, known as virtio-rng which is able to seed the kernel’s random pool,making entropy available via /dev/random. The QEMU/KVM backend for this device is able to feed it with entropy from the host’s /dev/random device, or from a daemon supporting the EGD protocol. QEMU will optionally rate limit the data that the guest can request, to prevent a single guest from exhausting the host’s entropy supply.</blockquote><blockquote>If QEMU’s built-in rate limiting policy is not sufficient, it is a simple matter to create a custom daemon supporting the EGD protocol that controls the flow of entropy to guests based on arbitrary application defined metrics</blockquote><blockquote>Although no implementation currently exists, it would also be possible to provide a virtualized TPM device to feed random entropy to the guest.</blockquote><p>Well, if we have enough available entropy in host OS, it should be available to guests, isn’t it? I installed rngd on host machine and told the daemon to read /dev/urandom and feed /dev/randon — voila! we have enough entropy in host machine. Nope. We don’t. As usual first available quick and dirty hack was not working. Let’s try to think.</p><p><a href="https://centos.pkgs.org/7/epel-x86_64/haveged-1.9.1-1.el7.x86_64.rpm.html">Haveged</a> available from EPEL can give us enough entropy in host machine. We don’t need to make modifications in guest OSes so we need to feed them with enough entropy.</p><p>Then we need to add virtualized hardware random number generator to VM template, we are modifying metadata, but not the OS image:</p><blockquote>glance image-update --property hw_rng_model=virtio [image_id]</blockquote><p>And than we need to change flavor metadata, to enable hardware RNG:</p><blockquote>hw_rng:allowed=True</blockquote><blockquote>hw_rng:rate_bytes — The allowed amount of bytes for the the guest to read from the host’s entropy per period. I left this untouched.</blockquote><blockquote>hw_rng:rate_period — Sets the duration of a read period in seconds. I left this untouched until I see any issues with lack of entropy.</blockquote><p>Congratulations — the number of available entropy was increased from ~150 to over 4K. The installation of our product with Oracle database was successful and I logged defect for hardcoded /dev/random. I hope someday there will be somebody available to fix it.</p><p>Next issue I found was fail of installation, if connection to Oracle is over SSL, so I can say:</p><p><em>To be continued…</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3332cc1c639f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some Ansible tricks for JDK and Docker installation.]]></title>
            <link>https://pamyatnih.medium.com/some-ansible-tricks-for-jdk-and-docker-installation-558fc23ec9ae?source=rss-baa1a0aff10e------2</link>
            <guid isPermaLink="false">https://medium.com/p/558fc23ec9ae</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[ansible]]></category>
            <dc:creator><![CDATA[Hristofor Pamyatnih]]></dc:creator>
            <pubDate>Tue, 20 Jun 2017 07:11:45 GMT</pubDate>
            <atom:updated>2017-06-20T07:12:24.950Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I went in trouble with setting up webex conference from Linux laptop. I resolved the issue with 32 bit Docker image started on my machine, but I realized it is not so easy to install Oracle JDK automatically. Yesterday my co-worker needed a number of machines with installed JDK and Docker. I made Ansible playbook for less than 3 minutes, but I found that Docker is not in standard RHEL 7 repo. If yo want to install Docker you need to enable “extras” repo. But Ansible subscription manager package does not support repository management. So I fallback to good-old shell module. Here are the results:</p><pre>- name: install packages <br>    yum: name={{item}} state=present <br>    with_items: <br>      - unzip                                                                                                                                                                                                   <br>      - wget                                                                                                                                                                                                    <br>                                                                                                                                                                                   <br>  - name: create install dir                                                                                                                                                                                    <br>    file: path=/jdk state=directory                                                                                                                                                                             <br>  - name: download java                                                                                                                                                                                         <br>    shell:  &#39;wget -c --header &quot;Cookie: oraclelicense=accept-securebackup-cookie&quot; http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-i586.tar.gz -O /jdk/jdk<br>-8u131-linux-i586.tar.gz &amp;&amp; cd /jdk &amp;&amp; tar -xzf jdk-8u131-linux-i586.tar.gz &amp;&amp; for f in `ls /jdk/jdk1.8.0_131/bin`; do update-alternatives --install &quot;/usr/bin/${f}&quot; &quot;${f}&quot; &quot;/jdk/jdk1.8.0_131/bin/${f}&quot; 1; done&#39;</pre><p>It looks ugly but I did not have enough time to write it using get_url module, just copy-pasted shell code from Docker file. Next time I’ll use this:</p><pre>- name: get JDK <br>  get_url:    <br>    url: <a href="http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-i586.tar.gz">http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-i586.tar.gz</a><br>    headers: &#39;Cookie:oraclelicense=accept-securebackup-cookie&#39;<br>    dest: /jdk/<a href="http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-i586.tar.gz">jdk-8u131-linux-i586.tar.gz</a><br>- name: unpack JDK<br>  unarchive: src=/jdk/<a href="http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-i586.tar.gz">jdk-8u131-linux-i586.tar.gz</a> dest=/jdk<br>- name: install<br>  shell: &#39;for f in `ls /jdk/jdk1.8.0_131/bin`; do update-alternatives --install &quot;/usr/bin/${f}&quot; &quot;${f}&quot; &quot;/jdk/jdk1.8.0_131/bin/${f}&quot; 1; done&#39;</pre><p>It will work until next JDK update, so it will be nice to use JDK fileaname and url as variables.</p><p>Next obstacle was missing repo management functionality. And next ugly solution was:</p><pre>- name: Add RedHat subscription <br>    redhat_subscription: state=present username={{rhn_username}} password={{rhn_password}} autosubscribe=true <br>  - name: add extras repo <br>    shell: subscription-manager repos --enable rhel-7-server-extras-rpms <br>   <br>  - name: install packages <br>    yum: name={{item}} state=present <br>    with_items: <br>      - unzip <br>      - wget <br>      - docker <br>      - git</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=558fc23ec9ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>