<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexander Chumakin on Medium]]></title>
        <description><![CDATA[Stories by Alexander Chumakin on Medium]]></description>
        <link>https://medium.com/@achumakin?source=rss-82b2093ff818------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oLcNdeN3WpaCbTiVamLNag.jpeg</url>
            <title>Stories by Alexander Chumakin on Medium</title>
            <link>https://medium.com/@achumakin?source=rss-82b2093ff818------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:03:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@achumakin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Continuous performance testing with K6 and Testkube]]></title>
            <link>https://medium.com/@achumakin/continuous-performance-testing-with-k6-and-testkube-58cb7818c27f?source=rss-82b2093ff818------2</link>
            <guid isPermaLink="false">https://medium.com/p/58cb7818c27f</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[testkube]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[k6]]></category>
            <dc:creator><![CDATA[Alexander Chumakin]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 09:24:38 GMT</pubDate>
            <atom:updated>2023-11-20T12:59:37.409Z</atom:updated>
            <content:encoded><![CDATA[<h3>Problem</h3><p>In a rapidly changing world, people are not used to waiting, and fast application performance becomes basic.</p><p>This loading icon makes your customers unhappy, and they might consider using a service of your competitors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*vSnpNNmIPddYsmRc" /></figure><p>To make sure that this problem will not affect your business even in peak hours, we need to test the performance of your application.</p><p>Performance testing becomes an essential part of quality gates in many companies and products.</p><h3>Target application</h3><p>I’ve built a little REST API service that will help us to understand the whole process.</p><p><a href="https://github.com/alexandrchumakin/payments-demo-quarkus-app">GitHub - alexandrchumakin/payments-demo-quarkus-app</a></p><p>All it can do is to create a new payment, get it by id, return a list of all the payments that and delete a payment by its ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1_43XOQau0B-l4zEfshNA.png" /></figure><p>I used Java 17 (as the latest LTS version) and <a href="https://quarkus.io/">Quarkus</a> framework to build the app. The reason why I really like this framework is that it has native support for Docker and <a href="https://kubernetes.io/">Kubernetes</a>. Most of the modern apps are already running on Kubernetes, so I decided to have a real-world example.</p><blockquote>Kubernetes is like a smart manager for your apps. It helps put your programs in little boxes (containers) and decides where to put them so they work well together. It also makes sure your apps can handle more users when needed and takes care of them, so you can focus on making your programs awesome.</blockquote><p>To run the app locally inside a Kubernetes cluster, we can use <a href="https://minikube.sigs.k8s.io/">Minikube</a>. It’s the simplest way to run a cluster locally when we don’t need to worry about multiple nodes (for example, when we test resources provisioning).</p><p>Here Quarkus is a great tool for this purpose as it provides the plugin that can help generate a configuration spec by adding a single plugin to the project.</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.quarkus&lt;/groupId&gt;<br>    &lt;artifactId&gt;quarkus-minikube&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><p>It will generate everything needed to run the application in the Minikube cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/0*fhm-6X6K1GM3w3Ul" /></figure><p>I created a <a href="https://github.com/alexandrchumakin/payments-demo-quarkus-app/blob/main/scripts/start-minikube.sh">little shell script</a> that will do everything for you in one go.</p><p>You can explore the resources using <a href="https://kubernetes.io/docs/reference/kubectl/">kubectl</a> command or <a href="https://k9scli.io/">k9s</a> tool if you like visualisation. The very basic architecture diagram of the app looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*hvuZkks9tNi0wvPpNZrBCw.jpeg" /></figure><p>We have a single entry point to the cluster done via <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> controller. It has a meaningful host that you set in the <a href="https://github.com/alexandrchumakin/payments-demo-quarkus-app/blob/82ca07dace921e5537ccbca09ab5c4894c81629f/src/main/resources/app-ingress.yaml#L8">resource configuration</a>. This controller connects us to the <a href="https://kubernetes.io/docs/concepts/workloads/pods/">Pod</a> where the app is running via <a href="https://kubernetes.io/docs/concepts/services-networking/service/">service</a> layer. Pods can “talk” to each other via services as well. Here application pod connects to the database pod via another service. And the database pod persists the data in our host via <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">Persistent Volume</a>. Most of the components are configured via a single <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">deployment</a> generated by Quarkus. So, when we need to change the app’s version, we just need to build a new version and recreate a deployment. There is a <a href="https://github.com/alexandrchumakin/payments-demo-quarkus-app/blob/main/scripts/re-deploy.sh">special script</a> to do it.</p><p>To access the app via this configured host, you need to add it to /etc/hosts file:</p><pre>127.0.0.1 payments-app.com</pre><p>We can explore the running app via <a href="https://curl.se/">curl</a> command in the terminal:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyoDP-iW_KPc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyoDP-iW_KPc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyoDP-iW_KPc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/65eed6c4f1b8f156e19677c5e15da4e7/href">https://medium.com/media/65eed6c4f1b8f156e19677c5e15da4e7/href</a></iframe><h3>Load tests</h3><p>We’re ready to create performance tests for the app. I used <a href="https://k6.io/">K6</a> tool for this purpose. Over the decade I used many tools for performance testing, and I founded K6 a year ago and believe it is the best choice for performance testing nowadays! It’s incredibly easy to start with, and you write your tests in <a href="https://www.javascript.com/">JavaScript</a> / <a href="https://www.typescriptlang.org/">TypeScript</a>. It doesn’t require strong programming language skills, learning a new tool (that you might never use again after you finish a current project) or making a complex setup of your local environment.</p><p>They also made an excellent documentation that is enough to quickly get started. You don’t need to add extra libraries to <a href="https://k6.io/docs/examples/generating-uuids/">generate</a> some data, make <a href="https://k6.io/docs/using-k6/checks/">assertions</a>, work with <a href="https://k6.io/docs/javascript-api/k6-http/">http client</a> or <a href="https://k6.io/docs/javascript-api/k6-ws/socket/">socket</a>.</p><p>There are different types of performance testing, and K6 allows configuring them via stages. I configured stages to emulate classic load tests with a slowly growing number of users and the same tear-down process.</p><p><a href="https://k6.io/docs/testing-guides/automated-performance-testing/#model-the-scenarios-and-workload">How to Automate Performance Testing: The k6 Guide</a></p><p>When you need to validate the result of performance tests automatically, K6 provides <a href="https://k6.io/docs/examples/tutorials/get-started-with-k6/test-for-performance/#assert-for-performance-with-thresholds">thresholds</a> where you can set expectations from a test run.</p><p>Here you can find my GitHub repository with load tests for the app described above:</p><p><a href="https://github.com/alexandrchumakin/payments-demo-load-tests">GitHub - alexandrchumakin/payments-demo-load-tests</a></p><p>You can run these tests from your terminal by via <a href="https://k6.io/docs/get-started/installation/">CLI</a>, <a href="https://k6.io/docs/get-started/installation/#docker">docker container</a> or from <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> with <a href="https://plugins.jetbrains.com/plugin/16141-k6">K6 plugin</a>. In the example below, you can see how the run looks. I pass a custom <em>BASE_URL</em> here to show how to work with custom arguments in K6 scripts.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fw4YwxaKb2A0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dw4YwxaKb2A0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fw4YwxaKb2A0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b3e8b684344593334c82580bd794cc5/href">https://medium.com/media/3b3e8b684344593334c82580bd794cc5/href</a></iframe><h3>Testkube</h3><p>To really measure the performance of your application, but not the network latency or performance of your hardware, I strongly recommend moving your tests as close to the running app as possible, ideally right inside the same cluster.</p><p>There is a new modern and fantastic tool called <a href="https://testkube.io/">Testkube</a> for this purpose. It’s fully open source and could be installed in any Kubernetes cluster with a single configuration file and one command via <a href="https://docs.testkube.io/articles/helm-chart/">Helm chart</a>.</p><p>Testkube provides a set of tools for testing that could be configured either via Kubernetes resources or via a very handy <a href="https://docs.testkube.io/articles/testkube-dashboard/">dashboard</a>. You can run many different tests there including K6.</p><p><a href="https://docs.testkube.io/category/test-types">Test Types | Testkube Documentation</a></p><p>After we deploy Testkube resources to our cluster, the architecture diagram will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*Vzgj2iKvQ2So-mvNVwZuKg.jpeg" /></figure><p>Remember to add two more hosts to /etc/hosts:</p><pre>127.0.0.1 testkube-api.com<br>127.0.0.1 testkube-dashboard.com</pre><p>All you need is to connect Testkube K6 runner to your tests, e.g. via a Git <a href="https://docs.testkube.io/articles/test-sources/">source</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vFWYUK-5l6GwQ7Wc" /></figure><p>You can configure environment variables right in the Testkube dashboard and trigger test execution:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKgY-KmVDyaU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKgY-KmVDyaU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKgY-KmVDyaU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/10731cfe778ebf68cf9239e3e132feda/href">https://medium.com/media/10731cfe778ebf68cf9239e3e132feda/href</a></iframe><h3>Grafana dashboard</h3><p>The latest part here is a visualisation of the test results. It’s really hard to read results from command line output and even harder to compare it with previous runs. I recommend using <a href="https://grafana.com/">Grafana</a> because of two main reasons:</p><ul><li>Market standard: almost any software engineer has used or at least heard of Grafana</li><li>K6 is a part of <a href="https://k6.io/blog/joining-grafana-labs/">Grafana Labs</a>, and it has a native integration with Grafana dashboards</li></ul><p><a href="https://k6.io/docs/results-output/grafana-dashboards/">Grafana dashboards</a></p><p>To build a dashboard from K6 tests results, we need to build a middle layer where results are stored. One of the easiest ways to do this is to use <a href="https://www.influxdata.com/">InfluxDB</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*2RX4N15Z0EYiqfDQ" /></figure><p>In the test repo, you’ll see <a href="https://github.com/alexandrchumakin/payments-demo-load-tests/blob/main/k8s/grafana-ingress.yaml">examples</a> of all the required Kubernetes resources. You can install all Testkube and Grafana related resources using a <a href="https://github.com/alexandrchumakin/payments-demo-load-tests/blob/main/scripts/start-helm.sh">single shell script</a> that I prepared.</p><p>To access the dashboard, you need to add one more host to /etc/hosts:</p><pre>127.0.0.1 grafana-local.com</pre><p>By default, you can log in to the dashboard using admin/admin credentials, and it will ask you to change the credentials on the first login.</p><p>You should not worry about building the dashboards yourself, but import the existing template. You can also do it via the prepared <a href="https://github.com/alexandrchumakin/payments-demo-load-tests/blob/main/scripts/grafana-dashboard.sh">shell script</a>. It will require providing an admin password for <a href="https://minikube.sigs.k8s.io/docs/commands/tunnel/">Minikube tunnelling</a>.</p><h4>End-to-end infra installation</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAbtB45B63CM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAbtB45B63CM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FAbtB45B63CM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/55dab6bfec92933f3dc3a317b4da2834/href">https://medium.com/media/55dab6bfec92933f3dc3a317b4da2834/href</a></iframe><h3>Continuous testing</h3><p>To make sure your tests are always running for a new version of app, you can configure <a href="https://docs.testkube.io/articles/test-triggers/">Testkube Trigger</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYMLRZMDazGY66IQrljwrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwrqE_8rH5dm_PZbZFzxWw.png" /></figure><p>It’s that simple! You just say WHEN to trigger and WHAT to trigger and Testkube will take care of the rest. I’ve created <a href="https://github.com/alexandrchumakin/payments-demo-quarkus-app/blob/main/scripts/re-deploy.sh">another script</a> that will build a new version of the service and re-create a deployment.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FiAEtlMFhGuc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DiAEtlMFhGuc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FiAEtlMFhGuc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/93484465bcd50273d7ed771689741118/href">https://medium.com/media/93484465bcd50273d7ed771689741118/href</a></iframe><h3>Afterword</h3><p>I really like all the tools above. They have their own pros and cons, but I don’t see any reason not to try it out! I’m open for any feedback and hope it will help you achieve your goals!</p><p>Cheers,</p><p>Alex.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58cb7818c27f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieve GitLab CI artifacts from child pipelines]]></title>
            <link>https://medium.com/@achumakin/retrieve-gitlab-ci-artifacts-from-child-pipelines-f427d1983d87?source=rss-82b2093ff818------2</link>
            <guid isPermaLink="false">https://medium.com/p/f427d1983d87</guid>
            <category><![CDATA[jq]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[curl]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <dc:creator><![CDATA[Alexander Chumakin]]></dc:creator>
            <pubDate>Thu, 13 Oct 2022 17:50:41 GMT</pubDate>
            <atom:updated>2022-10-13T17:50:41.712Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been using GitLab CI for 4 years so far and cannot enjoy it enough, but sometimes I still miss some interesting features that I wish I had out of the box. This story is about one of them.</p><h3>Problem</h3><p>I like how GitLab allows to work with independent pipelines from different git repos. For example when we have a development repo and a separate test repo that needs to reuse artifacts produced by dev code (say compilation output) we can simply do it like this:</p><ul><li>dev project</li></ul><pre>build:<br>  script:<br>    - echo &quot;Build project output&quot;<br>    - ...<br>  artifacts:<br>    paths:<br>      - project/output<br>   <br>trigger-tests:<br>  variables:<br>    ARTIFACTS_DOWNLOAD_REF: $CI_COMMIT_REF_NAME<br>  trigger: <br>    project: path-to-test-project<br>    branch: master</pre><ul><li>test project</li></ul><pre>run-tests:<br>  script:<br>    - echo &quot;check dev project artifacts&quot;<br>    - ls project/output<br>    - echo &quot;running tests&quot;<br>    - ...<br>  needs:<br>    - project: path-to-dev-project<br>      job: build<br>      ref: $ARTIFACTS_DOWNLOAD_REF<br>      artifacts: true</pre><p>That’s all! So we can trigger a test pipeline from dev pipeline and provide some variable like ref name, then we can grab artifacts from another pipeline based on its ref name and run tests against it!</p><p>But let’s imagine a more complex scenario where we create artifacts not in the general pipeline, but in a child downstream pipeline, e.g. for multi-module project where we use <a href="https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html#dynamic-child-pipelines">dynamic child pipelines feature</a> to automatically generate downstream pipelines on the fly based on affected modules.</p><h3>Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*BNRoHaBp8OVd0o5nCDOUfw.jpeg" /><figcaption>desired behaviour</figcaption></figure><p>I don’t want to over complicate things here, so let’s produce a list of two elements where it could be either empty, has one of elements or both of them randomly with a little Python script.</p><pre>import random<br><br>modules = [&#39;project_1&#39;, &#39;project_2&#39;]<br><br>for module in modules:<br>    print(module) if bool(random.getrandbits(1)) else None</pre><p>Now we can simply assign the result of this script to environment variable and emulate “affected modules” functionality in CI.</p><h3>Pipelines configuration</h3><p>I have initially 3 stages: randomly generate <br>“affected modules”, generate dynamic downstream pipeline depending on what I have generated in the previous step, then generate some artifact for dynamic job and run this downstream pipeline.</p><pre>stages:<br>  - build<br>  - generate-downstream<br>  - trigger-downstream<br><br>default:<br>  image: python:3.8-slim<br><br>build-modules:<br>  stage: build<br>  script:<br>    - export modules=$(python3 generate_modules.py)<br>    - echo &quot;MODULES=$modules&quot; &gt;&gt; build.env<br>  artifacts:<br>    reports:<br>      dotenv: build.env<br><br>prepare-pipeline:<br>  stage: generate-downstream<br>  script:<br>    - cd ci &amp;&amp; python generate_ci.py<br>  artifacts:<br>    paths:<br>      - .publish-artifacts.yml<br>  needs:<br>    - job: build-modules<br>      artifacts: true<br><br>trigger-child-pipeline:<br>  stage: trigger-downstream<br>  trigger:<br>    include:<br>      - artifact: .publish-artifacts.yml<br>        job: prepare-pipeline<br>    strategy: depend<br>  needs:<br>    - job: build-modules<br>      artifacts: true<br>    - job: prepare-pipeline<br>      artifacts: true</pre><p>where <a href="https://gitlab.com/alexandrchumakin/retrieve-child-artifacts/-/blob/master/ci/generate_ci.py"><em>generate_ci.py</em></a> is another Python script that generates dynamic downstream pipeline with jobs for affected modules.</p><p>Now we can create a script that will get artifacts from downstream pipeline for existing jobs (if any). I would prefer simple bash script to easily reuse it for other projects with different technologies.</p><pre><strong>#!/bin/bash<br><br></strong>perform_gitlab_call() {<br>  path=$1<br>  result=$(curl --silent --insecure --noproxy &#39;*&#39; --header &quot;PRIVATE-TOKEN:$ACCESS_TOKEN&quot; &quot;$CI_API_V4_URL/projects/$CI_PROJECT_ID/$path&quot;)<br>  echo &quot;$result&quot;<br>}<br><br>download_artifacts() {<br>  build_job_id=$1<br>  if [[ &quot;$build_job_id&quot; == &quot;null&quot; ]]; then<br>    echo &quot;build job id is null&quot;<br>  else<br>    echo &quot;Downloading artifacts from job $build_job_id&quot;<br>    curl --silent --insecure --noproxy &#39;*&#39; -L -o artifacts.zip --header &quot;PRIVATE-TOKEN:$ACCESS_TOKEN&quot; &quot;$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/$build_job_id/artifacts&quot;<br>    unzip -o -qq artifacts.zip<br><br>    echo &quot;Check if artifacts are extracted successfully&quot;<br>    ls *.txt<br>    rm -rf artifacts.zip<br>  fi<br>}<br><br>find_publish_job_id() {<br>  job_name=$1<br>  result=$(perform_gitlab_call &quot;pipelines/$ci_child_pipeline_id/jobs&quot; |<br>    jq --raw-output &#39;[.[]|select(.name==&#39;\&quot;$job_name\&quot;&#39;)][0].id&#39;)<br>  echo $result<br>}<br><br>get_pipelines_data() {<br>  pipeline_id=$1<br>  echo &quot;Trying to get artifacts from pipeline $pipeline_id&quot;<br>  ci_child_pipeline_id=$(perform_gitlab_call &quot;pipelines/$pipeline_id/bridges&quot; |<br>    jq &#39;[.[]|select(.name==&quot;trigger-child-pipeline&quot;)][0].downstream_pipeline.id&#39;)<br>  echo &quot;Found child pipeline id $ci_child_pipeline_id&quot;<br><br>  project1_build_job_id=$(find_publish_job_id publish-project_1)<br>  project2_build_job_id=$(find_publish_job_id publish-project_2)<br>  printf &quot;Project1 child pipeline id: %s\nProject1 child pipeline id: %s\n&quot; &quot;$project1_build_job_id&quot; &quot;$project2_build_job_id&quot;<br><br>  if [[ &quot;$project1_build_job_id&quot; == &quot;null&quot; ]] &amp;&amp; [[ &quot;$project2_build_job_id&quot; == &quot;null&quot; ]]; then<br>    echo &quot;Cannot find any artifacts for project1 or project2 pipelines, try to download it from another pipeline&quot;<br>    echo &quot;Found ${#pipelines_list[@]} pipelines for branch $CI_COMMIT_REF_NAME to scan&quot;<br>    echo &quot;${pipelines_list[@]}&quot;<br>    add_id=${pipelines_list[0]}<br>    if [ -z &quot;$add_id&quot; ]; then return ; else echo &quot;Retrieved additional pipeline id $add_id&quot;; fi<br>    unset pipelines_list[0]<br>    pipelines_list=( ${pipelines_list[@]} )<br>    get_pipelines_data $add_id<br>  else<br>    download_artifacts &quot;$project1_build_job_id&quot;<br>    download_artifacts &quot;$project2_build_job_id&quot;<br>  fi<br>}<br><br>pipelines_list=($(perform_gitlab_call &quot;/pipelines?ref=$CI_COMMIT_REF_NAME&amp;status=success&quot; | jq --raw-output &#39;.[].id | @sh&#39;))<br>get_pipelines_data $CI_PIPELINE_ID</pre><p>All I need to do to configure my project is to prepare ACCESS_TOKEN environment variable in CI that could be personal access token or group/project access token for paid versions of GitLab.</p><p>This script will try to find artifacts for latest pipeline in current branch, if there are no “affected modules” pre-generated, then will keep looking in previous pipelines for the same branch. In the end we will have artifacts in parent pipeline level from either this downstream pipeline jobs or from the previous pipelines. Now we just need to call it from pipeline:</p><pre>stages:<br>  - build<br>  - generate-downstream<br>  - trigger-downstream<br>  - child-artifacts<br><br>...<br><br>retrieve-artifacts:<br>  stage: child-artifacts<br>  image: alexandrchumakin/docker-jq-curl<br>  before_script:<br>    - chmod +x ci/download-artifacts.sh<br>  script:<br>    - ./ci/download-artifacts.sh<br>  artifacts:<br>    paths:<br>      - &#39;*.txt&#39;</pre><p>I only need to use an image with normal bash console (not old shell one), jq and curl, so I prepared one myself and pushed to dockerhub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ACbLJ6KtUvbhIsgPg89t2g.png" /></figure><h3>Conclusion</h3><p>Even being one of the best (if not just the best one) CI/CD tool in a market, GitLab still might miss some cool features, but with using of documentation, some simple API calls and a bit of enthusiasm and creativity you could implement everything you need quickly and nicely =)</p><p>Reach out to my <a href="https://gitlab.com/alexandrchumakin/retrieve-child-artifacts">GitLab repo</a> for full code base.</p><p>Happy CI-ing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f427d1983d87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wait pipelines in GitLab]]></title>
            <link>https://medium.com/@achumakin/wait-pipelines-in-gitlab-6a2f91432e77?source=rss-82b2093ff818------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a2f91432e77</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[shell]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[shell-script]]></category>
            <dc:creator><![CDATA[Alexander Chumakin]]></dc:creator>
            <pubDate>Sun, 26 Jun 2022 09:02:26 GMT</pubDate>
            <atom:updated>2022-08-22T08:50:36.170Z</atom:updated>
            <content:encoded><![CDATA[<p>For a few last years I work a lot with GitLab and can say with certainty that it becomes better and better every release, not like many other tools.</p><h3>One of the best new features</h3><p>For example triggering pipelines: it was only possible via API that is not a rocket science. This approach has a huge disadvantage: you trigger an external pipeline asynchronously and can’t even see if something goes wrong there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ml6rZmPZLlBZCkfs7mi2Mw.jpeg" /><figcaption>trigger external pipeline async</figcaption></figure><p>What if you don’t need just to trigger another pipeline from the current one, but also wait for it to finish and check the status? Oh, it becomes like a monster task to implement then! You need to write a script that will check whether this external pipeline is finished every N seconds and return the final result via intermidiate file (the easiest way coming to my mind). Fortunately, somebody already shared a docker image called <a href="https://gitlab.com/finestructure/pipeline-trigger">pipeline-trigger </a>that can help you with achieving this quite easily! But it was still quite dangerous to use open source image for such approach.</p><p>In the beginning of 2020 GitLab moved <a href="https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html">Multi-project pipelines</a> feature to all tiers and it became so easy just out of the box! This is a great example how other CI/CD platforms should implement it: super straightforward! There are still some little disadvantages comparing to Trigger API, but nice think to always try once needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*XtgrmrYracZ-Pd7TLiM5vA.png" /><figcaption>trigger external project pipeline synchronously</figcaption></figure><h3>A feature that’s still missing</h3><p>Despite the above, there are still some missing features which (in my opinion) should be implemented long time ago. I want to tell you about one of such features that I needed to implement myself.</p><p>Sometimes we use dedicated runner for particular kind of jobs, e.g. for test automation pipelines like in my case. I need to create some docker containers in the runner from a pipeline with specific ports and I cleanup it in the end. So, I cannot run parallel pipelines as I cannot allocate the same port for different docker containers or my cleanup from one pipeline can kill some containers that are in use in a different pipeline. I think, it’s very easy example and quite a lot of people experience the same issues.</p><p>Does GitLab provide us with some workarounds? Of course, one of the first features you might google is called <a href="https://docs.gitlab.com/ee/ci/resource_groups/">resource groups</a>. It allows you to mark jobs with some resource that you want to block not put all the others jobs in queue until this temp resource is available. But it can only be used in stage level, not in pipeline level, so it doesn’t help in my case. You can see that <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/208792">such issue</a> created years ago and still open.</p><p>In this simple example you can see that jobs marked with the same resource group will be running one by one, but it doesn’t guarantee that first you’ll have all jobs from the 1st pipeline finished before the next one is started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D46hJ1BVAmHSHVrHh_jBXg.png" /><figcaption>Resource group cannot be used in pipeline level</figcaption></figure><p>The next <a href="https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section">solution</a> that you can find is to go to GitLab Runner’s settings and set “concurrent” option in global level to “1”. Firstly, not everybody has this kind of access and even more so will want to do this kind of admin job. Secondly, we’ll change this settings for all the runners configured in this config.toml file. Thirdly, it only solve the same problem as resource group, so doesn’t help us at all. You can also check <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/16548">this opened issue</a> for some more details.</p><p>The same goes for another <em>popular</em> solution in web to set “limit” option to “1” in a <a href="https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section">runner section</a>. But it’s also only for concurrent jobs, not for pipelines.</p><p>But we can find quite a lot of tricky custom solutions like <a href="https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41560#note_98811340">this one</a>, but the idea is the same: use GitLab API to check unfinished pipelines and wait until we can proceed. None of them worked for me, so it’s time to…</p><h3>Create my own bicycle</h3><p>Let’s think what do we want to achieve.</p><ol><li>For every new pipeline check that there are no other <strong>active pipelines</strong> and we’re safe to continue.</li><li>Identify what is <strong>active pipeline</strong> for us: in my case I should take into account all running pipelines and pending pipelines with unfinished jobs. The last one is not obvious: we can run this call when another active pipeline has just finished one job and created another one, but not finished - then its status will be pending for a few seconds/milliseconds.</li><li>Wait until there are no active pipelines other than current one and we can proceed with further jobs.</li><li>Block this stage with waiting mechanism with <strong>resource_group</strong>! Otherwise two pipelines can start at the same time and it will be too difficult to identify which one should start first.</li></ol><p>For all concurrent pipelines I want to see the next picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAl_FYSZfqyhRzadbtCOmg.png" /><figcaption>Allow only 1 running pipeline with business jobs and 1 with wait job, the rest should wait for available resource</figcaption></figure><p>As for me, it looks like a really obvious feature that should be implemented in such a cool tool like GitLab CI! But okay, we have a plan, so let’s proceed.</p><h4>Let’s script!</h4><p>I decided to do it all in a raw shell script to make it easy to reuse for any language and to allow every engineer who is not a professional programmer to be able to understand what’s going on inside. Let’s try to use curl and jq only not to overcomplicate it with too many tools.</p><p>I’ll use a few environment variables here:</p><ul><li>PROJ_ACCESS_TOKEN - required to make API calls. I’ll describe below how to get one in “Project configuration” section</li><li>CI_API_V4_URL - predefined variable in GitLab CI that is equal to <a href="https://gitlab.com/api/v4">https://gitlab.com/api/v4</a> in our case</li><li>CI_PROJECT_ID - another predefine variable that equals to current GitLab project id.</li></ul><p>It’s very easy to get a count of currently active pipelines:</p><pre>active_pipelines=$(curl --silent --noproxy &#39;*&#39; --header &quot;PRIVATE-TOKEN:$PROJ_ACCESS_TOKEN&quot; &quot;$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines?status=running&quot;)<br>active_pipelines_count=$(echo $active_pipelines | jq &#39;. | length&#39;)</pre><p>A bit more tricky to get all pending pipelines, make some for loop on them and check whether each has either failed or succeded jobs (that means pipeline is in progress, just preparing a new job while we’re sending curl request). There is a little problem as well: for shell loops we cannot use an array that jq returns, so we can use while and read through all the parsed jq pipelines:</p><pre>pending_pipelines=$(curl --silent --noproxy &#39;*&#39; --header &quot;PRIVATE-TOKEN:$PROJ_ACCESS_TOKEN&quot; &quot;$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines?status=pending&quot;)<br>pending_jobs_count=0<br>echo &quot;$pending_pipelines&quot; | jq &#39;.[].id&#39; | while read -r pipeline_id; do<br>  finished_jobs_len=$(curl --silent --noproxy &#39;*&#39; --header &quot;PRIVATE-TOKEN:$OMNIBUS_UI_TESTS_ACCESS_TOKEN&quot; &quot;$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines/$pipeline_id/jobs?scope[]=failed&amp;scope[]=success&quot; | jq &#39;. | length&#39;)<br>  if [ $finished_jobs_len -gt 0 ]; then ((pending_jobs_count=pending_jobs_count+1)); fi<br>done</pre><p>Okay, we can wrap it into functions and check the total active pipelines count. Now we need to wait if we have more than 1 active pipeline (that is running our “wait” job) and check every N seconds (5 in my example) whether other active pipelines are finished</p><pre>pipelines_count=$(get_pipelines_count)<br>printf &quot;Currently $pipelines_count active pipeline$([ $pipelines_count -gt 1 ] &amp;&amp; echo &quot;s were&quot; || echo &quot; was&quot;) found\n&quot;<br>until [ $pipelines_count -eq 1 ]<br>do<br>  printf &#39;.&#39;<br>  pipelines_count=$(get_pipelines_count)<br>  sleep 5<br>done</pre><p>All I want to do in the end is to check total time execution and print it nicely with a format “…% minutes, % seconds”</p><p>Let’s store initial time before our script beging with “start_time=$(date +%s)” and finish it with:</p><pre>end_time=$(date +%s)<br>total_time=$(( end_time - start_time ))<br>minutes=$((total_time / 60))<br>seconds=$((total_time - 60*minutes))<br>final_time=$(echo &quot;$([ $minutes -ne 0 ] &amp;&amp; echo &quot;$minutes minutes, &quot; || echo &quot;&quot;)$seconds seconds&quot;)<br>printf &quot;\nLooks like it&#39;s only 1 active pipeline at this moment. Proceeding after $final_time\n&quot;</pre><p>The full script you can find <a href="https://gitlab.com/alexandrchumakin/wait-pipeline/-/blob/main/wait-pipelines.sh">here</a>.</p><h4>Project configuration</h4><p>You need to create access token in order to call GitLab API endpoints for your project. If you have ultimate license, it’s better to create project access token. Otherwise, you need to create personal access token. Only “api” scope should be enough for our case, so create token, copy it and put under your project &gt; Settings &gt; CI/CD &gt; Variables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*Tb8demO0ZKmnXBLhxoow6g.png" /><figcaption>GitLab API access token (either personal or project one)</figcaption></figure><p>To initiate example GitLab pipeline that will use free docker runners, we should find a correct image. All we need is two tools: jq and curl. There is a nice <a href="https://gitlab.com/gitlab-ci-utils/curl-jq">little image</a> existing in GitLab Registry that has both of them, let’s use it!</p><pre>image: registry.gitlab.com/gitlab-ci-utils/curl-jq:latest<br><br>wait_queue:<br>  resource_group: wait<br>  script:<br>    - chmod +x wait-pipelines.sh &amp;&amp; ./wait-pipelines.sh</pre><p>So, we have a job execution our wait script and we need some other job that will be running for a while to check that it’s working. For full example reach out to my <a href="https://gitlab.com/alexandrchumakin/wait-pipeline/-/blob/main/.gitlab-ci.yml">GitLab repo</a>.</p><h4>Final result</h4><p>I’ll start a few pipelines at once to see how my script can handle it.</p><h4>Final result</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzSu7G7sC_4C3-CBk_Ihjw.png" /><figcaption>Exactly like in the initial plan</figcaption></figure><p>In the output of wait jobs we can see that they actually work!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*crwJD7SH8nGY1vMjqJ1SFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*oPWMzgudLGstiRExJj6hrw.png" /></figure><p>You can find this example GitLab project <a href="https://gitlab.com/alexandrchumakin/wait-pipeline/-/blob/main/wait-pipelines.sh">here</a>.</p><p>In the end it’s not an easy, but quite stable and reusable solution. Feel free to share your ideas or comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a2f91432e77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How targeting advertising is effective and annoying]]></title>
            <link>https://medium.com/@achumakin/how-targeting-advertising-is-effective-and-annoying-b6e1fc05f510?source=rss-82b2093ff818------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6e1fc05f510</guid>
            <category><![CDATA[instagram]]></category>
            <category><![CDATA[target-ad]]></category>
            <category><![CDATA[online-advertising]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[instagram-ads]]></category>
            <dc:creator><![CDATA[Alexander Chumakin]]></dc:creator>
            <pubDate>Wed, 22 Jun 2022 18:15:14 GMT</pubDate>
            <atom:updated>2022-06-22T18:15:14.804Z</atom:updated>
            <content:encoded><![CDATA[<p>I had been working in <a href="https://site.adform.com/">Adform</a> at 2014/2015 that is a company working with everything related to targeting adverising. It was an amazing experience as all this advertising was not so popular, but already very competitive in a market and super interesting from technology point of view!</p><p>In order to win a bid for some place in website or app we needed to use the most modern and expensive technologies like <a href="https://aerospike.com/">Aerospike</a>, <a href="https://www.vertica.com/">Vertica</a>, <a href="https://aws.amazon.com/redshift/">Redshift</a>, etc. We didn’t have any other choice, but win any possible millisecond in huge data aggregation to be able to show ad to appropriate user before our competitors. There is a slide from the <a href="https://www.slideshare.net/alexchumakin12/selenium-camp-2017-alexander-chumakin">presentation</a> that I’ve used to explain this in one of the conferences at 2017:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*_mYB1PzzTl-kUTb0DAXu5w.png" /></figure><p>The idea I want to explain now is that from that kind of possibly useful online advertising all these algorythms became too annoying now. A few days ago I wanted to check a new Volvo xc40 recharge characteristics as one of my friends ordered it and was very excited about it. But I was stupid enough not to check this in Incognito mode… Right after that I’ve opened my Instagram and guess what I’ve seen?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*5omNcrZ8Fa2t1j5gil7R_Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*sy1sVgNq2S8M-VxPFzEVnQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*PGVgjfjATWqDfT9BEbpB4Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*28WKRarvGeMca2c6ZuJfYA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*o-GAn0nhJiu5yxdWHweboQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*WE6oPCXETjkeBSgt6YozUg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*Ed3482kBUWmRI10aSeQTwQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*-p85OUZdML-uPsOJwkOiVw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*Utp_OtWCEKHWfPdyDPmNLQ.jpeg" /></figure><p>I had different cars ads in more than 20% of all my Instagram feed! After I tried to hide all of them or report as not relevant, guess what I saw next?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*S9D89TXYZNTpZI6GgUjH_g.jpeg" /><figcaption>mortgage ad</figcaption></figure><p>Now, clever algorithms thought I just don’t have money right now and I need a morgage! Okay, I’ve closed Instagram and didn’t use it for a few hours. But from my experience I knew that such kind of interest will not expire so soon. Anyway, I was really surprised by what ads did I see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*0j17n8W5xAxDd0bcmvLANA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*pTHzZOA_MwLIEmrQOzXyTg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*4-LxTY3fNzK-Ma8LSyS9Bw.jpeg" /></figure><p>Seriously? If you don’t want to buy a car right now, let us suggest you something transportation-related, but completely different from your initial interest! Even after a few more days I see this kind of ads everywhere and hope it will expire in a week or so.</p><p>It will be very interesting to work within this domain again in future and try to convince company to be a little bit less agressive as nobody likes it. As for me, I would prefer vice versa not to buy from a brand that performs too annoying advertising company.</p><p>Be loyal and friendly to your potential customers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6e1fc05f510" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How did I run front-end tests in GitLab CI]]></title>
            <link>https://medium.com/@achumakin/how-did-i-run-tests-in-gitlab-ci-c6d8455c943b?source=rss-82b2093ff818------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6d8455c943b</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Alexander Chumakin]]></dc:creator>
            <pubDate>Fri, 17 Jun 2022 14:18:47 GMT</pubDate>
            <atom:updated>2022-06-17T16:00:48.452Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is this article about?</h3><p>How did I configure UI tests execution in GitLab CI for isolation testing of web app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*jj3cQr_abI4fSt0Kt5w_VA.png" /><figcaption>CI test execution architecture</figcaption></figure><h3>Briefly describe the original problem</h3><p>We have a few micro frontends built with Angular that we need to test. Our test environments are not the best place to rely on, so we decided to test it in isolation. For this we built this whole monorepo project as a single docker image and then used it in docker-compose to speen up all micro frontend components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*9NdRBvcMCvfAkNmLT1Re2g.png" /><figcaption>Dockerfile</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*NK7kF4X8Z9DY-lAnU37C5A.png" /><figcaption>docker-compose.yml</figcaption></figure><p>It was working quite OK as initial POC, but then we’ve faced with some problems:</p><ol><li>Because of huge amount of transitive dependencies coming from common design system and probably not the best organization of project dependencies we had a node_modules as an output of “npm ci” of 2+ Gb!</li><li>Simple npm command might take anonymous amount of time to execute or even crashed in some old development laptops.</li><li>Nobody was a big fun of docker-compose as an additional tool in top of everything we already had.</li><li>We needed to perform docker build command before using of docker-compose every time not to build it again and again for every microfrontend listed in docker-compose.</li><li>This setup was completely different from what we’ve used in real environments.</li><li>Actual docker-compose file was way more complicated than in the screenshot above.</li><li>The final point came when after some upgrades it was not possible to build this docker image for local development for some of our engineers.</li></ol><h3>Finding a solution</h3><h4>Reuse image from CI</h4><p>The first obvious think to try in my head was to reuse docker image from CI locally as it always sucessfully built in our runners.</p><p>We had two main problems with that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*g1xd7X9gIhuXrFsbPhWzCw.png" /></figure><ol><li>The size of this image that’s more than 3 Gb! We needed to publish this image from CI again and again for every build to our Artifactory host. And pulling of this image locally for everybody who needs it was probably not the best option as well.</li><li>More critical was that we’ve used a custom settings in this image to connect apps via docker bridge IP in CI and we needed to have a custom local settings pointing to localhost and mounting that into image with docker-compose. Wow, looks too tricky.</li></ol><h4>Use production-like images for compose</h4><p>What if we don’t build this gigantic docker image, but instead prepare a few little nginx-images for each app in the same way as we do for real environments? It’s a big process to prepare the output that nginx server can use, including all the licenses and custom configuration, but let’s use the output from CI!</p><p>Okay, now we have docker-compose with more clear structure and preparing of different images for each app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*71CNDyKvNDtNIqze2H9dSA.png" /><figcaption>Dockerfile</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*RMN_7r8cd_kkBoKw_DkIFQ.png" /><figcaption>docker-compose.yml</figcaption></figure><p>And you know what? It works locally just like a charm! We have now two little images (comparing to initial one) of ~100 Mb and all the web apps working pretty cool! But…</p><p>When it’s time to move this into CI there is a “little” challenge. Do you remember we had an additional step to build monster-image before using it in docker-compose? It looks like we had a panacea before we’ve faced an issue with docker-compose build!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*H6KPaIwYProCB6Qj4VhX-A.png" /><figcaption>output from docker-compose build from CI</figcaption></figure><p>Hmm, we cannot build image because of some strange error, let’s google it! You can find very a few possible solutions for this issue that are:</p><ol><li>Run docker buildx install</li><li>Run compose build with some parameters like --no-cache or --pull</li><li>Upgrade/downgrade docker-compose and docker version</li><li>Allocate more memory for this process</li><li>Check gitlab-runner permissions</li></ol><p>After checking everything and playing with version of docker-compose versions I’ve tried to build all the images one by one and use them in docker-compose.yml-file… and that worked. But then we needed to run a few docker build commands (for each micro frontend) and then run compose command that is quite weird.</p><p>But maybe there is an issue with our docker-in-docker approach then?</p><h4>Un-dockerize GitLab runner</h4><p>I’ve never being one who tried to setup servers with everything needed when we have such powerful tool like docker! But let’s try this out!</p><p>I’ve configured docker-compose with a correct version that supports my configuration versions somehow (oh yes, we have a separate story because of a few levels of proxies in our VMs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jacIZO0nbvFe1W4IMxp4Ow.png" /><figcaption>trying to find a correct version of compose in local registry under proxy</figcaption></figure><p>Alright, now everything should be like locally, right?.. Nope :(</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtha1b2jOsR-O-09n7r4OQ.png" /><figcaption>Waiting for web server to be up and running</figcaption></figure><p>After some investigations I saw that all apps are running and healthy, but simple curl fails because of some default servers’ configurations provided by our infra team. Looks to hard to fix server to support this quite dirty solution with shell runner!</p><h4>Time to un-compose!</h4><p>I have been asked many times not to use compose here because of the reasons I mentioned above, but I had a serious motivation to use compose! Nx cannot start apps in detached mode inside one container one by one and dynamically wait until it’s free to run the next one. You either need to add some static timeouts between running of the apps or execute another magic shell script for it. And of course total start time will be deceased significantly.</p><p>But we use nginx solution already, nothing to do with nx problems! Let’s try to run everything inside single docker container with all the apps inside! We only need to prepare correct nginx config-file that will start our micro frontends once it’s started. A few changes in configurations and … tadam! It works so smooth locally. But guess what? Right, again some problems with moving to CI.</p><p>When we have everything running locally, we can open that in a browser <a href="https://localhost:8080/">https://localhost:8080</a> and execute tests against it. But in CI we run tests in a separate docker container (default and recommended behavior for GitLab runners). So, all we need to do is to make sure our docker container with web apps is running in the same network as one that uses to run tests and replace localhost with correct IP address. What is correct address in this case?</p><p>One of the ways to connect containers in bridge network is to use Gateway, you can check this by running “docker inspect” command on the container</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*qknYDjYTaaY1XyodD1Rczw.png" /><figcaption>check Gateway of the container locally</figcaption></figure><p>Then you can connect to this container via this gateway IP and port that you exposes - exactly like we did with localhost. But there is a problem! We don’t have Gateway populated for this docker container in CI!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*l1var3G2JHDOfr4EusWqyg.png" /><figcaption>check Gateway of the container in CI machine</figcaption></figure><p>Hmm, maybe there is something wrong with the network gateway?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzJfbrle_HwG5C_jX9rc1g.png" /><figcaption>check network Gateway in CI</figcaption></figure><p>No, it looks exactly the same like locally.</p><p>But we have another option! Use IPv4 address of docker container, then we don’t even need to publish ports from initial docker container with apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6NGu8XsRWb_9qVtau2C30g.png" /><figcaption>Get docker container IP address</figcaption></figure><p>Now we can use this address to reach out to our app from another container, right?! Yes, website is reachable, but there is another problem of course: we fail to send authentication request to keycloak and always get redirection error</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*gX5VvWilYDwzr-aBmiIShA.png" /><figcaption>error from initial Keycloak call</figcaption></figure><p>All you can google for this keycloak issue is to either enable whatever address you use or use start (*) for this configuration. It’s safe as we’re not in some environment, but inside test CI pipeline, so it feals safe to try.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*lo90ZImk5P-x3ubz_mGZwA.png" /><figcaption>Keycloak configuration</figcaption></figure><p>But then I realize it didn’t help because of the way how we configured nginx configurations and some reverse proxy settings in a host.</p><h4>Light in the end of a tunnel</h4><p>Okay, just need to access app via localhost. Then it’s time to configure another GitLab runner under host network and run the container with web apps under the same network!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*voyPACKslUVq_EGiQgyb4g.png" /><figcaption>GitLab runner configuration under [runners.docker] section</figcaption></figure><p>And we can finnaly reach out to the host and run all our tests!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*Thowx0Rv7P78NKT42KuKiQ.png" /><figcaption>Connect to another docker container via host network</figcaption></figure><h3>Conclusion</h3><p>Engineering job moves toward close to DevOps tasks if we want to be flexible and provide fully end-to-end working solutions without any external dependencies.</p><p>It might be really challenging to move something “working perfeclty fine” locally to CI, especially when you’re limited with different infrastructure settings like firewalls and proxies. But this is what makes us better every day! Only a way to grow profesionally is to face with problems and solve them step by step until you and your team are happy with a final solution!</p><p>Happy coding! And happy … “infra configuring”?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6d8455c943b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>