<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[500 Internal server error - Medium]]></title>
        <description><![CDATA[Keboola Product and Engineering team - Medium]]></description>
        <link>https://medium.com/keboola-nerds?source=rss----9f95ce157e5e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>500 Internal server error - Medium</title>
            <link>https://medium.com/keboola-nerds?source=rss----9f95ce157e5e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:35:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/keboola-nerds" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a PHP 7.4.21 Docker image with support for GDB debugger]]></title>
            <link>https://medium.com/keboola-nerds/building-a-php-7-4-21-docker-image-with-support-for-gdb-debugger-fcbcf3768012?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/fcbcf3768012</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Ondřej Hlaváček]]></dc:creator>
            <pubDate>Tue, 03 Aug 2021 09:48:47 GMT</pubDate>
            <atom:updated>2021-08-03T10:05:16.580Z</atom:updated>
            <content:encoded><![CDATA[<p>When your process gets stuck somewhere, but you have no idea where, or if it segfaults, xdebug may not help. I had a process stuck in a loop outside of PHP. I tried strace on the process, but it only showed an infinite loop in a library outside of PHP, with no chance of tracing where in my PHP code that was.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5CnMfmrIZnCv2rgGDlXIrA.png" /></figure><p>So, what to do? Could I start the gdb debugger and get a stack trace from there?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cd0NcurKABLkkXpzwpDahg.png" /></figure><p>This does not tell us anything interesting, right?</p><p>It’s missing something — debug symbols for PHP. They enable the debugger to get you the real PHP stack trace. They cannot be installed; you have to compile PHP with an --enable-debug flag and a few other tweaks. I have the whole Dockerfile ready at <a href="https://gist.github.com/ondrejhlavacek/f929ad3ab6ae2563f1ce7df41ee5bf05">https://gist.github.com/ondrejhlavacek/f929ad3ab6ae2563f1ce7df41ee5bf05</a>.</p><h3>The diff</h3><p>Let’s have a look at the diff and take it step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ob74jb3jYRc4cK6wVO0HpA.png" /></figure><pre>+   gdb \</pre><p>Install gdb to the image, part of the apt-get install command.</p><pre>-ENV PHP_EXTRA_CONFIGURE_ARGS --with-apxs2 --disable-cgi<br>+ENV PHP_EXTRA_CONFIGURE_ARGS --with-apxs2 --disable-cgi --enable-debug</pre><p>Add an --enable-debug flag when running ./configure. This is pretty self-explanatory. It enables debug mode in PHP.</p><pre>-ENV PHP_LDFLAGS=&quot;-Wl,-O1 -pie&quot;<br>+ENV PHP_LDFLAGS=&quot;-Wl,-O0 -pie&quot;</pre><p>This turns off all optimizations, which is necessary as they may delete debug symbols.</p><pre>-find /usr/local/bin /usr/local/sbin -type f -executable -exec strip --strip-all ‘{}’ + || true; \</pre><p>This was the trickiest part. The strip program literally strips all redundant code from a binary, including debug symbols.</p><pre>-docker-php-source delete; \</pre><p>Having a PHP source in the image allows you to load PHP-specific debugging functions from the .gdbinit file in the debugger. Have a look at the file at <a href="https://github.com/php/php-src/blob/master/.gdbinit">https://github.com/php/php-src/blob/master/.gdbinit</a>.</p><p>So, once we have the Dockerfile ready, we can build it and test it.</p><pre>docker build . -t php:7.4.21-apache-buster-debug<br>docker run -i -t php:7.4.21-apache-buster-debug /bin/bash<br>gdb /usr/local/bin/php<br>source /usr/src/php/.gdbinit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlXOGnCn0wvhYHgJgZaDeA.png" /></figure><p>If you can see Reading symbols from /usr/local/bin/php...done., everything is in running order.</p><p>Once this is working, you can include this image in your Dockerfile and build your application. You can then attach gdb to any running PHP process and get a nice stack trace, using the zbacktrace function from the PHP&#39;s own .gdbinit file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mzMYdacii3o99FwFKIzYLA.png" /></figure><p>This is what I wanted. Shortly after locating where the process kept freezing, we were able to find the bug and fix it once and for all.</p><h3>Debugging in containers</h3><p>If you run your app in Docker, just add --cap-add SYS_PTRACE to the docker run command line.</p><p>If you’re debugging PHP in a Kubernetes pod, you have to set up both container and pod securityContext.</p><pre>spec:<br> securityContext:<br>   runAsUser: 0<br>   runAsGroup: 0<br>...<br> containers:<br> - name: my-container<br>   securityContext:<br>     capabilities:<br>       add: [ &quot;SYS_PTRACE&quot; ]<br>...</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcbcf3768012" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/building-a-php-7-4-21-docker-image-with-support-for-gdb-debugger-fcbcf3768012">Building a PHP 7.4.21 Docker image with support for GDB debugger</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MySQL + SSL + Doctrine]]></title>
            <link>https://medium.com/keboola-nerds/mysql-ssl-doctrine-183edd62cb70?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/183edd62cb70</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[doctrine]]></category>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[symfony]]></category>
            <dc:creator><![CDATA[Ondřej Popelka]]></dc:creator>
            <pubDate>Wed, 05 May 2021 16:11:47 GMT</pubDate>
            <atom:updated>2021-05-05T16:06:24.942Z</atom:updated>
            <content:encoded><![CDATA[<p>Enabling and <strong>enforcing</strong> SSL connection on MySQL is easy: Just <a href="https://dev.mysql.com/doc/refman/8.0/en/using-encrypted-connections.html">generate the certificates</a> and configure the server to <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_require_secure_transport">require secure transport</a>. And it’s even easier on the Azure Database for MySQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtg0hfZaNFb0BxH3Wl2U4A.png" /></figure><p>Just go to the Azure portal and enable “Enforce SSL connection” in the “Connection security” of the database server. Then, <a href="https://docs.microsoft.com/en-us/azure/mysql/concepts-ssl-connection-security#ssl-default-settings">download the certificates</a> and stuff them in your application. No problem — just two simple steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*KmhCr3_erbhXb3feCNYoxA.png" /></figure><p>On a hackathon, we’d just download the certificate, put it in a random folder, and move on. In reality, we’re in a bit different situation then. We’d have three (so far) services connecting to the server (using independent databases), and this setup repeats itself in several independent stacks and some of those stacks are Azure stacks and some are AWS stacks. The services themselves run in Kubernetes. If we simplify (i.e.; omit a hundred other services and resources), it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*IcwY9p61aujfw5qpFdzJ2g.png" /></figure><p>The rest of the story is all about Service 1 :)</p><h4>Step 1 — Store the certificate somewhere</h4><p>Given the above, it’s out of the question to store the certificate with the application, because the certificate is not constant. The service application itself doesn’t care whether it’s running in AWS or Azure or what server it is connecting to. Also the application itself is completely unaware of all the stacks it runs on. This means that the certificate must be supplied to the application from outside.</p><p>All our deployments are fully automated with Azure pipelines and happen several times a day. One option is to store the certificate in the pipeline variables and then pass it around in environment variables. This is possible but can be somewhat error-prone. The certificate contains special characters, so it needs to be encoded and decoded and if you mess up, debugging it is hard, because all you get is “Invalid certificate”.</p><p>To avoid this we took advantage of the fact that the services run in a Kubernetes cluster. An so we went with the option of mounting the certificate as a config map. This requires:</p><ul><li>downloading the certificate,</li><li>using kubectl to <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#create-a-configmap">generate</a> a config map <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#create-configmaps-from-files">from file</a>,</li><li>applying the config map.</li></ul><p>Condensed, it’s the following script:</p><pre># Root SSL cert for Azure MySQL<br>curl <a href="https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.pem">https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.pem</a> <br>--location --output /tmp/BaltimoreCyberTrustRoot.crt.pem</pre><pre>kubectl create configmap mysql-root-ca-cert <br>--from-file=MySQLRootCACert.pem=/tmp/BaltimoreCyberTrustRoot.crt.pem  --dry-run=client -o yaml | kubectl apply -n default -f -</pre><p>All of the above happens in our infrastructure deploy pipeline, which makes sure that all the shared resources for the applications are initialized properly.</p><h4>Step 2 — Get the certificate to the application</h4><p>This one is very straightforward. We just <a href="https://kubernetes.io/docs/concepts/storage/volumes/#configmap">mount the config map</a> as a volume:</p><pre>volumes:<br>- name: mysql-root-ca-cert<br>  configMap:<br>    name: mysql-root-ca-cert<br>    items:<br>    - key: MySQLRootCACert.pem<br>      path: MySQLRootCACert.pem</pre><p>Then mount the volume in the application container:</p><pre>volumeMounts:<br>- name: mysql-root-ca-cert<br>  mountPath: /code/var/mysql</pre><p>The application will now have the proper certificate available in /code/var/mysql/MySQLRootCACert.pem .</p><h4>Step 3 — Configure the application to use the certificate</h4><p>Now we’re getting to the interesting part. Our applications are written in PHP and use <a href="https://www.doctrine-project.org/projects/doctrine-dbal/en/latest/">Doctrine DBAL</a> to connect to the database. The doc is a bit skimpy about setting the SSL options, it just says that you should use the options field to supply the PDO driver options. Luckily, there is a Stack overflow post that <a href="https://stackoverflow.com/questions/16600028/how-to-connect-to-mysql-using-ssl-on-symfony-doctrine">gives a hint</a> — and we confirmed that this works:</p><pre>dbal:<br>        url: &#39;mysql://user:password@dbhost:3306/dbname&#39;<br>        server_version: &#39;8.0&#39;<br>        options:<br>            1012: &#39;/code/var/mysql/MySQLRootCACert.pem&#39;</pre><p>The magic value of 1012 represents the <a href="https://www.php.net/manual/en/ref.pdo-mysql.php#pdo-mysql.constants">constant</a> <strong>PDO::MYSQL_ATTR_SSL_CA , </strong>which is quite crazy, because constants were invented for a reason. And yes, the value is actually different for different PHP versions. Luckily, we also use Symfony, which comes to the rescue by supporting <a href="https://symfony.com/doc/4.2/service_container/parameters.html#constants-as-parameters">PHP constants in YAML configurations</a>; therefore I can write:</p><pre>dbal:<br>        url: &#39;mysql://user:password@dbhost:3306/dbname&#39;<br>        server_version: &#39;8.0&#39;<br>        options:<br>            !php/const:PDO::MYSQL_ATTR_SSL_CA: &#39;path/file.pem&#39;</pre><p>That’s great. The supplied values are of course variables, so the real doctrine.yaml looks like this:</p><pre>dbal:<br>        url: &#39;mysql://%env(DATABASE_USER)%:%env(DATABASE_PASSWORD)%@%env(DATABASE_HOST)%:%env(DATABASE_PORT)%/%env(DATABASE_NAME)%?serverVersion=8.0&#39;<br>        server_version: &#39;8.0&#39;<br>        options:<br>            !php/const:PDO::MYSQL_ATTR_SSL_CA: &#39;%env(DATABASE_SSL_CA_PATH)%&#39;</pre><p>We then supply the environment variables through the Kubernetes template of the application; the abbreviated template looks like this:</p><pre>---<br>kind: Deployment<br>apiVersion: apps/v1<br>metadata:<br>  name: app-name<br>  namespace: default<br>  labels:<br>    app: app-name<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: app-name<br>  template:<br>    metadata:<br>      labels:<br>        app: app-name<br>   spec:<br>      containers:<br>      - name: app-name<br>        image: app-repository:app-image<br>        ports:<br>        - containerPort: 80<br>        env:<br>        - name: DATABASE_HOST<br>          valueFrom:<br>            configMapKeyRef:<br>              name: app-name<br>              key: dbHost<br>        - name: DATABASE_PORT<br>          valueFrom:<br>            configMapKeyRef:<br>              name: app-name<br>              key: dbPort<br>        - name: DATABASE_USER<br>          valueFrom:<br>           configMapKeyRef:<br>             name: app-name<br>             key: dbUser<br>        - name: DATABASE_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: app-name-credentials<br>              key: dbPassword<br>        - name: DATABASE_NAME<br>          valueFrom:<br>            configMapKeyRef:<br>              name: app-name<br>              key: dbName<br>        - name: DATABASE_SSL_CA_PATH<br>          valueFrom:<br>            configMapKeyRef:<br>              name: app-name<br>              key: dbCertificatePath<br>       <br>        volumeMounts:<br>        - name: mysql-root-ca-cert<br>          mountPath: /code/var/mysql<br>      volumes:<br>        - name: mysql-root-ca-cert<br>          configMap:<br>            name: mysql-root-ca-cert<br>            items:<br>            - key: MySQLRootCACert.pem<br>              path: MySQLRootCACert.pem<br>---</pre><p>This is accompanied with the corresponding config map and secret created at the deploy time.</p><h4>Step 4 — Configure the application for local development</h4><p>Once the application is working in production, it’s time to fix it so that the local development is as easy as possible. This also includes running the CI pipelines of the application. The pipeline contains functional tests that require a working database connection and these are run on every push to the application repository. There are several main options:</p><ul><li>Have an Azure MySQL server running for development purposes (this costs money).</li><li>Spin up a new Azure MySQL server each time for development (this takes time — the whole pipeline runs for 5 minutes and spinning up a new MySQL server can take even more than that).</li><li>Spin up a local MySQL server for the purpose of development or the CI pipeline (this needs some thinking).</li></ul><p>Not liking to waste time or money we went with the third approach some time ago. The tests run in containers, so both the CI pipeline and the developer can run them the same way using:</p><pre>docker-compose run --rm tests composer ci</pre><p>The docker-compose.yml makes sure that a MySQL server exists:</p><pre>version: &quot;3&quot;</pre><pre>services:<br>tests: &amp;tests<br>    image: keboola/job-queue-internal-api<br>    build:<br>      context: .<br>      dockerfile: Dockerfile<br>    ports:<br>      - &quot;81:80&quot;<br>    environment:<br>      - DATABASE_USER=root<br>      - DATABASE_PASSWORD<br>      - DATABASE_HOST=mysql<br>      - DATABASE_PORT=3306<br>      - DATABASE_NAME=jobs<br>    depends_on:<br>      - mysql</pre><pre>mysql:<br>    image: mysql:8<br>    command: --default-authentication-plugin=mysql_native_password<br>    ports:<br>      - &quot;3308:3306&quot;<br>    environment:<br>      - MYSQL_DATABASE=jobs<br>      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}</pre><p>All the developer (or the CI pipeline) needs to do is set the DATABASE_PASSWORD environment to some random password which is then supplied to both containers. Simple &amp; Effective™. Also it has the benefit of the tests being stateless.</p><p>The SSL requirement disturbs this. Once the MYSQL_ATTR_SSL_CA option is set, there must be a certificate and it must be valid. If it is not, you get:</p><pre>Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[HY000] [2002]</pre><p>That’s basically the equivalent of “go fuck yourself!” However, if you were really good last year and Santa likes you, you might get something similar to:</p><pre>PDOException: PDO::__construct(): SSL operation failed with code 1. OpenSSL Error messages:<br> error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding<br> error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed<br> error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib<br> error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed</pre><p>That at least gives some idea that the problem is in the certificate.</p><p>First I went with the idea that I will generate the certificates mys(QL)elf. This was <a href="https://dev.mysql.com/doc/refman/8.0/en/creating-ssl-files-using-openssl.html">not difficult</a>, but I ended up with self-signed certificates. These are ok for MySQL, but are <a href="https://bugs.php.net/bug.php?id=74943">not ok for PHP</a>, because it uses strict checking by default (which is a good thing actually!). The option is to turn off the strict checking for local development and the CI pipeline. Remember that the test server is short lived — it exists only during the tests — so this doesn’t pose a security issue.</p><p>The option to disable certificate checking luckily exists and is available using the MYSQL_ATTR_SSL_VERIFY_SERVER_CERT <a href="https://www.php.net/manual/en/ref.pdo-mysql.php#pdo-mysql.constants">constant</a>. I modified the doctrine.yml to:</p><pre>dbal:<br>        url: &#39;mysql://%env(DATABASE_USER)%:%env(DATABASE_PASSWORD)%@%env(DATABASE_HOST)%:%env(DATABASE_PORT)%/%env(DATABASE_NAME)%?serverVersion=8.0&#39;<br>        server_version: &#39;8.0&#39;<br>        options:<br>            !php/const:PDO::MYSQL_ATTR_SSL_CA: &#39;%env(DATABASE_SSL_CA_PATH)%&#39;<br>            !php/const:PDO::MYSQL_ATTR_SSL_VERIFY_SERVER_CERT: &#39;%env(DATABASE_SSL_VERIFY_ENABLED)%&#39;</pre><p>However that raises a bit of a red flag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*9WGATYjO4pLgoA4k.jpg" /></figure><p>Having to set an option to <strong>enable</strong> certificate verification looks like a bad idea, because it implies that by default it is disabled. Luckily this is not entirely true, because the SSL connection is <strong>enforced</strong> by the configuration of the production MySQL server.</p><p>But doppelt hält besser. I would like to have an environment variable DATABASE_SSL_VERIFY_ENABLED that is enabled by default, but can be overridden to be disabled. Luckily Symfony comes to the rescue again with the possibility to set default values for <strong>environment variables</strong>. All it needs is you to set the corresponding environment in services.yml in the parameters section:</p><pre>parameters:<br>    locale: en<br>    app_name: my-app<br>    env(DATABASE_SSL_VERIFY_ENABLED): &#39;1&#39;</pre><pre>services:<br>    ...</pre><p>This is a nice little bit of magic that makes sure that if the environment DATABASE_SSL_VERIFY_ENABLED is not set, Symfony will set it to the value 1 . While, if the variable is set, Symfony will leave it alone. With this — I leave DATABASE_SSL_VERIFY_ENABLED undefined and only configure it locally or in a CI pipeline to successfully connect to the ephemeral MySQL server. It’s worth noting that in that case, the value of MYSQL_ATTR_SSL_CA becomes completely irrelevant — the referenced file does not even have to exist.</p><p>Done. Problem Solved. I drew the owl.</p><h4>P.S. Alternative solution</h4><p>The obvious question with disabling SSL is — why we didn’t use the Symfony environments. The answer to that is — we are managing the application environments on a higher level — in Kubernetes. The application runs in several different production environments as well as in several different test and development environments none of which care about the application and all of which are unknown to the application. For example, we use the same production Kubernetes template to run service application A within the tests of application B. Application A mimics the production environment (and is configured so) and application B runs its test suite. That means that switching the <a href="https://symfony.com/doc/current/configuration.html#selecting-the-active-environment">application environment</a> via APP_ENV makes this actually more complicated, because it applies to one application within the CI pipeline and not to the other. Plus, we’d have to generate the .env files. Contrary to that, the environment variable DATABASE_SSL_VERIFY_ENABLED is application agnostic — it’s on the same level as the database credentials and says that the given MySQL server does not take SSL too seriously.</p><h4>P.S. Additional craziness</h4><p>In an attack of euphoria, I also set the client certificate connection options:</p><pre>options:<br> !php/const PDO::MYSQL_ATTR_SSL_CA : &#39;MySQLRootCACert.pem&#39;<br> !php/const PDO::MYSQL_ATTR_SSL_KEY : &#39;client-key.pem&#39;<br> !php/const PDO::MYSQL_ATTR_SSL_CERT : &#39;client-cert.pem&#39;</pre><p>With the SSL on Azure MySQL server, you shouldn’t set those, because you have neither the client key nor the certificate. The thing is, if you do set these, nothing happens. The client will see them as invalid and simply ignore the value. That is, unless you manage to set them so that the referenced files actually exist! In that case, you will get:</p><pre>SQLSTATE[HY000] [2006] MySQL server has gone away</pre><p>Which is a completely off-track error. If you’re lucky, you might also get a:</p><pre>SSL operation failed with code 1. OpenSSL Error messages:<br>error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca</pre><p>Which is slightly better, but points at the wrong offender. The CA is OK, it’s the client certificate that does not match in this case. The resolution is simple — follow the docs and configure only the CA certificate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=183edd62cb70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/mysql-ssl-doctrine-183edd62cb70">MySQL + SSL + Doctrine</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Self Hosted Pipelines — Part 2]]></title>
            <link>https://medium.com/keboola-nerds/automating-self-hosted-pipelines-part-2-8d47d830ace?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/8d47d830ace</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-devops-pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[self-hosted]]></category>
            <dc:creator><![CDATA[Ondřej Popelka]]></dc:creator>
            <pubDate>Fri, 23 Apr 2021 18:57:36 GMT</pubDate>
            <atom:updated>2021-04-23T18:57:07.279Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Self-Hosted Pipelines — Part 2</h3><p><em>In the </em><a href="https://500.keboola.com/automating-self-hosted-pipelines-part-1-a635f154f021"><em>first part of the article</em></a><em>, I described how I automate the worker virtual machine setup using a </em><a href="https://github.com/keboola/pipeline-worker/blob/main/startup.sh"><em>Bash script</em></a><em>. Now I’ll describe how to automate that Bash script using Azure Pipelines.</em></p><p>Having gone through the first part of the journey, I now have a <a href="https://github.com/keboola/pipeline-worker/blob/main/startup.sh">Bash script</a> which takes two environment variables — a token and a worker name — and sets up a new Azure pipeline worker. That’s cool but I still have to create the token, write the instructions how to run the script, store the script somewhere, run the script somewhere, etc. Still some room for automation and repeatability.</p><h3>More Automation</h3><p>So, how about I build an Azure pipeline which creates an Azure pipeline worker?</p><p>Sure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*tQK0STHT7wF_a3wT" /></figure><p>We have a dedicated Azure subscription for CI services (like a testing Kubernetes cluster, or resources needed to run tests of applications). That makes it convenient for this case too. I.e., I can authorize a pipeline to the CI subscription knowing that it won’t break anything except the CI in the worst case.</p><p>Let’s recap a bit. I have the virtual machine automated by an ARM template. I have the configuration automated by a bash script. I need to run that bash script on that machine.</p><p>An obvious solution is to open an SSH port, log in and run the script there. And that’s a big “no” for me, because it’s completely unnecessary for the worker to have any open ports to the outside world. In the rare case I need to visit the worker, the <a href="https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-linux">Serial console</a> suffices.</p><p>Luckily it’s possible to run scripts as part of the virtual machine deployment. The two most suitable solutions are <a href="https://techcommunity.microsoft.com/t5/itops-talk-blog/how-to-run-scripts-on-azure-virtual-machines-with-the-run/ba-p/292446">Run command</a> and <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux">Custom Scripts</a> extension. The Run command can be used in this case, but the script has to be present on the target machine. This would be an extra moving part for me. I’d either have to create the command so that it self-downloads or create a custom VM image (another moving part). A custom script extension takes care of this. It has a limitation — that the script size cannot exceed 256 KB, which is more than plenty for me. The <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux#property-script">walkthrough</a> is nice, and that’s pretty much all I needed.</p><p>Custom script extension can be placed as a sub-resource of the virtual machine. Roughly like this:</p><pre>{<br>            &quot;name&quot;: &quot;[variables(&#39;resourceName&#39;)]&quot;,<br>            &quot;type&quot;: &quot;Microsoft.Compute/virtualMachines&quot;,<br>            &quot;apiVersion&quot;: &quot;2020-12-01&quot;,<br>            &quot;location&quot;: &quot;[resourceGroup().location]&quot;,<br>            &quot;tags&quot;: &quot;[variables(&#39;resourceTags&#39;)]&quot;,<br>            &quot;dependsOn&quot;: [...],<br>            &quot;properties&quot;: {<br>                &quot;hardwareProfile&quot;: {<br>                    &quot;vmSize&quot;: &quot;Standard_D2s_v3&quot;<br>                },<br>                ... snip ...<br>            },<br>             &quot;resources&quot;: [<br>                {<br>                    &quot;name&quot;: &quot;[concat(variables(&#39;resourceName&#39;),<br>                                 &#39;startup&#39;)]&quot;,<br>                    &quot;type&quot;: &quot;Extensions&quot;,<br>                    &quot;location&quot;: &quot;[resourceGroup().location]&quot;,<br>                    &quot;apiVersion&quot;: &quot;2019-03-01&quot;,<br>                    &quot;tags&quot;: {<br>                        &quot;displayName&quot;: &quot;Startup script&quot;<br>                    },<br>                    &quot;properties&quot;: {<br>                        &quot;publisher&quot;: &quot;Microsoft.Azure.Extensions&quot;,<br>                        &quot;type&quot;: &quot;CustomScript&quot;,<br>                        &quot;typeHandlerVersion&quot;: &quot;2.1&quot;,<br>                        &quot;protectedSettings&quot;: {<br>                            &quot;commandToExecute&quot;: <br>                              &quot;[concat(&#39;bash test.sh &#39;,    <br>                               parameters(&#39;patToken&#39;), <br>                               &#39; &#39;, variables(&#39;resourceName&#39;))]&quot;,<br>                            &quot;fileUris&quot;: [&quot;<a href="https://gist.githubusercontent.com/XXXX/test.sh">https://XXX/test.sh</a>&quot;]<br>                        }<br>                    }<br>                }            <br>            }<br>        },</pre><p>But then the error messages are highly unreadable (everything mangled on one line). So, I resorted to running it separately using the az vm extension set <a href="https://docs.microsoft.com/en-us/cli/azure/vm/extension?view=azure-cli-latest#az_vm_extension_set">command</a>.</p><pre>script_content=$(cat startup.sh | gzip -9 | base64 -w 0)<br>az vm extension set \<br>  --publisher &quot;Microsoft.Azure.Extensions&quot; \<br>  --name &quot;CustomScript&quot; \<br>  --version &quot;2.0&quot; \<br>  --ids &quot;XXXXXX&quot; \<br>  --protected-settings &quot;{\&quot;script\&quot;:\&quot;$script_content\&quot;}&quot;</pre><p>I also realized that I can directly pass the script code so it doesn’t have to be downloaded on the target machine. It just needs to be base64 encoded and optionally gzipped (also presented nicely in the <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux#property-script">official docs</a>).</p><h4>Setting up a pipeline</h4><p>With custom scripts extension I’m almost there, I just have to put the parts together. The worker can then be deployed using the following script (did I say I really try to avoid bash?):</p><pre>az group create \<br>  --subscription &quot;$SUBSCRIPTION&quot; \<br>  --location &quot;East US&quot; \<br>  --name &quot;$WORKER_NAME&quot; \<br>  --tags &quot;purpose=azure-pipeline&quot; &quot;workerName=$WORKER_NAME&quot;</pre><pre>az deployment group create \<br>    --subscription &quot;$SUBSCRIPTION&quot; \<br>    --resource-group &quot;$WORKER_NAME&quot; \<br>    --name &quot;$WORKER_NAME&quot; \<br>    --template-file ./template.json \<br>    --parameters \<br>        adminUsername=&quot;$ADMIN_USERNAME&quot; \<br>        adminPassword=&quot;$ADMIN_PASSWORD&quot;</pre><pre>vmId=$(<br>  az deployment group show \<br>    --subscription &quot;$SUBSCRIPTION&quot; \<br>    --resource-group &quot;$WORKER_NAME&quot; \<br>    --name &quot;$WORKER_NAME&quot; \<br>    --query &quot;properties.outputs.vmId.value&quot; \<br>    --output tsv<br>)</pre><pre>script_content=$(cat startup.sh | gzip -9 | base64 -w 0)</pre><pre>printf &quot;\nRunning Startup script&quot;<br>az vm extension set \<br>  --publisher &quot;Microsoft.Azure.Extensions&quot; \<br>  --name &quot;CustomScript&quot; \<br>  --version &quot;2.0&quot; \<br>  --ids &quot;$vmId&quot; \<br>  --protected-settings &quot;{\&quot;script\&quot;:\&quot;$script_content\&quot;}&quot;</pre><p>The script creates a resource group, deploys the virtual machine (and its accessories). Then it gets the id of the created machine, processes the script (encode &amp; gzip) and runs the custom extension with the script.</p><p>Creating a resource group for each worker is a little bit of overkill. I decided to do it so that I can remove the worker deployment easily. Unlike <a href="https://aws.amazon.com/cloudformation/">AWS Cloudformation</a>, Azure deployments don’t delete the resources created by them, which is hugely annoying. Luckily, I have a dedicated subscription available, so it’s not a huge issue.</p><p>Almost done.</p><h4>Authentication</h4><p>What remains is the authentication. I use PAT to register the worker because the other authentication methods seem too complex. I started with creating my PAT token in the UI and fed that using environment variables to the code. This is annoying because the token should expire and also it’s my personal <a href="https://www.youtube.com/watch?v=712GnEJHn_Y">secret (of Arabia)</a>.</p><p>Luckily, it is also possible to use the authorization of the pipeline job itself. While the job authorization is quite <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/access-tokens?view=azure-devops&amp;tabs=yaml#job-authorization-scope">complex with many options</a>, I ended up using the System.accessToken <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&amp;tabs=yaml#systemaccesstoken">built-in environment variable</a>. The system access token authorizes the job as a corresponding “build service”. The build service initially has no access to the agent pool so it is not able to add workers to it. This can be fixed by assigning the respective permissions in the agent pool (“Default” in my case).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GtxeQPTNwM7lyaT0-JD-Xg.png" /></figure><p>I’m not jumping for joy about giving the job an administrative role, but the project in which the pipeline job runs is a separate project with only two pipelines to create and delete the worker. So I accepted it (at least before I manage to have managed identity up and running or use the <a href="https://devblogs.microsoft.com/devops/personal-access-token-lifecycle-apis-now-publicly-available/">Personal access token API</a> which just became available).</p><p>Now the question is how to get the PAT to the script inside the machine. The custom script extension doesn’t have any variables. But luckily, some bash magic can solve it:</p><pre>envsubst &#39;${PAT_TOKEN} ${$WORKER_NAME}&#39; &lt; startup.sh &gt; startup-replaced.sh<br>script_content=$(cat startup-replaced.sh | gzip -9 | base64 -w 0)</pre><p>That essentially bakes the token (and the worker name) into the script right before it is run. I’m sending the script content inside the protected-settings parameter of the vm extension set command, so it gets <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux#property-value-details">encrypted</a>.</p><p>The thing to remember about envsubst is that if you don’t list the parameters, it will completely cripple the bash script, because it’ll<strong> replace all variables in it</strong>. Don’t even ask how much time it took before it occurred to me.</p><p>With this I have another bash script ready (did I say I try to avoid bash?) that deploys the Azure resources and runs the initializing script on the machine (which installs and configures the agent). The final version can be <a href="https://github.com/keboola/pipeline-worker/blob/main/run-startup.sh">seen here</a>.</p><p>I can now put this into an azure-pipelines.yml definition:</p><pre>pr: none<br>trigger: none<br>pool:<br>  vmImage: ubuntu-latest</pre><pre>stages:<br>  - stage: deploy<br>    displayName: &#39;Deploy Worker&#39;<br>    jobs:<br>      - job: deployWorker<br>        steps:<br>          - task: AzureCLI@2<br>            inputs:<br>              azureSubscription: &#39;XXXXX&#39;<br>              scriptType: &#39;bash&#39;<br>              scriptLocation: &#39;scriptPath&#39;<br>              scriptPath: &#39;run-startup.sh&#39;<br>            env:<br>              SUBSCRIPTION: $(SUBSCRIPTION)<br>              WORKER_NAME: $(WORKER_NAME)<br>              ADMIN_USERNAME: $(ADMIN_USERNAME)<br>              ADMIN_PASSWORD: $(ADMIN_PASSWORD)<br>              PAT_TOKEN: $(System.AccessToken)</pre><p>Then I just configure a new build pipeline with the above definition.</p><h3>Hurdles</h3><p>The VM <a href="https://github.com/keboola/pipeline-worker/blob/main/startup.sh">setup bash script</a> needs to restart the machine, because it changes GRUB. But restarting the machine needs to be done “asynchronously”, so that the custom script has a chance to finish successfully (and consequently, the deployment finishes successfully). A one-minute delay is sufficient:</p><pre>sudo shutdown -r +1 &quot;Rebooting.&quot;</pre><p>Without the delay the deployment fails (and yet everything is set up correctly). That took me a while to figure out. The delay means that the pipeline worker behaves slightly strangely — it’s up when the pipeline finishes, then it goes down and up again. That could be a problem if something else was hooked to the end of the pipeline run (e.g., starting tests on that worker), but I’m not doing that.</p><p>Having the custom script as a sub-resource of the VM resource seems convenient at first, but the errors are really unreadable. Often I needed to go to the Azure Portal UI to read the logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXOCX--i_0JDv5Lw5R630w.png" /></figure><p>Which still is not the hallmark of readability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Lg0wKCqnwh8lzOLWRbuGw.png" /></figure><p>Also, when the script is part of the VM resource, it takes much longer to execute the deployment (logically, because all the VM properties are checked). Having the custom script run separately from the VM template proved to be useful for faster iterations.</p><p>I also added a couple of safeguards to the script. The worker name (and resource group name) is entered as a user-overridable parameter of the pipeline, so I added a check:</p><pre>if [[ &quot;$WORKER_NAME&quot; != &quot;pipeline-&quot;* ]] ;<br>then<br>   printf &quot;&#39;%s&#39; is not a valid worker name (must match &#39;pipeline-*&#39;).&quot; &quot;$WORKER_NAME&quot;<br>   exit 1<br>fi</pre><p>to make sure that the pipeline does not interfere with some random resource group so as not to inadvertently change something unwanted. I also added a check for an already running worker to the setup script:</p><pre>if [[ -d &quot;/datadrive&quot; ]]<br>then<br>    echo &quot;Worker is already running&quot;<br>    exit 1<br>fi</pre><p>This simply checks if my custom directory is mounted. While primitive, it takes care of many possible failure scenarios. Generally, it allows retrying the VM creation part and avoids retrying anything that breaks when the machine is already running.</p><p>And last — even short scripts inside the pipeline YAML are a huge pain. There is no way to reasonably debug the script. The printf command doesn’t seem to work properly (echo does). But in either case, any mistake means to trigger the pipeline again, wait and see some other awkward error. In general, each debugging iteration is really slow. So, the best solution I found was to wrap everything in bash scripts (did I mention I try to avoid coding in bash?).</p><h3><strong>Shutdown</strong></h3><p>While I was at it, I also created a pipeline to remove a worker. This is pretty easy — use a Custom script extension to run a bash script (did I mention that I really try to avoid bash scripts?) that uninstalls the agent service and deregisters the worker.</p><pre>#!/usr/bin/env bash<br> set -Eeuo pipefail</pre><pre>printf &quot;\nUninstall Agent&quot;<br> cd /home/testadmin/azagent<br> # intentionally allow both things to fail<br> sudo ./svc.sh uninstall || true<br> printf &quot;cd /home/testadmin/azagent &amp;&amp; ./config.sh remove --unattended --auth pat --token $PAT_TOKEN&quot; &gt; ./wrap.sh<br> sudo chmod a+x ./wrap.sh<br> runuser -l testadmin -c &#39;/home/testadmin/azagent/wrap.sh&#39;</pre><pre>printf &quot;\nFinished successfully&quot;</pre><p>Intentionally, I allow both things to fail. The worst that can happen is that I’ll be left with a dangling worker in the agent pool.</p><p>Then I need to run this script in the VM. For that I can again use the Custom script extension. It’s useful to have the custom script as a separate resource to the VM itself as before.</p><pre>vmId=$(<br>   az deployment group show \<br>     --name &quot;$WORKER_NAME&quot; \<br>     --resource-group &quot;$WORKER_NAME&quot; \<br>     --subscription &quot;$SUBSCRIPTION&quot; \<br>     --query &quot;properties.outputs.vmId.value&quot; \<br>     --output tsv<br> )</pre><pre>envsubst &#39;${PAT_TOKEN}&#39; &lt;  shutdown.sh &gt; shutdown-replaced.sh<br> script_content=$(cat shutdown-replaced.sh | gzip -9 | base64 -w 0)</pre><pre>az vm extension set \<br>   --publisher &quot;Microsoft.Azure.Extensions&quot; \<br>   --name &quot;CustomScript&quot; \<br>   --version &quot;2.0&quot; \<br>   --ids &quot;$vmId&quot; \<br>   --protected-settings &quot;{\&quot;script\&quot;:\&quot;$script_content\&quot;}&quot;</pre><pre>printf &quot;Deleting deployment %s\n&quot; &quot;$WORKER_NAME&quot;</pre><pre>az group delete \<br>   --name &quot;$WORKER_NAME&quot; \<br>   --subscription &quot;$SUBSCRIPTION&quot; \<br>   --yes</pre><p>The bash script takes the worker name and PAT token environment variables, runs the custom script to deregister the agent and then deletes the entire resource group. Then I can run this bash script from a build pipeline. It’s very similar to the startup pipeline. All the code is available in <a href="https://github.com/keboola/pipeline-worker">the repository</a>.</p><h3>The End</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDHtANfyWbEpihySQZnySA.png" /></figure><p>The remove pipeline requires entering a worker name, so I made that an empty variable with allowed user override. I have to set it before running the pipeline (the script checks so):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*sFpbR8BHNmcHAuhooZUxyQ.png" /></figure><p>The create pipeline has a default value, so it is now a matter of <strong>one click</strong> to create a self-hosted pipeline worker.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d47d830ace" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/automating-self-hosted-pipelines-part-2-8d47d830ace">Automating Self Hosted Pipelines — Part 2</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Self Hosted Pipelines — Part 1]]></title>
            <link>https://medium.com/keboola-nerds/automating-self-hosted-pipelines-part-1-a635f154f021?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/a635f154f021</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-devops-pipeline]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Ondřej Popelka]]></dc:creator>
            <pubDate>Sun, 18 Apr 2021 09:07:24 GMT</pubDate>
            <atom:updated>2021-04-25T20:38:42.898Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Self-Hosted Pipelines — Part 1</h3><p>As part of our movement <a href="https://ondrej-popelka.medium.com/flee-from-travis-fd950e5ccea7">away from Travis CI</a> to Azure Pipelines, I came across a repository which needs to run docker commands with <a href="https://docs.docker.com/config/containers/resource_constraints/">swap limiting capabilities</a>. It is included in our application which runs docker containers and is in fact an important feature enabling the application developer to specify some of the container limits. When testing, we’re also testing the memory limiting some docker containers, which understandably does not really work on <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&amp;tabs=browser#microsoft-hosted-agents">Microsoft-Hosted Azure Pipelines agents</a>. It does not allow such low level operations:</p><pre>Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.</pre><p><em>This is a two-part article. First, I’ll describe how to automate the worker virtual machine setup using a </em><a href="https://github.com/keboola/pipeline-worker/blob/main/startup.sh"><em>Bash script</em></a><em>. In the </em><a href="https://500.keboola.com/automating-self-hosted-pipelines-part-2-8d47d830ace"><em>next part</em></a><em>, I’ll describe how to automate the worker setup even further using Azure Pipelines.</em></p><h3>Self- vs Microsoft-hosted Agent</h3><p>As I didn’t want to skip the pesky tests, I decided to run them using a <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&amp;tabs=browser#install">self-hosted Azure pipeline agent</a> since it allows a great degree of customization. Setting up the self-hosted agent proved to be <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops">fairly straightforward</a> and I kept using it for some months. There are a couple of differences between the self-hosted and Microsoft-hosted agent. First and foremost, the self-hosted agent is somewhat <strong>stateful</strong>. I run all of our tests inside containers so that makes this both an advantage and disadvantage. The advantage is speed — using a stateful machine means that the base image layers are cached. So when using a <a href="https://github.com/keboola/job-queue-api/blob/master/azure-pipelines.yml">pipeline like this</a>:</p><pre>stages:<br>  - stage: build_test<br>    jobs:<br>      - job: build<br>        steps:<br>          - script: &gt;<br>              docker-compose build<br>              docker-compose run --rm tests<br><br>  - stage: deploy_test<br>    jobs:<br>      - job: deploy<br>        steps:<br>          - task: Docker@2<br>            inputs:<br>              command: buildAndPush<br>              containerRegistry: $(acr_registry_pes)<br>              dockerfile: &#39;$(Build.SourcesDirectory)/Dockerfile&#39;<br>              repository: $(acr_repository)<br>              tags: |<br>                $(productionTag)<br>                $(buildTag)</pre><p>The buildAndPush command is almost instantaneous. While the docker-compose command builds a couple of images, the buildAndPush command just makes sure that the right image has the right tag.</p><p>At the same time, the statefulness can bite <strong>even with containerized tests</strong>. I basically ended up running the following commands before every tests:</p><pre>docker-compose down<br>docker-compose build --pull<br>docker-compose pull</pre><p>This ensures that any dependent images and containers are up to date. Yet I may still run into issues with, e.g., starting two MySQL servers in two containers and both listening on the same port.</p><p>Apart from that, there are also management and cost differences. For the self-hosted pipelines, I have to pay for the running machine instead of only the consumed runtime. I approached this by putting more AZ pipelines (I have over 30 of them now) to the self-hosted agent pool so that it has some basic usage. The pipelines using self-hosted agents obviously also offer a smaller degree of parallelism. Then there is a huge disadvantage (in my opinion) to self-hosted pipelines — <strong>I have to take care of them</strong>.</p><p>To conclude — both types of agents have their advantages and both make sense in given circumstances. What is super cool is that I can combine both in a single pipeline, because the pool option <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&amp;tabs=schema%2Cparameter-schema#pool">can be specified</a> on the job level. That means I can take the best from both worlds. I can run a single job of a pipeline on my machine with “obscure” settings and take advantage of higher parallelism using the Microsoft-hosted pipeline.</p><pre>- stage: Tests<br>    dependsOn: beforeTests<br>    jobs:<br>      - job: baseTests<br>        displayName: &#39;Base Tests&#39;<br>        steps:<br>          - script: |<br>              docker-compose run tests ./vendor/bin/phpunit <br>                               --testsuite base-tests</pre><pre>     - job: containerTests<br>        displayName: &#39;Container Tests&#39;<br>        pool: # Run on self-hosted worker<br>          name: &#39;Default&#39;<br>        steps:<br>          - script: |<br>              docker-compose run host-tests ./vendor/bin/phpunit <br>                               --testsuite container-tests</pre><p>In the above azure-pipelines.yml snippet, the baseTests job is run using the Microsoft-hosted agent, the containerTests job is run using the self-hosted agent, and both run in parallel.</p><h3>Setting up a self-hosted agent</h3><p>In my case, the main motivation for using the self-hosted agent is largely in the customization. To set up the agent, I basically just follow <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops">the instructions</a>:</p><ul><li>Have a virtual machine ready;</li><li>Do whatever setup is needed (in my case, install docker &amp; allow swap capabilities);</li><li>Generate a PAT (Personal Access Token);</li><li>Download and install the agent;</li><li>Set the pipeline to use the self-hosted agent pool.</li></ul><p>Done. The worker is running and there is no need to touch it.</p><p>Except that I realized that its disk fills up. The images we are building are fairly large, there are at least a dozen of them needed. So, the Standard D2s v3 with VM with a 30GB drive fills pretty quickly. So, from time to time, I have to login via the Serial console and do docker system prune --force --volumes. And then of course, do apt-get update -y &amp; apt-get upgrade -y. Also, don’t forget to update the agent, which is done in the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHYWdB0vGBDo94XAMBS7Ww.png" /></figure><p>Oh, screw that, can’t I just get rid of that worker and run a new one? (Oh my, that’s so insensitive to the workers’ feelings, no pension, just the <a href="https://reddwarf.fandom.com/wiki/Silicon_Heaven">silicon heaven</a>).</p><h3>Automation</h3><p>Running a new worker manually isn’t too bad — it’s like 20 minutes of work in the best case scenario. But it’s ~30 steps and if I miss one, it’s another 10 minutes delay. That’s a good candidate for some automation.</p><h4>VM Setup</h4><p>The first step is to prepare an <a href="https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/2019-03-01/virtualmachines?tabs=json">ARM template</a> to create a virtual machine. That’s pretty straightforward, click a machine through the UI, export the template, clean up all the parameters I don’t need, improve the naming of the resources and add tagging, remove unused parameters, add variables for resource names…</p><p>Next time, I’ll <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-ssh-secured-vm-from-template">do it from scratch</a>.</p><p>As a little f*up, I removed the public IP, but without it the VM <a href="https://social.msdn.microsoft.com/Forums/azure/en-US/54c9b3de-7bb9-4759-8178-89c273b55caf/how-does-azure-vm-connect-to-the-internet">can’t access the internet</a> (or you have to set the Load Balancer). This is in the end quite logical, but didn’t really occur to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*nJsXSTF5mmHvYF3I" /></figure><p>Anyway.</p><p>The second step is to attach a data disk. This is surprisingly difficult (or I’m missing something obvious) as shown by <a href="https://raw.githubusercontent.com/Azure-Samples/compute-automation-configurations/master/prepare_vm_disks.sh">the sample script</a> referenced in the <a href="https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/scripts/cli-sample-attach-disks">tutorial</a>.</p><p>Not being a bash guru, I don’t really like using huge bash scripts I don’t understand. In general, I try to avoid bash scripts completely — so I wrote one for this myself (masochism?). Turned out that I could simplify this. In the beginning, I even started with a hardcoded disk as sdc, which worked reasonably well and failed only twice in 10 attempts :) But it’s not the correct way, since the disk names are not deterministic. <a href="https://singhkays.com/blog/how-to-identify-azure-linux-vm-disks/">An article</a> which I stumbled upon gave me the idea to use LUN (Logical Unit Number) to identify the disk.</p><p>So, I assigned LUN=10 to the datadisk in the ARM template:</p><pre>&quot;dataDisks&quot;: [<br>    {<br>        &quot;lun&quot;: 10,<br>        &quot;name&quot;: &quot;[variables(&#39;dataDiskName&#39;)]&quot;,<br>        &quot;createOption&quot;: &quot;Attach&quot;,<br>        &quot;caching&quot;: &quot;ReadOnly&quot;,<br>        &quot;managedDisk&quot;: {<br>            &quot;storageAccountType&quot;: &quot;StandardSSD_LRS&quot;,<br>            &quot;id&quot;: &quot;[resourceId(&#39;Microsoft.Compute/disks/&#39;, variables(&#39;dataDiskName&#39;))]&quot;<br>        },<br>        &quot;diskSizeGB&quot;: 256<br>    }<br>],</pre><p>And then ended up with the following script piece:</p><pre># pick up the disk identified by LUN specified in the template<br>disk_record=$(lsblk -o HCTL,NAME | grep &quot;:10&quot;)<br>disk_name=$(printf &quot;%s&quot; &quot;$disk_record&quot; | <br>        gawk &#39;match($0, /([a-z]+)/, a) {print a[1]}&#39;)<br><br>sudo parted &quot;/dev/$disk_name&quot; <br>        --script mklabel gpt mkpart xfspart xfs 0% 100%<br>part_name=$(printf &#39;%s1&#39; &quot;$disk_name&quot;)<br>sudo mkfs.xfs -f &quot;/dev/$part_name&quot;<br>sudo partprobe &quot;/dev/$part_name&quot;<br><br>block_record=$(sudo blkid | grep &quot;xfs&quot;)<br>uuid=$(printf &quot;%s&quot; &quot;$block_record&quot; | <br>        gawk &#39;match($0, /UUID=&quot;([a-zA-Z0-9\-]*)&quot;/, a) {print a[1]}&#39;)<br>printf &quot;UUID=%s /datadrive xfs defaults,nofail 0 2\n&quot; &quot;$uuid&quot; | <br>        sudo tee -a /etc/fstab<br>sudo mkdir /datadrive<br>sudo mount /datadrive</pre><p>The script finds the disk_name of a disk with LUN 10, partitions (part_name), formats the partition, finds the UUID of the partition and adds it to fstab so that it’s mounted. It also mounts it manually so that the rest of the setup can continue without restarting the VM.</p><p>The next step is to install Docker Engine and Docker Compose. This is pretty straightforward — just follow the steps in the docs for <a href="https://docs.docker.com/engine/install/ubuntu/">Docker Engine</a> and <a href="https://docs.docker.com/compose/install/">Docker Compose</a>. The only change is that I wanted docker to use the data drive. This is easily done by the following setting (sudo tee is just to append to a protected file):</p><pre>printf &quot;{\&quot;data-root\&quot;: \&quot;/datadrive/docker\&quot;}&quot; | <br>        sudo tee -a /etc/docker/daemon.json</pre><p>Now Docker has plenty of space.</p><p>The next step is enabling swap capabilities, again fairly straightforward according to <a href="https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities">the docs</a>:</p><pre>printf &quot;\nGRUB_CMDLINE_LINUX=\&quot;cgroup_enable=memory swapaccount=1\&quot;\n&quot; | <br>       sudo tee -a /etc/default/grub<br>sudo update-grub</pre><p>Adding this single line to the GRUB config is one of the main reasons I’m doing all this :).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Bx759w5jXRnsYYHJ.jpg" /></figure><h4>Agent Setup</h4><p>Having the VM ready, I need to install and enable the Azure Pipelines agent so that it can be used to run pipeline jobs. This again is fairly straightforward <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops">following the guide</a>. A little problem is that it has to be run under a non-root user and it needs a PAT (Personal Access Token). The Personal access token can be created in the user settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXffsSjnhKwiUevN1sIFuw.png" /></figure><p>The agent setup requires two environment variables — worker name (as it would appear in the Azure pipelines agent pool) and the above created PAT token with manage permissions on the pool so that it can be used to register the pipeline agent. I got a little problem running runuser and passing the environment values around, so I created a little wrap.sh as needed and didn’t really bother to find out (did I mention that I really try to avoid bash?).</p><pre>cd /home/testadmin<br>curl -s https://vstsagentpackage.azureedge.net/agent/2.183.1/vsts-agent-linux-x64-2.183.1.tar.gz -o agent.tar.gz<br>mkdir azagent &amp;&amp; cd azagent<br>tar zxvf ./../agent.tar.gz &gt; /dev/null<br>sudo chown testadmin /home/testadmin/azagent/<br>printf &quot;/home/testadmin/azagent/config.sh <br>    --replace --acceptTeeEula --unattended <br>    --url https://dev.azure.com/keboola-dev/ <br>    --auth pat --token $PAT_TOKEN --pool default <br>    --agent $WORKER_NAME --work /datadrive/_work&quot; &gt;<br>    /home/testadmin/azagent/wrap.sh<br>sudo chmod a+x /home/testadmin/azagent/wrap.sh<br>runuser -l testadmin -c &#39;/home/testadmin/azagent/wrap.sh&#39;</pre><p>Then I need to make sure that the agent uses the designated datadrive and starts automatically:</p><pre>sudo mkdir /datadrive/_work<br>sudo chown testadmin /datadrive/_work<br>sudo /home/testadmin/azagent/svc.sh install</pre><p>All done. Now I have to restart the machine so that the GRUB changes take effect. Also, it’s good to check that I haven’t messed anything up, the disk is auto-mounted and the agent starts automatically.</p><p>Finally, I ended up with a nice bash script (final version <a href="https://github.com/keboola/pipeline-worker/blob/main/startup.sh">available on Github</a>) that takes two environment variables: a token and a worker name and sets up the pipeline worker. Quite ok, although I still don’t like bash. In the next part, I’ll show how I made this a one-click operation in the Azure Pipelines UI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a635f154f021" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/automating-self-hosted-pipelines-part-1-a635f154f021">Automating Self Hosted Pipelines — Part 1</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practices for  Azure DevOps (Release) Pipelines]]></title>
            <link>https://medium.com/keboola-nerds/best-practices-for-azure-devops-release-pipelines-8bc5ec309da?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/8bc5ec309da</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Ondřej Hlaváček]]></dc:creator>
            <pubDate>Tue, 02 Mar 2021 19:22:36 GMT</pubDate>
            <atom:updated>2021-03-02T19:22:36.767Z</atom:updated>
            <content:encoded><![CDATA[<h3>Best Practices for Azure DevOps (Release) Pipelines</h3><p>As a part of our <a href="https://medium.com/p/d75878fd8b3e">Azure adoption</a>, we&#39;re also trying out <a href="https://azure.microsoft.com/en-us/services/devops/">Azure DevOps</a>. Our current CI/ CD pipelines run almost exclusively on <a href="https://travis-ci.com/">Travis CI</a>. We didn’t want to start rewriting the existing code, so we decided to append Azure DevOps to the existing processes and only use them for the new, Azure related stuff. Out of the five DevOps offerings (Boards, Repos, Pipelines, Test plans, and Artifacts), we’re only using Release Pipelines to deploy and update the application and its Azure infrastructure. The rest of the CI/CD process remains intact so far.</p><p>There are two kinds of pipelines in Azure DevOps — Build and Release. Two main differences (as I perceive them) between them are definitions and triggers.</p><p><strong>Definitions</strong> — Build Pipelines are defined and stored in YAML, and can (most likely) be stored in the repository along with the application code (like Travis CI). Release Pipelines, on the other hand, can only be defined in the Azure Devops UI. They can be exported to JSON (and imported back), but that&#39;s as far as their “pipeline-as-a-code” approach goes. Things are likely to change, so I&#39;m not worried about it right now, but it would be nice to have the definition along with the code and be able to update the Release Pipeline with a commit in a repository.</p><p><strong>Triggers </strong>— Build and Release definitions have a different set of triggers. The main difference for us is the <a href="https://azure.microsoft.com/en-us/services/container-registry/">Azure Container Registry</a> trigger, which is only available in Release Pipelines. (I am actually taking a shortcut here; it should be theoretically possible to trigger a Build Pipeline from ACR using this <a href="https://github.com/microsoft/azure-pipelines-yaml/blob/master/design/pipeline-triggers.md">undocumented feature</a>, but I wasn&#39;t successful in <a href="https://github.com/microsoft/azure-pipelines-yaml/issues/270">making it work</a>.)</p><p>During the months spent in Release Pipelines DevOps, we came up with a few tips we&#39;d like to share with you.</p><h3>Scripts first</h3><p>We develop all parts of the Release Pipeline as small Bash scripts that accept “arguments” in the form of ENV variables. The Release Pipeline then consists of a series of Bash, <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest">Azure CLI</a> and <a href="https://aws.amazon.com/cli/">AWS CLI</a> tasks. We can easily run these scripts locally or in a different CI/CD tool. We try to keep these scripts small; that makes debugging easier. Plus we can run only some of them in certain scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*WuGbo-rRS0_W93Psp2jPdA.png" /></figure><h3><strong>Infrastructure as a code</strong></h3><p><a href="https://azure.microsoft.com/en-us/resources/templates/">ARM templates</a> are a great way to define Azure infrastructure (again with some exceptions like Active Directory and some networking stuff) and we use them to create and update infrastructure parts of the application. The infrastructure definition stays in our codebase and we use Azure CLI tasks to deploy it.</p><pre>az group deployment create \<br>  --name connection-mysql \<br>  --resource-group $RESOURCE_GROUP \<br>  --template-file ./connection/azure/resources/mysql-db.json \<br>  --parameters keboolaStack=$KEBOOLA_STACK \<br>    administratorLogin=keboola \<br>    administratorLoginPassword=$PASSWORD \<br>    aksClusterVnetName=$VNET \<br>    aksNodeResourceGroup=$NODE_RG \<br>    skuName=$MYSQL_SKU_NAME \<br>    skuSizeMB=$MYSQL_SKU_SIZE_MB\<br>    backupRetentionDays=7</pre><h3><strong>Do not use (overly) specialized tasks</strong></h3><p>They are simple, and good for a PoC, but that’s it. Complex operations, debugging, passing variables, parametrization or execution control are almost impossible. You have to hardcode resource names directly into the task, which makes change management impossible. We tried the <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops">Kubectl task</a>, but after a short while we have turned that into simple Bash scripts running kubectl commands manually, which gave us so much more control.</p><pre>kubectl apply -f ./provisioning/kubernetes/deploy/connection-api.yaml<br>kubectl rollout status deployment/connection-api --timeout=900s</pre><p>This Bash script is so much easier to understand than two Kubectl tasks with dozens of input fields in DevOps. The deploy process definition also stays close to your code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*yGgxKf2s5OakjT6pIfhNXw.png" /></figure><h3><strong>Use dynamic resource names and IDs</strong></h3><p>Except for <a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/overview">ARM Template Deployment</a> names. We assume that each instance of our application (e.g. a region) lives in a separate subscription. Unique name requirements vary greatly among resources — some must be unique within the whole Azure platform, while some only have to be unique within a single region in a single subscription. We ended up suffixing all resource names with hashed strings containing a stack name, region and few other variables to satisfy the uniqueness constraints. Thanks to that it’s, quite difficult to identify resources for a given instance. To avoid confusion (at least the programmatic kind) we&#39;ve embraced ARM Template Deployments in Azure and now store all required names and IDs in the deployment outputs. Using Azure CLI, you can easily extract a required variable value from an existing deployment. These scripts typically have very few arguments.</p><pre><br>{<br>...<br>  &quot;variables&quot;: {<br>    &quot;clusterName&quot;: &quot;[concat(&#39;connection-&#39;, uniqueString(parameters(&#39;keboolaStack&#39;)))]&quot;<br>  },<br>...<br>  &quot;outputs&quot;: {<br>    &quot;clusterName&quot;: {<br>      &quot;type&quot;: &quot;string&quot;,<br>      &quot;value&quot;: &quot;[variables(&#39;clusterName&#39;)]&quot;<br>    }<br>  }<br>}</pre><p>The above snippet from the ARM template stores the variable in the Template Deployment output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29DRrbA8CdsEIpb3qBM9-Q.png" /></figure><p>From there, it can be obtained and used further down in any other script.</p><pre>CLUSTER_NAME=`az group deployment show \<br>  --resource-group $RESOURCE_GROUP \<br>  --name connection-aks \<br>  --query &quot;properties.outputs.clusterName.value&quot; \<br>  --output tsv`</pre><p>If this is a part of a larger script, it has only one argument, RESOURCE_GROUP, and all other variables are dynamically read from their deployments or resources.</p><h3><strong>Variables are great, but one way only</strong></h3><p>Pipeline variables map directly to ENV variables in Bash, so the ACR Name variable is available as ACR_NAME env. We use variables to configure the application and its parameters (e. g., nodes count and database size) but never to store any state. Another huge benefit of variables is that they can store different values for each stage.</p><p>Though it’s possible for a Pipeline task to write to a variable, it&#39;s <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&amp;tabs=yaml%2Cbatch#set-variables-in-scripts">quite difficult</a> and we&#39;ve never done it successfully. If you need to store something for later use, use appropriate resources — like <a href="https://azure.microsoft.com/en-us/services/key-vault/">Key Vault</a> for passwords and other secrets and Template Deployment outputs for names of resources, or use Azure CLI to obtain the values directly from the resources when required. Subnet ID is a bad variable, number of nodes of an Azure Kubernetes Service Node Pool is a good variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vu3jZMk8dfG7pfgnPBJA_w.png" /></figure><h3><strong>Do not store secrets in Pipeline Variables</strong></h3><p>Use Key Vaults. It’s easy to operate from within Azure CLI.</p><pre>AZURE_RESOURCE_GROUP=myAzureResourceGroup</pre><pre>AZURE_KEY_VAULT_NAME=$(az group deployment show \<br>  --name myKeyVaultDeployment \<br>  --resource-group $AZURE_RESOURCE_GROUP \<br>  --query &quot;properties.outputs.myKeyVault.value&quot; \<br>  --output tsv \<br>)</pre><pre>az keyvault secret set \<br>  --name mySecret \<br>  --vault-name $AZURE_KEY_VAULT_NAME \<br>  --value &quot;my secret value&quot;</pre><p>The above script obtains a Key Vault name from a deployment and stores a secret in it. Retrieving secret values is very similar.</p><pre>az keyvault secret show \<br>  --name mySecret \<br>  --vault-name $AZURE_KEY_VAULT_NAME \<br>  --query value \<br>  --output tsv</pre><h3><strong>Evolve the toolkit</strong></h3><p>While Bash scripts and ARM templates can take us a long way, we need to be aware of their limitations and start thinking about expanding our toolbox. Some situations will require using Terraform for infrastructure or a proper programming language to handle deployment of our app. Currently, we still stick to Bash and ARM templates, but we know they won&#39;t be there forever. Something will replace them sooner or later.</p><h3>What&#39;s next?</h3><p>We start with baby steps and deploy working Release Pipelines as soon as possible to make our lives easier. But we have a long road ahead of us. So where are we heading?</p><ul><li><strong>YAML</strong> — Store the Release Pipeline as code within the repository.</li><li><strong>Scaling up</strong> — Currently we have only two stages (testing and production), but we&#39;ll soon need to have expand this to many production environments. Our nightmare is having to set up each new environment manually. This has to be automated somehow.</li><li><strong>Organization </strong>— We&#39;ve piled up all Release Pipelines randomly across two projects, one called Dev and the other Prod (ahem). We&#39;re not focused on making sense of it right now, but we&#39;ll need to do that soon.</li><li><strong>Safe development </strong>— How do we change the Release Pipeline without possibly breaking production? What measures do we need to take to achieve that?</li><li><strong>Governance</strong> — Who can have access, and where? This is a wide topic spanning from our developers to our customers&#39; subscriptions and Azure DevOps in between. Everyone has to feel safe.</li></ul><h3>What about you?</h3><p>What are your best practices for using Azure DevOps? What is the single piece of advice you wish you had known from day one? What&#39;s your biggest headache? Are we wrong? Share with us; we&#39;re here to learn! Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bc5ec309da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/best-practices-for-azure-devops-release-pipelines-8bc5ec309da">Best Practices for  Azure DevOps (Release) Pipelines</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Pre-Stop Hook Reflections]]></title>
            <link>https://medium.com/keboola-nerds/kubernetes-pre-stop-hook-reflections-26dd2c69920f?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/26dd2c69920f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[monolog]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Ondřej Popelka]]></dc:creator>
            <pubDate>Fri, 19 Feb 2021 16:21:56 GMT</pubDate>
            <atom:updated>2021-02-18T09:10:33.712Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes PreStop Hook Reflections</h3><p>I have an application which runs as a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/">Kubernetes job</a>. The application is implemented as a PHP Symfony <a href="https://symfony.com/doc/current/console.html">console command</a>, which is specified in the Dockerfile entrypoint:</p><pre>FROM php:7.4-cli<br>ENV COMPOSER_ALLOW_SUPERUSER 1<br>WORKDIR /code</pre><pre>RUN curl -sS <a href="https://getcomposer.org/installer">https://getcomposer.org/installer</a> | php -- --install-dir=/usr/local/bin/ --filename=composer<br>COPY . .<br>RUN composer install $COMPOSER_FLAGS</pre><pre>CMD php /code/bin/console app:run</pre><p>When the image is built and the container starts, the PHP process runs a console command app:run. When the command terminates, the container terminates and the Kubernetes job finishes. So far, nothing special.</p><h3>Terminating the job</h3><p>The next thing on my list was to implement termination of the job. This is actually trivial to do through kubectl (or the corresponding API):</p><pre>&gt; kubectl delete job job-674602133<br>job.batch &quot;job-674602133&quot; deleted</pre><p>That’s all. However, here comes the interesting part. The app:run command process spawns some other processes which need to be cleaned manually if the command does not finish gracefully. In fact, these are other running docker containers, but that’s for another long story.</p><p>If it were an API, I would send some termination request to it and asynchronously and gracefully terminate the application. But since the job is a console command, all I can do is send a <a href="http://&gt; kubectl delete job job-674602133 --namespace dev-job-runner job.batch &quot;job-674602133&quot; deleted">SIGTERM</a> signal to the process. I quite hate working with process signals (especially inside PHP), so I was trying to avoid this a bit.</p><p>Luckily, there is another option, which is related to the Kubernetes <a href="http://&gt; kubectl delete job job-674602133 --namespace dev-job-runner job.batch &quot;job-674602133&quot; deleted">pod termination</a>. Kubernetes offers to configure a <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks">PreStop hook</a> which runs before the pod termination starts. Kubernetes even allows to run a CLI command (together with the obvious HTTP request option), which is great for my console application. It is worth noting that it does not behave exactly the same way as sending and reacting to a SIGTERM/SIGKILL signal to the main container process. The PreStop hook starts another command in the same container (using docker exec). That means I end up with two unrelated processes — one is running the container and the other one is “terminating” the container.</p><p>This behavior is as designed and quite logical considering the fact that the PreStop hook is intended, for example, to save the state of the running pod. It has to run before the main process starts terminating. In my case, I end up having theapp:run and app:cleanup commands running simultaneously. This is sort of weird, but not a problem, fortunately. It is the price for abusing the hook.</p><h3>Hook Configuration</h3><p>The official Kubernetes docs explains, in a lot of detail, how <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks">PreStop hooks</a> work. But the information on how to actually configure them seems to be missing :]. Luckily, someone else has already <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks">found out</a> and also detailed how to configure the <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks">grace period</a>. With that help, I ended up with the following template (abbreviated):</p><pre>template:<br>    spec:<br>      containers:<br>        - name: job-runner<br>          image: my-image:my-tag<br>          resources:<br>            ...<br>          env:<br>            ...<br>          lifecycle:<br>            preStop:<br>              exec:<br>                command: [&#39;php&#39;, &#39;/code/bin/console&#39;, &#39;app:cleanup&#39;]<br>      terminationGracePeriodSeconds: 60</pre><p>Piece of cake and it works great. Except that it doesn’t log anything. This is even mentioned in the <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-details">docs</a>.</p><blockquote>The logs for a Hook handler are not exposed in Pod events.</blockquote><p>Which, for me, means that it does not work. At <a href="https://www.keboola.com/">Keboola</a>, we are generally quite obsessed with traceability, and this is no exception. The cleanup command removes some running processes and we absolutely have to know that this has happened. We trace all outputs of all pods in <a href="https://www.keboola.com/">Papertrail</a>, so I wanted to see in the pod output that the pod was terminated and what the cleanup command did before this happened.</p><h3>Logging</h3><p>Now comes the hard part that took me a bit of digging and experimenting with some dead ends. I was finally led down the right track when reading <a href="https://stackoverflow.com/questions/42958854/docker-run-docker-exec-and-logs/42959071#42959071">this largely unrelated post</a> on StackOverflow.</p><p>When Kubernetes is running a container, it captures the pod outputs using a <a href="https://docs.docker.com/config/containers/logging/configure/">logging driver</a>. That means that the process <a href="https://en.wikipedia.org/wiki/Standard_streams">stdout</a> is redirected to the socket of the logging driver. This can be easily verified — open a terminal within the pod and run ps -ef:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gzYueCgtLS3ZjWFUMqjXsA.png" /><figcaption>Process list</figcaption></figure><p>Let’s ignore the garbage now and concentrate on the important part which is the process with PID 1. It’s a shell command because I am using the <a href="https://stackoverflow.com/questions/47904974/what-are-shell-form-and-exec-form">shell form</a> of the CMD in Dockerfile, but that’s not important. In either case this is the main container process and the one I’m interested in. Now let’s run ls -l /proc/1/fd which lists the details of the main process <a href="https://man7.org/linux/man-pages/man5/proc.5.html">file descriptors</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*_mvuUZucJeZFKyvwFfUsbQ.png" /><figcaption>Standard output and standard error redirection</figcaption></figure><p>Indeed — both standard output and error output are redirected. When the cleanup process is run using the PreStop hook (using docker exec), its output is not redirected. That could be verified using the same commands.</p><p>The solution is “easy” — redirect the output of the cleanup process to the same pipe used by the main process. The pipe is unknown, so it’s easier to redirect the cleanup process output of the output of the process with PID 1. To test this, I can run the following inside the pod:</p><pre>echo &quot;test&quot; &gt; /proc/1/fd/1</pre><p>Indeed, this shows test in the logs of the main process and the container and the pod. The /proc/1/fd/1 means that I am writing to the stdout of the process with PID 1. Redirecting to /proc/1/fd/2 would <a href="https://man7.org/linux/man-pages/man5/proc.5.html">write to stderr</a>.</p><p>Now the only remaining task is force the PHP Symfony command to redirect its output. I’m also using <a href="https://man7.org/linux/man-pages/man5/proc.5.html">Monolog</a> configured so that every log output gets sent to standard output. So I was trying to avoid some hara-kiri interfering with the logger.</p><p>One option was to reconfigure the logger to open the /proc/1/fd/1 file stream and write to it. This failed consistently — looks like PHP is unhappy that it’s already open.</p><p>So I resorted to doing the redirection outside of the application itself. But passing the redirection in the command doesn’t work because the value gets escaped:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&#39;php&#39;, &#39;/code/bin/console&#39;, &#39;app:cleanup&#39;, &#39;1&gt;&amp;/proc/1/fd/1&#39;]</pre><p>The shell form does not seem to work either, because then I’d have to escape the redirection:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: &#39;php /code/bin/console app:cleanup 1&gt;&amp;/proc/1/fd/1&#39;</pre><p>I got tired and used the most reliable solution — wrap the entrypoint in a shell script:</p><pre>#!/usr/bin/env bash<br>set -e</pre><pre>php /code/bin/console app:cleanup 1&gt;&amp;/proc/1/fd/1</pre><p>which can be run via the PreStop hook without problems:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&#39;/code/cleanup.sh&#39;]</pre><p>Problem solved.</p><p>Now when the pod is deleted, Kubernetes fires the PreStop hook. This executes the app:cleanup Symfony command, which can use the standard Monolog methods such as this one:</p><pre>$logger-&gt;info(sprintf(&#39;Terminating process &quot;%s&quot;.&#39;, $myProcess));</pre><p>And that message appears in the log feed of the POD as if it were coming from the main process. This gives the perfect impression that when the pod terminates, the main process terminates gracefully.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26dd2c69920f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/kubernetes-pre-stop-hook-reflections-26dd2c69920f">Kubernetes Pre-Stop Hook Reflections</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hodil by se nám další buddy do party]]></title>
            <link>https://medium.com/keboola-nerds/hodil-by-se-n%C3%A1m-dal%C5%A1%C3%AD-buddy-do-party-cfff997787aa?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/cfff997787aa</guid>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[keboola]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Vojta Tuma]]></dc:creator>
            <pubDate>Thu, 10 Sep 2020 05:37:47 GMT</pubDate>
            <atom:updated>2020-09-11T06:35:30.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQBeg6CHL9c0CV8wFUIFIg.jpeg" /></figure><p>Napsat zajímavej pracovní inzerát v dnešní době je docela složitý, a proto jsme se rozhodli, že ho prostě psát nebudeme. Místo toho popíšeme jak to u nás chodí, protože to je stejně to, co vás zajímá.</p><p>Psal se rok 2010, když se Padák rozhodoval jak se bude naše firma jmenovat. Generátor slov mu vyhodil Keboola a nebo Megake. Volba nebyla zase tolik složitá! A tak se začala vyvíjet firma, která, jak popsal jeden z prvních zaměstnanců Halama, byla překydávátko CSVéček. Doba šla dál a nám se podařilo udělat úspěšný ETL nástroj, který pokryl velkou část E-commerce sektoru v Čechách. Od té doby jsme se dost posunuli a dneska provozujeme platformu pro celý datový proces od ETL, přes DataScience a automatizaci. Po Čechách jsme prorazili i na západním pobřeží v Americe. Aktuálně máme pobočku ve Vancouveru, kde máme 4 kolegy z professional services (oddělení starající se o implementace u zákazníků). Mezi zajímavý klienty v Americe patří například síť restaurací Firehouse, což je síť restaurací, která díky datům začala raketově růst (<a href="https://www.firehousesubs.com/All-Locations/">https://www.firehousesubs.com/All-Locations/</a>).</p><p>Rok a půl dozadu se nám začal otevírat svět enterprise klientů a my začali upravovat Keboolu tak, aby více vyhovovala i větším zákazníkům. S tím přišlo spousta requestů na novou funkcionalitu. Víme jak ty věci udělat, ale hodilo by se nám, aby nám s nima někdo pomohl.</p><p>Jelikož rozumíme datům a víme, jak stavět procesy kolem nich, tak jsme se letos na jaře rozhodli pomoct státu se řešením situace kolem Covid-19 a vytvořili jsme v Keboole technickou část systému, kterému se říká “Chytrá karanténa”. Máme kolem toho spoustu historek, protože práce se státní sférou je doopravdy “jiná”.</p><p>To co vlastně Keboola je trochu složitý na vysvětlení, ale máme o tom super video, kde to Odin vysvětluje <a href="https://www.youtube.com/watch?v=Oxy0JFbuzX8">https://www.youtube.com/watch?v=Oxy0JFbuzX8</a> . Já bych to popsal, že učíme lidi jak správně pracovat s datama a mít v nich uklizeno.</p><p>Celou dobu, co firma existuje, máme černý čísla a nemáme investora. Organicky rosteme okolo 80% za rok. Aktuálně se ale už po investorovi díváme, aby nám pomohl nejem rychleji růst, ale taky otevíral dveře v dalších velkých firmách. Gartner nás delší dobu neuměl zařadit do žádné škatulky až pro nás vymyslel vlastní kategorii a tou je “Data Operating Systems”. Ve firmě máme zavedený ESOP. Pokud nevíš, co ta zkratka znamená, tak ve zkratce to je, že máš právo na firemní akcie pokud u nás budeš delší dobu (a to my chceme).</p><p>Celý náš vývojový tým je různě v Čechách až na jednoho kolegu, který bydlí na Slovensku. Full remote máme v DNA a víme jak v něm pracovat. Jeden kolega půl roku cestoval kolem zeměkoule a normálně u toho pracoval. Pokud to ale není tvůj případ, tak klidně můžeš chodit do kanceláře v Praze (naše hlavní) nebo do Brna. Aby sme se ale líp poznali tak jezdíme 4x do roka na offsite, což je akce, kdy si pronajmeme chatu někde v přírodě a 3 dny diskutujeme všechno. Když tě něco trápí nebo bys chtěl něco změnit, tak tohle je to pravý místo, kde reprodukovat svoje nápady.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vNU9UKm82_JrwggGpq1dlQ.jpeg" /><figcaption>Offsite, kde právě vytváříme nově týmy</figcaption></figure><p>Jelikož jsme zatím všichni vývojáři z CZ nebo SK, tak mezi sebou mluvíme česky, ale s partnerama probíhá komunikace anglicky. Je teda potřeba, aby si uměl dobře anglicky psát, protože to budeš potřebovat v komunikaci s klientama, když budeš sedět na supportu.</p><p>Hodně aktivně spolupracujeme s Microsoftem, protože stavíme Keboolu nad Azure (doteď byla pouze nad AWS). Tam máme přístup ke spoustě profíků, kteří nám pomáhají urychlovat vývoj. Budeme s nimi dlouhodobý strategický partnerství, kdy nás chtějí u svých klientů používat jako datovou platformu.</p><figure><img alt="Microsfot nás pozval na OpenHack konferenci do Paříže" src="https://cdn-images-1.medium.com/max/1024/1*T1Yw3j-CjFF7OG38PF5yrg.jpeg" /><figcaption>Microsoft nás pozval na OpenHack konferenci do Paříže</figcaption></figure><p><strong>Tady začínají důležitý fakta.</strong> Ve vývoji nás je aktuálně 18 (a ve zbytku firmy cca 50) a jsme rozdělený do 6 týmů</p><ul><li><strong>Connection </strong>— řeši jádro produktu a hlavní API na ukládání dat a metadat</li><li><strong>Platform Services</strong> — servisní a metadata aplikace, docker orchestrace (máme napsanou vlastní), serverless appky</li><li><strong>UI </strong>— frontend</li><li><strong>Components </strong>— docker appky na zpracování dat</li><li><strong>SRE </strong>— servisní tým pro pomoc s devops</li><li><strong>Produkt </strong>— plánování a směřování produktu</li></ul><p>Co se týče technologií, tak vše máme v AWS a v Azure (jsme multicloud), všechny appky jsou v Dockeru (i když jsou třeba uvnitř spíš legacy). Na backendu používáme <strong>PHP </strong>(pro Dockerový appky) a <strong>NodeJS </strong>(serveless), na frontendu React. Pokud ale přijde jiná technologie nebo jazyk, který se nám bude hodit na řešení našeho problému, tak ho použijeme. Z frameworků používáme někde Zend (tuším, že prehistorická verze 1) a jinde Symfony, ale snažíme se psát appky tak, aby byly na frameworku co nejvíc nezávislé.</p><p>Posledních pár let jsme měli “best effort based” vývoj, kde všichni dělali tak nějak všechno, ale to s rostoucím týmem nebylo možný udržet, takže jsme přešli na takový modifikovaný Scrum, kde se snažíme udržet základní principy, ale hlavním cílem je, aby to fungovalo s našem full remote systému. Standupy máme například asynchronní do Slacku. Sprinty trvají 14 dní a mezi každým sprintem jsou 2 dny na vydechnutí a přípravu issues do dalšího sprintu. Všechny tyhle procesy se snažíme stavět tak, že je vyzkoušíme a pokud nám pomáhají tak je dodržujeme. Pokud ne, tak je po měsíci přestáváme používat.</p><p>Děláme code reviews a snažíme se dodržovat jednotný code style. Pomáhá nám v tom phpstan, phpunit, integrační testy, CI/CD … Pokud jde něco zautomatizovat tak to uděláme. Kde je to potřeba tak děláme dark launching a feature toggle. Jsme zvyklí releasovat několikrát za den a to malý věci. Drtivou většinu věcí máme otestovanou, ale stane se i že musíme něco rollbacknout.</p><p>Jako vývojáři se rotujeme na Level 2 supportu (používáme Zendesk). Znamená to, že jednou za 18 týdnů (je nás 18 aktuálně) někdo sedí týden na podpoře a v pracovní době odpovídá klientům na jejich problémy. Je to Level 2, takže odpovídáme pouze na tickety, který už nevyřešil Level 1. K tomu máme ještě on-call nad platformou 24h, který se rotuje denně a jeden člověk ho má max 2x do měsíce. Na on-call používáme PagerDuty, která nás notifikuje, že je něco špatně společně s runbookem jak to opravit. Nový člověk má k sobě vždycky backup, který mu vždycky poradí pokud něco nebude vědět. Všechny incidenty píšeme na <a href="https://status.keboola.com/">https://status.keboola.com</a> a je tam vidět, že řešíme něco přibližině 1–2krát za měsíc.</p><p>Je nám jedno v kolik budeš vstávat nebo chodit spát. Pokud budeš chtít dovolenou, tak to napíšeš do kanálu #dovolená na Slacku a tím je to vyřešený. V rámci týmu si musíš akorát dát pozor, aby tam někdo zbyl. Snažíme se řešit zastupitelnosti a proto každou věc musí umět alespoň 2 lidi. Pokud se v půlce dne rozhodneš, že chceš jet na kolo, tak si jeď na kolo. Nenárokujeme si 100% tvýho života v době 8:55–17:35. Já třeba v úterý a ve čtvrtek chodím dopoledne cvičit.</p><p>Fluktuace lidí je u nás minimální a za posledních 5 let jsme se rozloučili s 3 kolegama. Důvod byl vždycky stejnej a to, že jsme si vzájemně nesedli. Nehledáme lidi na konkrétní pozice, ale lidi, který baví pracovat mezi náma a budovat Keboolu. Není důležitý na 103% umět všechny jazyky a frameworky, ale spíš nám jde o to, abychom si sedli. Nehledáme člověka, který si sedne a bude čekat na přesný rozkazy co má dělat, ale spíš ti dáme problém a budeme o tebe čekat s nápady jak to vyřešit. Jsme extrémně otevřená firma a prakticky všechno se řeší public. Pokud máš s něčím problém nebo potřebuješ pomoct, tak napíšeš do Slacku, kde ti vždycky někdo pomůže.</p><p>Obecně se dá říct, že máme dobrej produkt i klienty, kteří nás mají rádi. Je normální, že když pro klienta něco uděláš, tak ti poděkuje.</p><h3>Příklady z Kebooly</h3><p><strong>Na posledním offsite jsme řešili například:</strong></p><ul><li><strong>Měli by se lidi v týmech rotovat?</strong> — máme víc týmů, kteří se starají o různé části systémů a otázka byla jestli chce někdo do jiného týmu. Odpověď byla ne a že jsou všichni spokojení s tím, co dělají.</li><li><strong>Technologický ownership</strong> — máme hromadu technologie, který používáme a občas ta technologie udělá BC break a my to nějak musíme řešit. Musí existovat lidi, kteří se starají o různý technologie a sledují changelogy. Odpověď je, že tohle ještě nemáme pevně vyřešený a postupně ty technologie si nacházejí svoje vlastníky. Teďka jsme například díky tomu podchytili, že Azure updatoval K8S a my jsme měli dost času si odladit jestli to bude fungovat s novou verzí</li><li><strong>Jsme stále API first? </strong>— všechno co vyvineme má nejdřív API a pak k tomu teprve děláme UI. Jakmile to má UI, tak to dáváme do na status page(<a href="http://status.keboola.com/">status.keboola.com</a>) jako novinku. Odpověď byla, že se snažíme být pořád API first.</li></ul><p><strong>S Microsoftem jsme řešili například:</strong></p><ul><li>Přepis z AWS ECR na Kubernetes (aby jsme mohli být jednodušeji multicloudový)</li><li>Adopci Azure Blob storage (úložiště — v našem případě náhrada pro AWS S3)</li><li>Adopci Azure Synapse (databáze — v našem případě alternativa pro Snowflake)</li><li>Networking v K8S (dokážeme provozovat celou platformu i v private networkingu, kdy není přístup do internetu)</li></ul><p><strong>Aktuálně máme ve sprintech user stories například na:</strong></p><ul><li>Vytvoření DEV/PROD režimu — chceme mít v Keboole produkční a vývojové prostředí, které bude oddělené a uživatelé si budou moci “hrát” na svém písečku. V Jiře, kterou používáme na plánování, to pak vypadá asi takhle:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*3IT4lVgwMMqYwu8pLA9e_Q.png" /></figure><ul><li>Pay-as-you-go — Chceme aby Keboola šla platit klasicky kreditkou.</li><li>Youtube deprekuje API a potřebujeme ověřit, že náš extraktor bude dál fungovat</li><li>Vylepšení monitoringu v DataDogu pro Azure stacky, abychom měli detailní monitoring pro K8S</li></ul><p><strong>Na Zendesku řešíme například v Level 2 ticketech:</strong></p><ul><li>MSSQL writer při chybě serveru nepoužívá správně retry mechanismus — založila se issue v Jiře pro komponentní tým a ten to v následujicím sprintu opravil a napsal klientovi, že je oprava nasazená</li><li>CSV obsahující JSON soubory timeoutuje při parsování — zjistili jsme, že se všechno dělá správně, ale databáze nestihne rozparsovat velké JSON soubory, takže jsme zvedli timeout v databázi</li><li>Reset MFA u uživatele — uživatel ztratil telefon a potřebujeme vypnout MFA na svém účtu. Ověřili jsme přes třetí osobu na straně klienta, že se nejedná o social hacking a vypnuli jsme MFA (<a href="https://keboolamanagementapi.docs.apiary.io/#reference/users/manage-user/disable-mfa-for-user">https://keboolamanagementapi.docs.apiary.io/#reference/users/manage-user/disable-mfa-for-user</a>), aby se uživatel mohl přihlásit</li></ul><p><strong>Onboarding nových kolegů:</strong></p><ul><li>Přečteš si security standards a “how we roll”</li><li>Vytvoříme ti kanál na slacku #hello_your_name, kde se budeš ptát</li><li>Vytvoříš si svojí první komponentu podle <a href="https://developers.keboola.com/extend/component/tutorial/">https://developers.keboola.com/extend/component/tutorial/</a></li><li>Budeš s náma společně alespoň týden v kanceláři v Praze nebo Brně, kde ti pomůžeme všechno nastavit a pochopit. Budeš mít k ruce jednoho vyvojáře, který ti se vším pomůže.</li><li>Upravíš existující komponentu a publikuješ ji</li><li>Dostaneš větší feature, která bude vyžadovat analýzu tvojí strany</li><li>Vybereš si v jakým týmu chceš dělat</li><li>Pracuješ v Keboole až do důchodu</li></ul><p>Samozřejmě, že každýmu nemusí sednout styl práce, kterým u nás fungujeme. Pokud se ti to ale líbí, tak klidně napiš na <a href="/service/mailto:vojta@keboola.com">vojta@keboola.com</a> a já se postarám o to, aby sme se sešli. Pokud nejsi programátor(ka) a zaujalo tě, jak u nás fungujeme, tak nám stejně napiš, protože hledáme lidi na všechny pozice (sales, customer success, marketing …).</p><p>Díky, Vojta z Kebooly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfff997787aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/hodil-by-se-n%C3%A1m-dal%C5%A1%C3%AD-buddy-do-party-cfff997787aa">Hodil by se nám další buddy do party</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keboola Connection Going Azure]]></title>
            <link>https://medium.com/keboola-nerds/keboola-connection-going-azure-d75878fd8b3e?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/d75878fd8b3e</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[keboola]]></category>
            <category><![CDATA[multi-cloud]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Ondřej Hlaváček]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 09:29:49 GMT</pubDate>
            <atom:updated>2020-03-31T09:29:49.526Z</atom:updated>
            <content:encoded><![CDATA[<p>Back in April last year, we made the business decision to make <a href="https://www.keboola.com/">Keboola Connection</a> available on Azure too (no, we&#39;re not quitting AWS). The story behind the decision is quite simple — our customers want it. So, some of our team went to Microsoft’s <a href="https://openhack.microsoft.com/">OpenHack</a> event in Paris to accelerate the adoption, have a guided, hands–on experience and get in touch with the Microsoft folks. From our point of view, the event was a success and so the work started.</p><figure><img alt="Our team at OpenHack in Paris" src="https://cdn-images-1.medium.com/max/1024/0*8obKGj8jwc4MYFcn" /></figure><h3>Planning</h3><p>Originally, our application only ran in AWS, and had some deep dependencies on the AWS services… That sounded like a headache.</p><p>So where to start? We laid out a plan to migrate parts of the platform to Azure technologies separately, while keeping the unmigrated rest of the technological stack in AWS (with peering in between). With this approach, we can migrate the stack in parts and still keep the whole thing running — no big bang, but small steps. The implementation stages are defined to capture their business importance and business impact.</p><ol><li>Data storage — Customer&#39;s data is stored in Azure, but all processes (e.g., <a href="https://help.keboola.com/transformations/">transformations</a>) are still executed in AWS.</li><li>Data manipulation — All data manipulation happens in Azure, but platform metadata (e.g., job metadata, component code) is still handled in AWS.</li><li>Metadata services — Everything runs in Azure, and there is no connection to AWS.</li></ol><h3>Implementation challenges</h3><p>Our platform is fully containerized, which makes the transition a lot easier, but we still use many proprietary solutions and rely heavily on managed services in AWS. As our operations team is very small (and focuses mainly on SRE), managed services are essential for us — we have no interest in running our own <a href="https://www.elastic.co/">Elastic</a> clusters, for example. Each service we use in AWS has to have its counterpart in the Azure stack and should ideally be generic enough so that we can use additional cloud providers in the future as customers require them (we&#39;re looking at you, <a href="https://cloud.google.com/gcp/">GCP</a>). We have encountered a few different scenarios:</p><p><strong>Drop-in replacement</strong> is the easiest — the application doesn&#39;t change — it only connects to another endpoint (e.g., <a href="https://aws.amazon.com/rds/mysql/">Amazon RDS for MySQL</a> — <a href="https://docs.microsoft.com/en-us/azure/mysql/">Azure Database for MySQL</a>).</p><p><strong>Similar service</strong> offers the same functionality but with a different API. We have to change the application to support both APIs but our internal APIs don&#39;t change (e.g., <a href="https://aws.amazon.com/s3/">Amazon S3</a> — <a href="https://azure.microsoft.com/en-us/services/storage/blobs/">Azure Blob Storage</a>).</p><p><strong>Switching to a non-proprietary technology</strong> requires some coding, but will allow us to be more flexible in the future. We&#39;ve decided to leave <a href="https://aws.amazon.com/ecs/">Amazon Elastic Container Service</a> in favor of <a href="https://kubernetes.io/">Kubernetes</a>, which offers managed services on all major clouds. Azure is now our proving ground and if <a href="https://azure.microsoft.com/en-us/services/kubernetes-service/">Azure Kubernetes Service</a> and Kubernetes prove successful, we&#39;ll switch to <a href="https://aws.amazon.com/eks/">Amazon Elastic Kubernetes Service</a> too.</p><p><strong>Self–hosting</strong> is not something we&#39;ve had to do, at least not yet, but we have to keep it in mind. Some cloud providers may not offer managed services we use (e.g., Azure doesn&#39;t have managed Elastic) and switching to something else may be more difficult than self–hosting, at least before we figure out a different solution.</p><h3>Automation</h3><p>Our CI/CD pipelines run on <a href="https://travis-ci.com/">Travis CI</a>. The Travis job builds, tests and deploys the application automatically, but the infrastructure is a different story. Launching a new instance (region) of Keboola Connection is a tedious manual process described in long README files across multiple repositories.</p><p>The main goal of our Azure adoption is to have the ability to run single tenant instances in customers&#39; subscriptions, so it also makes sense to embrace automatic infrastructure deployments.</p><p>I didn’t want to start rewriting the existing code, so I decided to append <a href="https://azure.microsoft.com/en-us/services/devops/">Azure DevOps</a> to the existing processes and only use them for the new, and Azure–related stuff.</p><p><strong>Application deployment </strong>is triggered by a successful Travis CI job (which is triggered by a commit on the master branch) and launches a DevOps Release Pipeline that deploys the latest image of the application from <a href="https://azure.microsoft.com/en-us/services/container-registry/">Azure Container Registry</a> into Azure Kubernetes Service. This is fairly straightforward.</p><p><strong>Infrastructure deployment</strong> includes only Azure resources and exclusively runs in Azure DevOps. We have the infrastructure described in a GitHub repository with <a href="https://azure.microsoft.com/en-us/resources/templates/">Azure Resource Manager</a> templates coupled with simple Bash scripts that handle passing variables between templates. All variables (nodes count, database size, etc.) are stored in the DevOps Release Pipeline. A new commit in the master branch of the repository triggers a new deployment/update of the infrastructure. Of course, the update runs in a staging environment first and if it&#39;s successful, a production update is executed.</p><h3>Consultations</h3><p>We&#39;ve contracted a few local consulting companies (namely <a href="https://networg.com/">NETWORG</a> and <a href="https://www.boldbrick.cz/">BoldBrick</a>) to help us with understanding and designing the resources (Azure) and processes (DevOps). In hindsight, it was a very clever decision that helped us avoid many costly dead ends. We&#39;ll stick to this approach.</p><h3>Look and feel</h3><p>So far, I can tell Microsoft has made a huge leap forward. The whole environment feels developer friendly — there&#39;s a lot of great documentation, I love their integration with GitHub (all documentation issues are stored and handled in GitHub), and Microsoft staff watches community boards and helps where possible. I also tried the Azure email support channel and had a great experience.</p><p>There are a lot of free Azure training and bootcamp events backed by Microsoft. We were even able to get free help from internal Microsoft experts.</p><p>Sometimes you can feel Microsoft&#39;s corporate (old–school?) approach, but it mostly made me smile, rather than worry — e.g., their support staff called me when I hadn&#39;t replied to their response in 24 hours to check if everything was OK.</p><p>Microsoft sure expects we&#39;ll spend some money there, but the difference between its approach and AWS&#39;s approach is like night and day.</p><h3>What&#39;s next?</h3><p>We&#39;re not finished. Lots of challenges are still ahead of us, but this was expected. We wanted to get familiar with the new environment before the s**t hit the fan and we were facing more complicated challenges. We&#39;ll keep working and sharing our experiences with Azure and DevOps (and a multi–cloud environment). If you have any relevant experience, please let us know. We&#39;re here to learn, and would love to hear from you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d75878fd8b3e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/keboola-connection-going-azure-d75878fd8b3e">Keboola Connection Going Azure</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hľadáme ďalšieho človeka na UIčka]]></title>
            <link>https://medium.com/keboola-nerds/h%C4%BEad%C3%A1me-%C4%8Fal%C5%A1ieho-%C4%8Dloveka-na-ui%C4%8Dka-1656d127648d?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/1656d127648d</guid>
            <category><![CDATA[keboola]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[ui]]></category>
            <dc:creator><![CDATA[Vladimír Kriška]]></dc:creator>
            <pubDate>Tue, 03 Mar 2020 13:58:52 GMT</pubDate>
            <atom:updated>2020-03-03T13:58:52.341Z</atom:updated>
            <content:encoded><![CDATA[<p>Do Kebooly hľadáme JavaScript vývojára, ktorý by sa pripojil k UI tímu a pomohol nám tvoriť a vylepšovať UI pre dátovú platformu <strong>Keboola Connection</strong> a ostatné podporné nástroje.</p><h3>Čo je Keboola Connection?</h3><p>Perfektne to pred pár týždňami popísal <a href="https://medium.com/@petrsimecek">Padák</a> (jeden zo zakladateľov Kebooly) <a href="https://padak.keboola.com/sh%C3%A1n%C3%ADm-do-kebooly-produk%C5%A5%C3%A1ka-7e7b8c1926cd">v článku, kde hľadá produkťáka.</a></p><p>Keboola Connection je platforma na zber, analýzu a transformáciu dát, pričom nie je podstatné odkiaľ dáta sú alebo kam idú. Akonáhle sú dáta u nás v systéme, možnosti sú neobmedzené. Napríklad môžu byť sprístupnené všetkým vo firme alebo jednoducho len poslané niekde ďalej, napr. na vizualizáciu. A navyše, nech už je ten proces akýkoľvek, je možné ho zautomatizovať a naplánovať jeho pravidelné spúšťanie.</p><p>Keď sa na to ale pozrieme viac technicky, tak Keboola Connection je <a href="https://developers.keboola.com/overview/api/">set viacerých REST APIs</a>, pomocou ktorých je možné celú platformu ovládať. To, čo sa navonok chová ako jedna aplikácia je v skutočnosti niekoľko systémov.</p><p>A pre túto platformu tvoríme UIčka. Jedná sa o Single-page aplikácie, ktoré poháňa React.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4o5xEfBRtWXI3Q50E1U-Gw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmzTMy-1gMt_M8l5HqdYrQ.png" /><figcaption>Keboola Connection UI (vľavo) a Developer Portal UI (vpravo)</figcaption></figure><h3>Ako to funguje v Keboole?</h3><p>Firemná kultúra (dúfam, že to ešte nie je nadávka) sa v Keboole piluje už dosť dlho, a napriek rozdeleniu do tímov, je v Keboole stále možnosť veľa vecí ovplyvňovať, dať im smer a kúsok svojej identity.</p><p>Máme 6 tímov a každý z nich má svoje oblasti zodpovednosti:</p><ul><li><strong>SRE</strong> tím má na starosti infraštruktúru, AWS, Azure, atď.</li><li><strong>Connection</strong> tím rieši naše Storage API a Manage API, ktoré sú pevným základom Keboola Connection.</li><li><strong>Platform Services</strong> má za úlohu starať sa o podporné služby ako Docker Runner, Orchestrator, Job Queue alebo Oauth Broker.</li><li><strong>Component</strong> tím je zodpovedný za všetky naše Docker komponenty.</li><li><strong>UI</strong> tím rieši všetky veci súvisiace s UI (detailnejšie nižšie).</li><li><strong>Documentation</strong> tím dbá na to aby sme mali celú platformu pekne zdokumentovanú, či už pre bežných používateľov alebo vývojárov.</li></ul><p>Vôbec nie je nezvyklé presunúť sa do tímu, kde by to človeka viac bavilo alebo tam, kde si myslí, že by mohol mať väčší prínos.</p><p>A tým sa dostávame k ďalším vlastnostiam, vďaka ktorým Keboola vytŕča z radu:</p><ul><li>Môžeš <strong>makať odkiaľ chceš.</strong> Jeden z kolegov je momentálne vo Vietname, a pred niekoľkými rokmi sme sa boli viacerí mesiac na Bali.</li><li>Môžeš <strong>makať kedy chceš</strong>. Ak chceš mať poobedie s deťmi, nikto to nerieši.</li><li><strong>Neriešime dovolenku</strong> a keď cítiš, že potrebuješ vypnúť, dostaneš na to priestor.</li><li>Chodíme cca <strong>3x do roka na tzv. Offsite</strong>, niekam na chalupu, kde sa bavíme všetci spolu o ďalšom smerovaní, riešeniach, nových nápadoch a pod.</li><li>Nie sme agentúra. Keboola Connection je náš produkt a sami sme zodpovední za to, kam ho posunieme.</li></ul><p>A najnovšie štartujeme <a href="https://en.wikipedia.org/wiki/Employee_Stock_Ownership_Plan">ESOP</a> — opčný plán, a je možné získať podiel vo firme.</p><h3>Čo budeš robiť?</h3><p>Pridáš sa k UI tímu, ktorý má len pár ľudí a so všetkými budeš spolupracovať takmer na dennej báze.</p><p>Sme zodpovední za viac oblastí, no najdôležitejšie sú:</p><ol><li><strong>Správa front-end aplikácii</strong>, s ktorými zákazník Keboola Connection prichádza do styku. Či už ako používateľ nášho hlavného rozhrania, alebo ako 3rd party developer, ktorý chce u nás spustiť Docker komponentu (<a href="https://components.keboola.com/">Developer Portal</a>).</li><li><strong>Tvorba nových React komponent</strong> do spomínaných aplikácií a v neposlednom rade do <a href="https://indigo.keboola.com/">Indigo UI</a> — našej komponentovej knižnice, ktorá zároveň obsahuje základné CSS.</li><li><strong>Code Review</strong>, bez ktorého sa u nás do produkcie nedostane takmer ani bodka.</li><li><strong>Spolupráca s produktovým tímom</strong>, plánovanie a validácia navrhovaných riešení.</li><li>Spolupráca s ostatnými tímami, ktoré zabezpečujú bezproblémový chod APIs, ktoré konzumujeme.</li><li>Správa našej CDN (Cloudfront).</li></ol><h4>Keboola Connection UI</h4><p>UI pre Keboola Connection nie je projekt, ktorý sa dá spraviť za víkend. Nikdy sme nedospeli k záveru, že je lepšie všetko zahodiť napísať nanovo. Projekt nedávno oslávil <strong>5 rokov</strong> a <a href="https://status.keboola.com/new-ui-for-keboola-connection">koncom októbra sme vydali jeho novú verziu</a> (veľký redesign), ktorú stále vylepšujeme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r2CiV0EK-6nPy6ykelbP_Q.png" /><figcaption>Keboola Connection UI — Github Insights (Contributions to master, excluding merge commits)</figcaption></figure><p>Prešli sme si aj veľkými aktualizáciami (napr. od React 0.13 až po React 16) a niektoré veci sme popísali aj na našom Tech Blogu <a href="https://500.keboola.com/">500 Internal Server Error</a>.</p><ul><li>Ako nám pomáha Prettier: <a href="https://500.keboola.com/formatting-javascript-code-using-prettier-automatically-in-a-container-c55da7702ec5">https://500.keboola.com/formatting-javascript-code-using-prettier-automatically-in-a-container-c55da7702ec5</a></li><li>Ako sme prehodili ~200 React komponent z .coffee na .jsx <a href="https://500.keboola.com/how-we-converted-coffeescript-files-with-about-220-react-components-to-javascript-f4b76387888d">https://500.keboola.com/how-we-converted-coffeescript-files-with-about-220-react-components-to-javascript-f4b76387888d</a></li></ul><h3>Koho hľadáme?</h3><p>Hľadáme JavaScript vývojára, ktorému nie je téma UI cudzia a rád pracuje na rozhraniach, s ktorým prichádza do styku zákazník.</p><p>Mal by si mať <strong>skúsenosti s React</strong>, vedieť napísať interaktívnu komponentu, vedieť kde držať stav komponenty. A bez problémov určiť prečo sa komponenta prekreslila. A tiež poznať best-practices.</p><p>Mal by si vedieť <strong>testovať</strong> ale aj vedieť určiť kedy je test zbytočný.</p><p>Mal by si vedieť čo sú HTTP status kódy, čo je REST API a ako by sa malo chovať aby bolo dobré (a to nielen pre UI).</p><p>Nemali by ti byť cudzie pojmy ako CORS, Preflight request, TTFB, či gzip a mal by si ľahko vedieť spoznať, či je aplikácia rýchla alebo pomalá a prečo to tak je (z pohľadu UI).</p><p>Musíš <strong>vedieť dotiahnúť veci do konca</strong> a byť s nimi spokojný. Ale nesmieš sa uspokojiť len tak s hocičím. Musíš byť trpezlivý, mať <strong>cit pre detail</strong>, vedieť nájsť chybu alebo <strong>vyriešiť problém a nevzdať sa</strong>. Ale aj <strong>nájsť kompromis alebo požiadať o radu.</strong></p><p>Nemusíš vedieť všetko, to nikto neočakáva. Chceme však niekoho so skúsenosťami s <strong>prácou v tíme</strong>. Dôležité je, aby sme si sadli, nie to, koľko technológií ovládaš.</p><p>A mal by si vedieť anglicky. Na úrovni aby si sa v pohode dohovoril s našimi kolegami, ktorí nehovoria česky resp. slovensky.</p><h4>Technológie</h4><p>Pri vývoji UI sa u nás určite stretneš s technológiami ako:</p><ul><li>JavaScript, React, Immutable, Create React App, Flux, Redux, Prettier, Webpack, Jest</li><li>HTML, CSS, Less, Sass, Bootstrap, Storybook, Figma</li><li>S3, Cloudfront, Travis CI, Docker</li><li>či REST, Apiary, Sentry</li></ul><h3>Ako sa prihlásiť?</h3><p>Pošli email na <a href="/service/mailto:vlado@keboola.com">vlado@keboola.com</a> a napíš, čo ťa na UIčkach baví a aký projekt alebo technológia ťa posledne nadchla. Môžeš pripojiť odkaz na svoj Github profil alebo na projekty, na ktorých si pracoval.</p><p>Potom sa ti ozveme, kde sa dozvieš, či sme našli prienik v tvojich skúsenostiach a našich požiadavkách. Môže sa stať, že budeme chcieť ešte niečo upresniť.</p><p>Ďalej bude nasledovať videohovor alebo osobné stretnutie, kde sa trošku oťukáme a pobavíme sa viac o tvojich skúsenostiach. Skúsime ti popísať nejaký problém a ty povieš ako by si ho riešil. Nebudeme programovať na papier ani porovnávať výkonnosť zoraďovacích algoritmov.</p><p>Nakoniec sa porozprávaš s našou anglicky hovoriacou kolegyňou.</p><p>Ak toto všetko vydržíš a obe strany budú cítiť, že by to mohlo fungovať, môžeš s nami začať makať. Sme rôzne rozlezení po ČR, asi polovica remote a ďalší v kanceláriách v Brne a v Prahe.</p><p>Teším sa až sa ozveš! 👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1656d127648d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/h%C4%BEad%C3%A1me-%C4%8Fal%C5%A1ieho-%C4%8Dloveka-na-ui%C4%8Dka-1656d127648d">Hľadáme ďalšieho človeka na UIčka</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PHP serverless application in AWS (part 1)]]></title>
            <link>https://medium.com/keboola-nerds/php-serverless-application-in-aws-part-1-4dd0ad1fd69b?source=rss----9f95ce157e5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/4dd0ad1fd69b</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[keboola]]></category>
            <dc:creator><![CDATA[Miroslav Čillík]]></dc:creator>
            <pubDate>Mon, 20 Jan 2020 14:22:15 GMT</pubDate>
            <atom:updated>2020-01-20T14:22:15.360Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*C8imhYT71nhrG8eRDFugrg.png" /></figure><p>AWS Lambda, the backbone of serverless applications in AWS, currently supports Java, Go, PowerShell, Node.js, C#, Python, and Ruby languages. If you’ve ever wanted to use serverless architecture with your PHP applications, it is now possible, thanks to the features AWS announced in November 2018:</p><ul><li><strong>Lambda Layers</strong> — a way to centrally manage code and data shared across multiple functions.</li><li><strong>Lambda Runtime API</strong> — a simple interface to use any programming language, or a specific language version, to develop your functions.</li></ul><p>These features allow developers to use virtually any programming language to write their lambda functions.</p><p>In this blog post, we look at how to write a simple API app in PHP using the Serverless and Bref frameworks, the latter providing the PHP runtime so we don’t have to <a href="https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html">write our own</a>.</p><p>To handle the dependencies of the app and to easily test it locally, we will use Docker and Docker Compose.</p><h3>AWS</h3><p>You will need an AWS account to deploy your app to AWS using the Serverless framework.</p><p>Create one following <a href="https://serverless.com/framework/docs/providers/aws/guide/credentials/">this guide</a> and then paste your AWS credentials into an <strong>.env</strong> file:</p><pre>#.env</pre><pre>AWS_ACCESS_KEY_ID=****<br>AWS_SECRET_ACCESS_KEY=****</pre><p>This solution is just for the sake of this tutorial. I don’t recommend using user credentials this way in production. In our production apps, we use different roles for the deployment and execution of our Lambda functions. Check out <a href="https://serverless.com/framework/docs/providers/aws/guide/iam/">this guide</a> to learn how Serverless works with AWS roles, or check out our <a href="https://github.com/keboola/serverless-demo-app">Serverless demo app</a> for an example of a more secure solution.</p><p>We are also in the process of developing a new solution to work with AWS policies for our serverless applications using AWS pipelines. I might write about it in a future blog post.</p><h3>Docker</h3><p>We will use Docker to manage the dependencies of our app and create environments for development, testing and deployment. First, we create a Dockerfile. We use <a href="https://hub.docker.com/r/keboola/docker-amazonlinux-php">keboola/docker-amazonlinux-php</a> as our base image. This base image is the Amazon Linux image used by AWS Lambda with PHP installed on top — you can see its Dockerfile <a href="https://github.com/keboola/docker-amazonlinux-php/blob/master/Dockerfile">here</a>. It’s also possible to use one of the <a href="https://hub.docker.com/u/bref">images provided by Bref</a>.</p><p>Let’s take a look at our Dockerfile. In addition to the base image, we are installing Node, Yarn, and Composer, to install the PHP dependencies and Serverless framework.</p><p>The WORKDIR is also important: “/var/task” is the path where the AWS Runtime API expects the function code to be.</p><pre>#Dockerfile</pre><pre>FROM keboola/docker-amazonlinux-php:latest<br><br># install node, yarn and composer<br>RUN curl --silent --location https://rpm.nodesource.com/setup_12.x | bash -<br>RUN curl --silent https://dl.yarnpkg.com/rpm/yarn.repo &gt; /etc/yum.repos.d/yarn.repo<br>RUN yum -y install findutils nodejs npm yarn python27 wget<br>RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin/ --filename=composer<br><br># working directory<br>WORKDIR /var/task<br>COPY . .<br><br># install composer dependencies<br>RUN composer install<br><br># install serverless<br>RUN yarn global add serverless@1.57</pre><p>Now we create the docker-compose.yml file. Docker Compose simplifies the running of different Docker services. We will have two services “deploy” and “tests”. As the names suggest, “tests” will be used for testing and development purposes and “deploy” for deployment to AWS. There’s also a third service called “tests-local”, which extends the “tests” service. The only difference between the two is that “tests-local” has a mapped volume, which is useful during development as changes to the code are propagated into the container and vice versa. We also pass the AWS credentials into the containers from the .env file.</p><pre>#docker-compose.yml</pre><pre>version: &quot;3&quot;</pre><pre>services:<br>  deploy:<br>    build:<br>      context: .<br>    environment:<br>      - AWS_ACCESS_KEY_ID<br>      - AWS_SECRET_ACCESS_KEY<br>    command: serverless deploy --force</pre><pre>  tests: &amp;tests<br>    build:<br>      context: .<br>    environment:<br>      - AWS_ACCESS_KEY_ID<br>      - AWS_SECRET_ACCESS_KEY<br>    command: composer ci</pre><pre>  tests-local:<br>    &lt;&lt;: *tests<br>    volumes:<br>      - .:/var/task</pre><p>Add the <strong>composer.json</strong> file:</p><pre>{<br>  &quot;name&quot;: &quot;keboola/hello-world-bref&quot;,<br>  &quot;license&quot;: &quot;MIT&quot;,<br>  &quot;type&quot;: &quot;project&quot;,<br>  &quot;description&quot;: &quot;Hello World with bref&quot;,<br>  &quot;keywords&quot;: [ &quot;keboola&quot;, &quot;bref&quot;, &quot;hello&quot;, &quot;world&quot; ],<br>  &quot;authors&quot;: [<br>    {<br>      &quot;name&quot;: &quot;Keboola&quot;,<br>      &quot;email&quot;: &quot;devel@keboola.com&quot;<br>    }<br>  ],<br>  &quot;autoload&quot;: {<br>    &quot;psr-4&quot;: {<br>      &quot;Keboola\\HelloWorldBref\\&quot;: &quot;src/&quot;<br>    }<br>  },<br>  &quot;autoload-dev&quot;: {<br>    &quot;psr-4&quot;: {<br>      &quot;Keboola\\HelloWorldBref\\Tests\\&quot;: &quot;tests/&quot;<br>    }<br>  },<br>  &quot;require&quot;: {<br>    &quot;php&quot;: &quot;^7.3&quot;,<br>    &quot;ext-json&quot;: &quot;*&quot;,<br>    &quot;bref/bref&quot;: &quot;^0.5.12&quot;<br>  },<br>  &quot;require-dev&quot;: {<br>    &quot;phpunit/phpunit&quot;: &quot;^7.1&quot;<br>  },<br>  &quot;scripts&quot;: {<br>    &quot;tests&quot;: &quot;phpunit&quot;,<br>    &quot;ci&quot;: [<br>      &quot;@composer validate --no-check-all --strict&quot;,<br>      &quot;@tests&quot;<br>    ]<br>  },<br>  &quot;config&quot;: {<br>    &quot;sort-packages&quot;: true<br>  }<br>}</pre><p>Let’s install the dependencies using the “tests-local” container:</p><pre>$ docker-compose run --rm tests-local composer install</pre><p>(The dependencies were already installed in the Dockerfile, but the working directory inside the container was then replaced by the mapped volume from the local disk. Installing the dependencies now will also install them into our local directory.)</p><h3>Bref framework</h3><p>The Bref framework is still quite young, but it’s developing very quickly and provides an easy solution to make PHP work in the AWS Lambda environment. Check out the Bref <a href="https://bref.sh/docs/first-steps.html">documentation</a> for more information about this framework.</p><p>Initialize the Bref framework:</p><pre>$ docker-compose run --rm tests-local ./vendor/bin/bref init</pre><p>From the options choose <strong>[0] PHP function</strong>. This will generate a very simple configuration file <strong>serverless.yml,</strong> for the Serverless framework with following content:</p><pre>#serverless.yml</pre><pre>service: app</pre><pre>provider:<br>  name: aws<br>  region: us-east-1<br>  runtime: provided</pre><pre>plugins:<br>  — ./vendor/bref/bref</pre><pre>functions:<br>  function:<br>    handler: index.php<br>    description: &#39;&#39;<br>    layers:<br>      — ${bref:layer.php-73}</pre><p>We specify here that we want to use the “provided” runtime, and then, in “functions” we specify exactly which “layer” (runtime) we want to use.</p><p>The ${…} notation is the <a href="https://serverless.com/framework/docs/providers/aws/guide/variables/">syntax to use variables</a> in serverless.yml. Bref provides a Serverless plugin (“./vendor/bref/bref”) that provides these variables:</p><ul><li>${bref:layer.php-74}</li><li>${bref:layer.php-73}</li><li>${bref:layer.php-72}</li><li>${bref:layer.php-74-fpm}</li><li>${bref:layer.php-73-fpm}</li><li>${bref:layer.php-72-fpm}</li><li>${bref:layer.console}</li></ul><p>Bref currently provides runtimes for PHP 7.2, 7.3, and 7.4. You can read more in the Bref <a href="https://bref.sh/docs/runtimes/">documentation</a>.</p><p>One confusing aspect about Bref is figuring out which layer to choose. Our generated files now use “${bref:layer.php-73}”, and when you look at the generated “index.php” file, we see our code is wrapped in the “lambda()” function. This function handles the communication with the AWS Runtimes API.</p><p>The FPM runtime layer also incorporates a <a href="https://github.com/brefphp/bref/blob/master/runtime/layers/fpm/bootstrap#L28">loop</a>, in which it listens for incoming API Gateway events, forwards them to our handler running in a php-fpm process, and returns a response in a format compatible with API Gateway. In other words, it does some of the work for us, but for this tutorial I will use the non-FPM runtime because this way, I have access to the raw <a href="https://serverless.com/framework/docs/providers/aws/events/apigateway/#example-lambda-proxy-event-default">API Gateway event data</a>, which is not available in the FPM/Website style.</p><p>(I noticed that there’s already an issue about this exact problem, so the FPM/website style should support access to the event data soon: <a href="https://github.com/brefphp/bref/pull/501">https://github.com/brefphp/bref/pull/501</a>).</p><p>Replace the content of the <strong>index.php</strong> file with:</p><pre>&lt;?php </pre><pre>declare(strict_types=1);</pre><pre>require __DIR__.&#39;/vendor/autoload.php&#39;;</pre><pre>lambda(function ($event) {<br>    return [<br>        &#39;statusCode&#39; =&gt; 200,<br>        &#39;body’ =&gt; json_encode(&#39;Hello world from PHP in Lambda&#39;),<br>    ];<br>});</pre><p>And add routing information for API Gateway into the <strong>serverless.yml</strong> file. <br>I’ve also modified the “service” and “function” names.</p><pre>#serverless.yml</pre><pre>service: php-hello-world</pre><pre>provider:<br>  name: aws<br>  region: us-east-1<br>  runtime: provided<br><br>plugins:<br>  - ./vendor/bref/bref<br><br>functions:<br>  api:<br>    handler: index.php<br>    description: &#39;&#39;<br>    layers:<br>      - ${bref:layer.php-73}<br>    events:<br>      - http: &#39;ANY /&#39;<br>      - http: &#39;ANY /{proxy+}&#39;</pre><p>You can now deploy your function to AWS:</p><pre>$ docker-compose run --rm deploy</pre><p>If the deployment is successful, you should see the address of the API Gateway endpoint for your API in the output of the deploy command.</p><pre>endpoints:<br> ANY - <a href="https://xpc2xj29n4.execute-api.us-east-1.amazonaws.com/dev">https://xpc2xj29n4.execute-api.us-east-1.amazonaws.com/dev</a><br> ANY - <a href="https://xpc2xj29n4.execute-api.us-east-1.amazonaws.com/dev/%7Bproxy+">https://xpc2xj29n4.execute-api.us-east-1.amazonaws.com/dev/{proxy+</a>}</pre><p>After opening this endpoint in your browser you should see:</p><p><strong>“Hello world from PHP in Lambda”</strong></p><h3>More complex application</h3><p>Now we will refactor our code to create something that at least resembles an actual, real-world application. Let’s first modify the <strong>serverless.yml</strong> file. We add two different path templates into our function event declaration:</p><pre>#serverless.yml</pre><pre>functions:<br>  api:<br>    handler: index.php<br>    description: &#39;&#39;<em><br>    </em>layers:<br>      - ${bref:layer.php-73}<br>    events:<br>      - http:<br>          method: get<br>          path: &quot;/{name}&quot;<br>      - http:<br>          method: post<br>          path: &quot;/&quot;</pre><p>We now modify the <strong>index.php</strong> file to create a “front controller”:</p><pre>&lt;?php</pre><pre>declare(strict_types=1);</pre><pre>use Keboola\HelloWorldBref\App;</pre><pre>require __DIR__.&#39;/vendor/autoload.php&#39;;</pre><pre>lambda(function ($event) {<br>    $app = new App();</pre><pre>    $resourceMap = [<br>        &#39;/{name}&#39; =&gt; [<br>            &#39;GET&#39; =&gt; function ($event) use ($app): array {<br>                return [<br>                    &#39;statusCode&#39; =&gt; 200,<br>                    &#39;body&#39; =&gt; json_encode(<br>                        $app-&gt;getHelloWorld($event)<br>                    ),<br>                ];<br>            },<br>        ],<br>        &#39;/&#39; =&gt; [<br>            &#39;POST&#39; =&gt; function ($event) use ($app): array {<br>                return [<br>                    &#39;statusCode&#39; =&gt; 201,<br>                    &#39;body&#39; =&gt; json_encode(<br>                        $app-&gt;postHelloWorld($event)<br>                    ),<br>                ];<br>            },<br>        ],<br>    ];</pre><pre>    try {<br>        if (empty($event[&#39;httpMethod&#39;]) <br>            || empty($event[&#39;resource&#39;])<br>        ) {<br>            throw new \Exception(&#39;Bad Request&#39;, 400);<br>        }<br>        if (empty($resourceMap[$event[&#39;resource&#39;]])) {<br>            throw new \Exception(&#39;Route Not Found&#39;, 404);<br>        }<br>        $resource = $resourceMap[$event[&#39;resource&#39;]];<br>        if (empty($resource[$event[&#39;httpMethod&#39;]])) {<br>            throw new \Exception(&#39;Method Not Allowed&#39;, 405);<br>        }<br>        $actionFn = $resource[$event[&#39;httpMethod&#39;]];</pre><pre>        return $actionFn($event);<br>    } catch (\Throwable $e) {<br>        return [<br>            &#39;statusCode&#39; =&gt; 500,<br>            &#39;body&#39; =&gt; json_encode([<br>                &#39;errorMessage&#39; =&gt; $e-&gt;getMessage(),<br>            ])<br>        ];<br>    }<br>});</pre><p>The <strong>$resourceMap</strong> array serves as a router. Based on the path and the http method of the event, the appropriate function from the App class will be called.</p><p>We haven’t made the App class yet, so let’s create it in the <strong>src/App.php </strong>file:</p><pre>&lt;?php<br><br>declare(strict_types=1);<br><br>namespace Keboola\HelloWorldBref;<br><br>class App<br>{<br>    public function getHelloWorld($event)<br>    {<br>        return sprintf(<br>            &#39;Hello %s&#39;, <br>            $event[&#39;pathParameters&#39;][&#39;name&#39;]<br>        );<br>    }<br><br>    public function postHelloWorld($event)<br>    {<br>        $body = json_decode($event[&#39;body&#39;], true);<br>        return sprintf(&#39;Hello %s&#39;, $body[&#39;name&#39;]);<br>    }<br>}</pre><p>Deploy the app (don’t forget to rebuild the “deploy” image):</p><pre>$ docker-compose build deploy &amp;&amp; docker-compose run --rm deploy</pre><p>You might encounter an error like:</p><blockquote>An error occurred: ApiGatewayResourceNameVar — A sibling ({proxy+}) of this resource already has a variable path part — only one is allowed</blockquote><p>In that case, you need to delete the Cloud Formation stack created by the Serverless deploy manually via AWS console and run the deploy again or use this <a href="https://github.com/serverless/serverless/issues/3785#issuecomment-340784848">workaround</a>. :) If you delete the stack, it will delete the API Gateway endpoint, so after another deployment the endpoint will be different.</p><p>After a successful deployment, our API can now respond to two types of https requests:</p><ul><li>GET requests with “name” as a path parameter, which should result in a response like “Hello miro”, for example, with status code 200.</li><li>POST requests with a JSON payload like:<br>{ “name”: “miro” }, resulting in the same response with status code 201.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ptK6bwc_u1TPa108" /><figcaption>Making a POST request to our API in Postman</figcaption></figure><h3>Unit testing</h3><p>The reason we created the App class was not only to better structure our code but also to easily write unit tests for our App class. A simple unit test placed in the <strong>tests/AppTest.php</strong> file might look like this:</p><pre>&lt;?php<br><br>declare(strict_types=1);<br><br>namespace Keboola\HelloWorldBref\Tests;<br><br>use Keboola\HelloWorldBref\App;<br>use PHPUnit\Framework\TestCase;<br><br>class AppTest extends TestCase<br>{<br>    public function testGetHelloWorld()<br>    {<br>        $app = new App();<br>        $event = [<br>            &#39;pathParameters&#39; =&gt; [<br>                &#39;name&#39; =&gt; &#39;miro&#39;<br>            ]<br>        ];<br><br>        $this-&gt;assertEquals(&#39;Hello miro&#39;, $app-&gt;getHelloWorld($event));<br>    }<br><br>    public function testPostHelloWorld()<br>    {<br>        $app = new App();<br>        $event = [<br>            &#39;body&#39; =&gt; json_encode([<br>                &#39;name&#39; =&gt; &#39;miro&#39;<br>            ])<br>        ];<br><br>        $this-&gt;assertEquals(&#39;Hello miro&#39;, $app-&gt;postHelloWorld($event));<br>    }<br>}</pre><p>To run the phpunit test suite, we need a <strong>phpunit.xml.dist</strong> config file:</p><pre>&lt;?xml version=”1.0&quot; encoding=”UTF-8&quot;?&gt;<br>&lt;phpunit backupGlobals=”false”<br> backupStaticAttributes=”false”<br> colors=”true”<br> convertErrorsToExceptions=”true”<br> convertNoticesToExceptions=”true”<br> convertWarningsToExceptions=”true”<br> processIsolation=”false”<br> stopOnFailure=”false”<br> bootstrap=”vendor/autoload.php”&gt;<br> &lt;testsuite name=”Tests”&gt;<br> &lt;directory&gt;tests/&lt;/directory&gt;<br> &lt;/testsuite&gt;<br>&lt;/phpunit&gt;</pre><p>Run the test suite:</p><pre>$ docker-compose run --rm tests-local ./vendor/bin/phpunit</pre><p>All tests should be green. :) You might have noticed that I’m manually creating the API Gateway events in the tests. You can check the API Gateway event structure <a href="https://serverless.com/framework/docs/providers/aws/events/apigateway/#example-lambda-proxy-event-default">here</a> under “Example “LAMBDA-PROXY” event (default)”.</p><p>What we are not testing is the output of the Bref wrapper “lambda” function. We will look at this in the next blog post.</p><p><a href="https://github.com/keboola/hello-world-bref">Here</a>’s the source code for this example application.</p><h3>Conclusion</h3><p>Developing even a simple serverless application requires a good knowledge of many tools, frameworks, and services, not to mention using custom runtime for unsupported languages such as PHP. So, I hope this brief guide will help you navigate the serverless waters a bit better.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd0ad1fd69b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keboola-nerds/php-serverless-application-in-aws-part-1-4dd0ad1fd69b">PHP serverless application in AWS (part 1)</a> was originally published in <a href="https://medium.com/keboola-nerds">500 Internal server error</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>