<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by George Gatheca on Medium]]></title>
        <description><![CDATA[Stories by George Gatheca on Medium]]></description>
        <link>https://medium.com/@gatheca-george?source=rss-c761e483cd5b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*m2nSIvyed4_IE6ff</url>
            <title>Stories by George Gatheca on Medium</title>
            <link>https://medium.com/@gatheca-george?source=rss-c761e483cd5b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:47:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gatheca-george/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting up grafana mimir in docker swarm environment]]></title>
            <link>https://blog.devops.dev/setting-up-grafana-mimir-in-docker-swarm-environment-1a92bffa5035?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a92bffa5035</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[cryptomanufaktur]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[mimir]]></category>
            <category><![CDATA[docker-swarm]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Mon, 19 Feb 2024 18:59:55 GMT</pubDate>
            <atom:updated>2024-02-20T18:03:02.643Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Prometheus</strong>!! Well who doesn&#39;t love and use our beloved open source monitoring system. It allows us to collect metrics from various services we are running and then do alerting and/or visualizations on them. For a medium to large enterprises running many systems in various environments, there comes a need for central metrics aggregation instead of having multiple data sources in grafana for each of the prometheus instance running on each environment. Thats where mimir comes in, it acts as a remote receiver for metrics, therefore each prometheus instance running anywhere can push those metrics to one central mimir.</p><p>Although you can run a single instance of mimir and handle millions of queries and metrics, it is always good to run multiple instances for higher availability and performance. If you have docker swarm already setup, then deploying mimir instances is as easy as adding any other compose stack. All the mimir instances will act as one and work together to handle queries or metrics write from all those prometheus instances.</p><h4>Advantages of mimir</h4><ol><li>Prometheus compatible — It uses similar promql language to query metrics. Setting data source in grafana is the same as setting a prometheus data source. Even dashboards and alerts on grafana remain the same, only data source is changed.</li><li>Scalability — mimir can handle a lot of metrics and can even be horizontally scaled if needed. This tutorial shows how to run 3 instances of mimir, with requests been load balanced between them using an nginx proxy server.</li><li>Fast metrics retrieval — For large queries, grafana mimir is able to distribute the queries across its instances making queries faster.</li><li>Permanent storage of metrics — Mimir uses S3 compatible storage to store the metrics. Therefore it is cheaper to store metrics permanently. You can also setup a time period for metrics whereby after the time passes, mimir will go ahead and clear them from S3 storage.</li><li>and many more advantages. You can read more about mimir on their <a href="https://grafana.com/oss/mimir/">documentation here</a>.</li></ol><h3>Load balance mimir in docker swarm environment</h3><p>To run mimir in docker swarm environment, you can use a docker compose file with mimir services defined as follows. Here we are defining 3 instances of mimir called <em>mimir-lb-a</em>, <em>mimir-lb-b</em> and <em>mimir-lb-c</em>. We also have an <em>nginx</em> service that acts a load balancer between the 3 mimir instances.</p><pre>version: &#39;3.9&#39;<br><br># Logging settings defaults we can just append to any service config.<br>x-logging: &amp;logging<br>  logging:<br>    driver: json-file<br>    options:<br>      max-size: 2000m<br>      max-file: &quot;3&quot;<br>      tag: &#39;{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}&#39;<br><br>services:<br>  mimir-lb:<br>    image: nginx:latest<br>    volumes:<br>      - $PWD/mimir-lb-nginx.conf:/etc/nginx/nginx.conf<br>    depends_on:<br>      - &quot;mimir-lb-a&quot;<br>      - &quot;mimir-lb-b&quot;<br>      - &quot;mimir-lb-c&quot;<br>    hostname: mimir-lb<br>    ports:<br>      - 9010:9010<br>    deploy:<br>      mode: replicated<br>      replicas: 1<br>    &lt;&lt;: *logging<br><br>  mimir-lb-a:<br>    image: grafana/mimir:latest<br>    volumes:<br>      - mimir-lb-a-data:/data<br>      - $PWD/mimir-lb-config.yml:/etc/mimir-template.yml<br>      - $PWD/mimir-entrypoint.sh:/etc/mimir-entrypoint.sh<br>    entrypoint: [ &quot;sh&quot;, &quot;/etc/mimir-entrypoint.sh&quot; ]<br>    hostname: mimir-lb-a<br>    deploy:<br>      mode: replicated<br>      replicas: 1<br>      placement:<br>        constraints: [&quot;node.role == worker&quot;]<br>    &lt;&lt;: *logging<br><br>  mimir-lb-b:<br>    image: grafana/mimir:latest<br>    volumes:<br>      - mimir-lb-b-data:/data<br>      - $PWD/mimir-lb-config.yml:/etc/mimir-template.yml<br>      - $PWD/mimir-entrypoint.sh:/etc/mimir-entrypoint.sh<br>    entrypoint: [ &quot;sh&quot;, &quot;/etc/mimir-entrypoint.sh&quot; ]<br>    hostname: mimir-lb-b<br>    deploy:<br>      mode: replicated<br>      replicas: 1<br>      placement:<br>        constraints: [&quot;node.role == worker&quot;]<br>    &lt;&lt;: *logging<br><br>  mimir-lb-c:<br>    image: grafana/mimir:latest<br>    volumes:<br>      - mimir-lb-c-data:/data<br>      - $PWD/mimir-lb-config.yml:/etc/mimir-template.yml<br>      - $PWD/mimir-entrypoint.sh:/etc/mimir-entrypoint.sh<br>    entrypoint: [ &quot;sh&quot;, &quot;/etc/mimir-entrypoint.sh&quot; ]<br>    hostname: mimir-lb-c<br>    deploy:<br>      mode: replicated<br>      replicas: 1<br>      placement:<br>        constraints: [&quot;node.role == worker&quot;]<br>    &lt;&lt;: *logging<br><br>volumes:<br>  mimir-lb-a-data:<br>  mimir-lb-b-data:<br>  mimir-lb-c-data:</pre><p>The load balancer runs on port 9010 and has a configuration file mounted which tells nginx how to route requests.</p><p>Each mimir service has a configuration file mounted that configures mimir and an entrypoint that sets up a variable in config. The variable is called <strong><em>advertise_addr</em></strong> and is needed for the mimir instances to <strong><em>gossip</em></strong> with each other. We also override the entrypoint so that we can execute the entrypoint script when container starts.</p><p>Under deploy section for each mimir service, we are running only 1 replica, and constrain it to run only on worker nodes. You can have as many instances as needed but its advisable to run 1 for each worker you have. For our example if you have 3 workers you run 3 mimir services <em>mimir-lb-a</em>, <em>mimir-lb-b</em> and <em>mimir-lb-c </em>with each running 1 replica<em>.</em>You can set constraints to something like</p><pre>constraints: [&quot;node.role == worker&quot;, &quot;node.hostname == worker1&quot;]</pre><p>so that to restrict each instance to run on a particular worker.</p><h4>Mimir Configuration</h4><p>To configure mimir the following config is used. It sets s3 storage to use and adds members to a list so each member knows existence of the others.</p><p>The most important section is <em>advertise_addr: ADVERTISE_ADDR_REPLACE</em></p><p>The placeholder <em>ADVERTISE_ADDR_REPLACE</em> will be overwritten by the entrypoint script when container starts. The value is the IP of the container on eth0 interface. The reason its important is because if the container is connected to many networks which mostly is the case using docker swarm and portainer, then mimir is confused on which ip to use. Setting the ip to one in eth0 interface makes sure all mimir instances can communicate and gossip properly under the same network.</p><pre>target: all<br><br>common:<br>  storage:<br>    backend: s3<br>    s3:<br>      region: &#39;&#39;<br>      endpoint: minio:9000<br>      access_key_id: myminio<br>      secret_access_key: mypassword<br>      insecure: true<br>      bucket_name: mimir<br><br># Blocks storage requires a prefix when using a common object storage bucket.<br>blocks_storage:<br>  storage_prefix: blocks<br>  tsdb:<br>    dir: /data/ingester<br><br># Use memberlist, a gossip-based protocol, to enable the Mimir replicas to communicate <br>memberlist:<br>  join_members: [&quot;mimir-lb-a&quot;, &quot;mimir-lb-b&quot;, &quot;mimir-lb-c&quot;]<br>  randomize_node_name: false<br>  advertise_addr: ADVERTISE_ADDR_REPLACE<br><br># ingester:<br>#   ring:<br>#     replication_factor: 1<br><br>limits:<br>  max_label_names_per_series: 100<br>  ingestion_rate: 100000<br>  ingestion_burst_size: 1000000<br>  max_global_series_per_user: 0<br><br>server:<br>  log_level: info</pre><h4>Nginx configuration</h4><p>Nginx is configured using the sample config below. It adds all mimir instances in upstream section and then sets the server to listen on port 9010 and proxy requests to the upstream backend.</p><pre>events {<br>    worker_connections 1024;<br>}<br><br>http {<br>    upstream backend {<br>        server mimir-lb-a:8080 max_fails=1 fail_timeout=1s;<br>        server mimir-lb-b:8080 max_fails=1 fail_timeout=1s;<br>        server mimir-lb-c:8080 max_fails=1 fail_timeout=1s backup;<br>    }<br><br>    server {<br>        listen 9010;<br>        access_log /dev/null;<br>        location / {<br>            proxy_pass http://backend;<br>        }<br>    }<br>}</pre><h3>Test environment</h3><p>I have created a repo that you can clone to run a test environment of mimir. The repo can be found here <a href="https://github.com/gathecageorge/lb-mimir-instances">gathecageorge/lb-mimir-instances</a></p><p>To test the repo using an online docker playground, go to <a href="https://labs.play-with-docker.com/">https://labs.play-with-docker.com/</a> and login with your docker hub account. Then you can provision a test environment by adding a new docker instance. You just then have to open the terminal and execute the following commands:</p><pre>git clone https://github.com/gathecageorge/lb-mimir-instances.git<br>cd lb-mimir-instances<br>docker compose up -d</pre><p>Once all services are running, open the port 3000 and access grafana to see the metrics. You can use explore option to see all available metrics.</p><h4>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a92bffa5035" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/setting-up-grafana-mimir-in-docker-swarm-environment-1a92bffa5035">Setting up grafana mimir in docker swarm environment</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4. Grafana alerting and custom dashboards]]></title>
            <link>https://blog.devops.dev/4-grafana-alerting-and-custom-dashboards-162752e18637?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/162752e18637</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[alertmanager]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 10:09:34 GMT</pubDate>
            <atom:updated>2023-06-08T09:50:25.900Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>For a more interactive tutorial, you can access my killercoda scenario using this link <a href="https://killercoda.com/gathecageorge/scenario/3-grafana-alerting-create-dashboard">https://killercoda.com/gathecageorge/scenario/3-grafana-alerting-create-dashboard</a>.</blockquote><p>In my other tutorial, we learnt how to setup grafana, loki/promtail and prometheus. This tools help in understanding how the services are doing.</p><p>However its useless if we just reach there. We want to be alerted when something happens in the system. i.e</p><ol><li>When there are many error logs from an application</li><li>When disk capacity is almost running out</li><li>When there is a spike in CPU or memory usage</li><li>etc</li></ol><p>To do that we need alerts. For this scenario, we will setup alerts to slack from grafana.</p><h3>Setup test environment</h3><p>We will use github repo <a href="https://github.com/gathecageorge/vps-monitoring-prometheus-loki">https://github.com/gathecageorge/vps-monitoring-prometheus-loki</a> to setup grafana/promtail/loki/prometheus/portainer as we did in the other scenario.</p><p>The steps are outlined here but if you need more information you can open this scenario <a href="https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana">https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana</a></p><ol><li>Clone the repo git clone <a href="https://github.com/gathecageorge/vps-monitoring-prometheus-loki.git">https://github.com/gathecageorge/vps-monitoring-prometheus-loki.git</a></li><li>Change directory cd vps-monitoring-prometheus-loki/</li><li>Cp .env_template to .env cp .env_template .env</li><li>You can open .env and check its contents. Not required to change for test environment</li><li>Update docker-compose if you have an earlier version. Docker compose version 2 or greater is required. curl -SL https://github.com/docker/compose/releases/download/v2.5.0/docker-compose-linux-x86_64 -o /usr/bin/docker-compose</li><li>Run all services docker-compose up -d</li><li>Open grafana on port 3000. See instructions to open port below.</li></ol><h3>Add custom dashboard</h3><p>To add custom dashboard, click on New Dashboard as shown below. Then Add a new panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jadrKcgJYyXIDIR6.png" /><figcaption>Adding custom Dashboard</figcaption></figure><p>You need to modify the panel as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_M4CmkPzmuSXQaKA.png" /><figcaption>Panel configuration</figcaption></figure><ol><li>For time choose Last 30 minutes</li><li>For title enter Memory Usage</li><li>For Data source choose Prometheus. Change to code from builder.</li><li>For Query type the following 100 - ((node_memory_MemAvailable_bytes{} * 100) / node_memory_MemTotal_bytes{}). This calculates the percentage of the memory available in the server by taking the total free memory / total available memory * 100.</li><li>Expand the options and under Legend choose custom option. Then enter {{instance}} as the value</li><li>Click on Save, enter the name of the dashboard to anything like My Dashboard</li></ol><h3>Now to create alert.</h3><p>We want to be notified when memory usage exceed 70% for a 5min time period.</p><p>Edit the panel as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VTXzn0jIu9nAhauL.png" /><figcaption>Edit Dashboard</figcaption></figure><p>Click on Create alert rule from this panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*Iiri8N9dFwNTv8T9.png" /><figcaption>Create alert</figcaption></figure><p>When the alerts page is open edit the value in C to the percentage you want to be alerted when reached. In this case alert if the memory usage is above 70%.</p><p>Also for folder and Evaluation group, since no value exists, type My Alerts and press enter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LxD0ZbBJC2qw0hxI.png" /><figcaption>Alert configuration</figcaption></figure><p>Scroll down and under Add details for your alert rule click on Add annotation, Choose Summary and enter the value as {{ $labels.instance}} is reporting {{ $values.B.Value }} % memory usage for the last 5 minutes. Kindly review.</p><p>Scroll to the top and click Save and exit. You will be taken to alerts page showing 1 rule and status normal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1q2o6AtgTgVhNDUL.png" /><figcaption>Showing alerts</figcaption></figure><h3>Slack setup</h3><p>You need to create a slack app to receive notifications. Follow the link <a href="https://api.slack.com/apps">https://api.slack.com/apps</a> to create an app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bo3gxDrT3fCthwgU.png" /><figcaption>Create app slack</figcaption></figure><p>Choose From Scratch and enter app name and choose workspace you are a part of.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/0*r6EXW00pWn9WWfmA.png" /><figcaption>App name slack</figcaption></figure><p>Click on Incoming Webhooks and turn on. Then click on Add new webhook to workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/0*nSYh40WkwGjltgVm.png" /><figcaption>Click on incoming webhooks</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/0*k1Onor6ITPdULVFO.png" /><figcaption>Activate webhook and add new webhook</figcaption></figure><p>Then choose the channel where alerts will be sent to. You can create a new channel in slack app if you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*12WqYwA9KSTWt1iu.png" /><figcaption>Choose channel for alerts</figcaption></figure><p>Copy the webhook URL for use on grafana next page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*BLdQspMhtDWg3Cu7.png" /><figcaption>Copy webhook url</figcaption></figure><h3>Grafana alert setup</h3><p>Go back to grafana alerts page and click on Contact points and then Add contact point. Enter details as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xpCda8VzzwwJx3MA.png" /><figcaption>Add new contact point grafana</figcaption></figure><p>Name: slack-alert</p><p>Integration: Slack</p><p>Webhook URL: COPIED FROM ABOVE</p><p>Click on Test and then Send test notification. Make sure you are able to receive alert in Slack. Then you can save contact point.</p><h3>Change alerts to come to slack</h3><p>Click on Notification policies and under Root Policy, click edit and choose default contact point to slack alert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HI6PQoqDJUdYEbT3.png" /><figcaption>Change notification to come to slack policy</figcaption></figure><p>At this time no alert will come since memory usage is below 70%. We can edit the alert and put a value like 20% so the alert is triggered. Also you can change evaluation time to 1m to make it alerted faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y5I-s58tC_uQAEB6.png" /><figcaption>Edit alert</figcaption></figure><p>Click save and exit and wait a while for alert to be evaluated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EzWdkV02YoCd6cWv.png" /><figcaption>Edit alert configuration</figcaption></figure><p>You will receive an alert message like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pCSbaen981VVHQEx.png" /><figcaption>Sample alert received</figcaption></figure><h3>Create alert from logs</h3><p>We need to create an alert from logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9xeD6S8lIp8am6Wr.png" /><figcaption>Add panel to dashboard</figcaption></figure><ol><li>Go to dashboard and choose the previous dashboard My Dashboard.</li><li>Click on add panel icon and choose add new panel</li><li>For name add Loki Logs</li><li>For time choose 30min</li><li>For data source choose Loki</li><li>Choose code option and enter query count_over_time({container_name=&quot;loki&quot;} != &quot;level=info&quot; [1m]). This counts the number of non info logs produced by loki every 1 min.</li><li>Expand options and lengend custom. Enter {{container_name}}</li><li>Click save and then go to alert.</li></ol><h3>Add alert</h3><p>Lets say you want to be alerted when there are more than 5 error logs every 1 min over a 5min period. Enter 5 for value C.</p><p>Choose Folder as My alerts and Evaluation group as My alerts. For time choose 1min so alert is evaluated faster instead of 5min</p><p>Under details for your alert rule, add annotation and choose summary. Add this as value {{ $labels.container_name}} is reporting {{ $values.B.Value }} Errors for the last 1 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*n4EpE-iEqQp2WzFZ.png" /><figcaption>Logs alert configuration</figcaption></figure><p>Click on save and exit.</p><p>Click save when back on dashboard and then go back. You should receive an alert after a while as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6q0q_0Y93cEq-WUI.png" /><figcaption>Sample alert from logs</figcaption></figure><h3>Task</h3><p>To test your learning, create a new panel to show error/warn logs from a random logger docker container. You can start the container using command below. This will generate random logs every 1–3 seconds.</p><pre>docker run -d --name randomlogger --log-driver json-file --log-opt tag=&quot;{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}&quot; chentex/random-logger:latest 1000 3000</pre><p>Once the container is running, you can visualize the logs count per minute using Explore in grafana and the following query. (Select non debug/info logs count)</p><pre>count_over_time({container_name=&quot;randomlogger&quot;} !~ &quot;DEBUG|INFO&quot; [1m])</pre><p>From the dashboard, create an alert so you can be notified when error the value falls below 1. This means be alerted when the container is stopped or does not exist since if its there then the value should always be greater than 1.</p><h3>What&#39;s next</h3><p>From here you can create other alerts as needed. When an alert condition is resolved, say when memory is no longer above 70% or logs are below 5, then you will get an alert of type RESOLVED as below</p><p>Resolved memory usage</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hyEtMLOcUUKTx2HN.png" /><figcaption>Resolved memory alert</figcaption></figure><p>Resolved logs (Note have changed the values trigger resolved like adding high number of logs to alert. Same for memory.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oY4sSrXhggAy-VzM.png" /><figcaption>Resolved logs alert</figcaption></figure><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=162752e18637" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/4-grafana-alerting-and-custom-dashboards-162752e18637">4. Grafana alerting and custom dashboards</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[1. Introduction to Docker and Docker Compose]]></title>
            <link>https://blog.devops.dev/introduction-to-docker-and-docker-compose-bb69d3438794?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb69d3438794</guid>
            <category><![CDATA[killercoda]]></category>
            <category><![CDATA[interactive-content]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[services]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Mon, 06 Mar 2023 19:14:14 GMT</pubDate>
            <atom:updated>2023-06-08T09:46:49.980Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial, we learn about docker and how to deploy containers. You can follow this tutorial if you have docker and docker-compose installed on your machine.</p><blockquote>If you need a test environment on your browser to learn instead of setting up everything on your machine, you can access my Killercoda scenario using this link <a href="https://killercoda.com/gathecageorge/scenario/1-docker-intro">https://killercoda.com/gathecageorge/scenario/1-docker-intro</a>. Killercoda is a free to use interactive environment that allows you to try everything easily without setting up everything on your machine.</blockquote><h3>What is Docker</h3><p>Docker allows you to run your services inside containers. A container is a self contained package of your application and all dependencies required making it possible to run in any machine without any issues.</p><p>You can either package your application to a docker image or use some of the community images from docker hub or any image repository.</p><p>In the first part of this tutorial, we will use the community docker image repository to deploy <a href="https://hub.docker.com/_/mysql">mysql database server</a> and <a href="https://hub.docker.com/_/phpmyadmin">phpmyadmin web interface</a> to access the database.</p><h3>What is Docker compose</h3><p>Docker compose is a tool that helps to define and share multi container applications. Instead of deploying every container manually, you create a <strong><em>yaml </em></strong>file and then use docker compose to deploy all the services defined within that file.</p><p>In the second part of the tutorial, we will write a yaml file to deploy both the services above using one command instead of executing each container individually.</p><p>Lets jump straight to work and run some containers.</p><h3>Step 1</h3><h4>Run Mysql Server</h4><p>We will use the official docker image for MySQL database <a href="https://hub.docker.com/_/mysql">mysql:latest</a> to run the database server. Copy or write the following command to deploy MySQL database server. Its recommended to write the command to for better understanding and mastery.</p><pre>docker run --name mysql-server -e MYSQL_ROOT_PASSWORD=password -d mysql:latest</pre><p><strong>Explanation</strong></p><ol><li>— name Provides the container with a name in this case <strong><em>mysql-server</em></strong></li></ol><p>2. -e Sets an environment variable inside the container. In this case setting the root password for the database. Here you can provide any number of environment variables required by your application by specifying many -e variables. E.g.</p><pre>-e ENV_VAR_1=value1 -e ENV_VAR_2=value2 -e ENV_VAR_3=value3</pre><p>3. -d means we run container in detached mode, i.e. run in background</p><h4><strong>Run phpMyAdmin</strong></h4><p>In order to access the database, we will use <a href="https://hub.docker.com/_/phpmyadmin">phpmyadmin:latest</a>. This will be linked to the mysql-server and expose a web interface on port <strong><em>8080</em></strong>, from there you can login to the database using username <strong><em>root </em></strong>and password <strong><em>password</em></strong></p><pre>docker run --name phpmyadmin -d --link mysql-server:db -p 8080:80 phpmyadmin:latestExplanation</pre><ol><li>— link This is used to link the mysql database to the phpmyadmin. This means the containers can communicate with each other</li></ol><p>2. -p This allows us to expose a port from the container so that we can access it outside in host machine. We want to access port <strong><em>80 </em></strong>inside container using port <strong><em>8080 </em></strong>outside in the host. In your machine for example, opening <a href="http://localhost:8080/">http://localhost:8080</a> will open the container port <strong><em>80</em></strong>.</p><h3>Check running containers</h3><p>You can confirm all containers using the command below. The -a allows you to view all containers whether running or not. If you just want running ones, omit the -a.</p><pre>docker ps -a</pre><h3>Docker compose</h3><p>We need to create a compose file specifying all the services we will need. For file name enter docker-compose.yml</p><p>Then put the contents below to the file.</p><pre>version: &#39;3.1&#39;<br><br>services:<br>  mysql-server:<br>    image: mysql:latest<br>    restart: always<br>    environment:<br>      MYSQL_ROOT_PASSWORD: password<br><br>  phpmyadmin:<br>    image: phpmyadmin:latest<br>    restart: always<br>    # We use port 8081 here because port 8080 is already <br>    # been used by phpmyadmin in the previous step<br>    ports:<br>      - 8081:80<br>    # Instead of linking, we use environment variable to <br>    # set the database server for phpmyadmin<br>    environment:<br>      - PMA_HOST=mysql-server<br></pre><p>Check that the file is saved by running command below. This will show the file content in the terminal.</p><pre>cat docker-compose.yml</pre><h3>Start the services</h3><p>To start all the services specified in the file, run the command below. This starts the services in the background because of the -d attribute. You can now open the target ports and access the port as the previous step showed.</p><pre>docker-compose -f docker-compose.yml up -d</pre><h3>Delete the services</h3><p>To delete the services, just run command below. The -v means delete any docker volumes that might have been created by the services also. Volumes are like persistent storage for the containers.</p><pre>docker-compose -f docker-compose.yml down -v</pre><h3>Delete containers from previous step</h3><p>This deletes the containers from previous step in one command. The -f means force remove the containers even if they are running. Otherwise you would have to stop docker stop mysql-server phpmyadmin them first and then remove them docker rm mysql-server phpmyadmin.</p><pre>docker rm -f mysql-server phpmyadmin</pre><h3>Congratulations</h3><p>You have successfully completed this tutorial. From here you have learnt how to deploy a container and access the container port using the browser.</p><p>Next you will learn about monitoring applications and services running inside your docker environment. For this you can use open source tools like prometheus for metrics collection, loki/promtail for viewing of the logs, grafana for visualizing and querying of the metrics and logs, portainer for management of the docker environment using a web interface.</p><p>You can continue to next chapter in this tutorial here <a href="https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa">https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa</a></p><p>Did you find this tutorial to be helpful, well give it a clap or suggest ways to improve. The most interactive you make it, the more it will help others also.</p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb69d3438794" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/introduction-to-docker-and-docker-compose-bb69d3438794">1. Introduction to Docker and Docker Compose</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3. Introduction to Micro services monitoring (Portainer)]]></title>
            <link>https://blog.devops.dev/2-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-149d1b24f94?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/149d1b24f94</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[promtail]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[portainer]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Thu, 02 Mar 2023 21:09:41 GMT</pubDate>
            <atom:updated>2023-06-08T09:54:35.976Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>For a more interactive tutorial, you can access my killercoda scenario using this link <a href="https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana">https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana</a>.</blockquote><p>In the previous article, we learned about Prometheus/Loki/Grafana for monitoring services. You can checkout the tutorial here <a href="https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa">https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa</a></p><p>When you go through the tutorial, you will already have a running test environment that we can improve here.</p><h3>Portainer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*8olQ_0_xvd9dKOtl.png" /><figcaption>Portainer</figcaption></figure><p>Portainer is a powerful tool that helps in container management. Using this tool, its easy to deploy new services using a web interface. You can read more about it here <a href="https://www.portainer.io/">https://www.portainer.io/</a></p><p>If you are coming here from the last tutorial, you already have portainer running, if not then start one before going through this tutorial using command below in any host with docker installed. Then access the portainer UI using “http://ip_address:9000”</p><pre>docker run -d -p 8000:8000 -p 9443:9443 -p 9000:9000 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer-ce:latest</pre><h3>Portainer User Interface</h3><p>If you want to see portainer running, open it on port 9000. It will allow you to see the containers running at a glance and also run others from its web interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8v7cQwqgnRwRpYkutOIEg.png" /><figcaption>Portainer</figcaption></figure><p>To test deploying a service using portainer, Click on Stack, Add new Stack, Repository and fill details as below.</p><p>Name: mystack</p><p>Repository URL: <a href="https://github.com/gathecageorge/php-docker-template.git">https://github.com/gathecageorge/php-docker-template.git</a></p><p>Compose Path: docker-compose.yml</p><p>You also need to add this environment variables. Click on advanced mode to be able to put the values in text form. See image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQftzVzoO4mgl1Bv7wSjCw.png" /><figcaption>Portainer new stack</figcaption></figure><p>Environment variables content</p><pre>COMPOSE_PATH_SEPARATOR=:<br>COMPOSE_FILE=docker-compose.yml<br>WEB_SERVER_PORT=8081<br>PHPMYADMIN_PORT=8082<br>DATABASE_ROOT_PASSWORD=root<br>DATABASE_USER=testinguser<br>DATABASE_USER_PASSWORD=AveryHardPassword<br>MYSQL_DATABASE=testingdatabase<br>MYSQL_HOST=mysql</pre><p>From there just click Deploy and wait. It will launch a new service with a PHP application and MySQL database.</p><p>You can access PHP application at port 8081 and MySQL phpMyAdmin at port 8082. Username and password for database are indicated above in config file.</p><p>If you are coming from previous article, you can see new open ports in play with docker labs as shown below.</p><p>Showing new ports in play with docker. This shows ports from last lesson (<a href="https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa">https://gatheca-george.medium.com/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa</a>) after adding the portainer stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NP6-eL_GVzEkxze4_yqEDw.png" /><figcaption>Showing Open ports</figcaption></figure><h3>Conclusion</h3><p>Portainer helps you to have an overview of your docker environment and also deploy and run new docker-compose stacks or individual containers from a web interface instead of using docker commands in terminal.</p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=149d1b24f94" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/2-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-149d1b24f94">3. Introduction to Micro services monitoring (Portainer)</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2. Introduction to Micro services monitoring (Prometheus/Grafana/Loki/Promtail)]]></title>
            <link>https://blog.devops.dev/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b16e76f942fa</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[portainer]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Thu, 02 Mar 2023 21:02:52 GMT</pubDate>
            <atom:updated>2023-06-08T09:56:48.953Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>For a more interactive tutorial, you can access my killercoda scenario using this link <a href="https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana">https://killercoda.com/gathecageorge/scenario/2-micro-services-monitoring-grafana</a>.</blockquote><p><em>This tutorial assumes you are already familiar with docker and can run containers in a docker environment. If you need a refresher, check the previous article here </em><a href="https://gatheca-george.medium.com/introduction-to-docker-and-docker-compose-bb69d3438794"><em>https://gatheca-george.medium.com/introduction-to-docker-and-docker-compose-bb69d3438794</em></a></p><p>In the world where micro services have become the norm in software releases, there comes need to be able to monitor performance of the servers and their services. The most common way of packaging this applications have always been docker images. If you have multiple containers running in a server environment, there is need to collect logs and metrics that can assist in troubleshooting and understanding of the way the services are performing.</p><p>This is where this tools come in. This tutorial is structed as below:</p><ol><li>Prometheus highlight</li><li>Promtail/Loki highlight</li><li>Grafana highlight</li><li>Setting up the tools using a github sample library. You can skip upto here to setup a test environment easily and then come back from top to learn the components.</li></ol><h4>1. Prometheus</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*FqL35xvASbIQfYbd.png" /><figcaption>Prometheus metrics collection</figcaption></figure><p>Prometheus is an open source tool that collects metrics and stores them for easier querying using PromQL language. You can collect metrics exposed by any supported source. Using this metrics you can easily setup alerts so that you can be notified when something interesting happens in your server environment like service down.</p><p>To collect metrics about any service, you will need to configure prometheus with scrape targets. This is usually the endpoint and port where metrics are: e.g. <a href="http://myservice:9000/">myservice:9000</a></p><p>You configure prometheus using a yaml file with configuration options. Sample config below</p><pre>global:<br>  scrape_interval: 5s<br>  evaluation_interval: 5s<br><br>scrape_configs:<br>  - job_name: prometheus<br>    scrape_interval: 5s<br>    static_configs:<br>      - targets: <br>        - &#39;localhost:9090&#39;<br>        labels:<br>          instance: &#39;prometheus&#39;<br><br>    # Auto discover scrape endpoints<br>    file_sd_configs:<br>      - files:<br>          - &quot;/etc/prometheus/file_sd/*.yml&quot;</pre><p>Instead of modifying the config file each time you want to add a new scrape target, we can use file discovery as indicated on above config. That way any “*.yml” file created under “/etc/prometheus/file_sd” directory is automatically scraped. Example to scrape node-exporter service that provides metrics about a server like memory usage, disk usage, network etc just create a new file named “/etc/prometheus/file_sd/node-exporter.yml” with content below. You can modify labels as needed, they are really important to query and filter results.</p><pre>- labels:<br>    instance: node-exporter<br>  targets:<br>  - node-exporter:9100</pre><h4>2. Promtail/Loki</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*RaNmq79lnboPtITY.png" /><figcaption>Promtail/Loki/Grafana Logs collection</figcaption></figure><p>Every service will have their own logs. One way to view the logs from a service running in docker would be to use docker logs command as below:</p><pre>docker logs -f --tail 10 mywebserver</pre><p>However this way you can only view the logs and cannot do anything interesting with them. Plus if you have multiple services it becomes a problem to check for each.</p><p>Promtail helps in getting logs from different sources like files, stdin or docker driver. This logs are then forwarded to loki for storage. To identify the logs, labels are used to indicate source, service name, log type ie info, debug, error among many others. You can configure the labels as required for your use case. A sample loki config is below</p><pre>server:<br>  http_listen_port: 9080<br>  grpc_listen_port: 9081<br>  log_level: warn<br><br>positions:<br>  filename: /tmp/positions.yaml<br><br>clients:<br>  - url: http://loki:3100/loki/api/v1/push<br><br>scrape_configs:<br># Collect logs from files in the system<br>- job_name: system<br>  static_configs:<br>  - targets:<br>      - localhost<br>    labels:<br>      job: varlogs<br>      __path__: /var/log/*log<br><br>  - targets:<br>      - localhost<br>    labels:<br>      job: apache<br>      __path__: /var/log/apache2/*log<br><br># Collect logs from docker containers<br>- job_name: containers<br>  static_configs:<br>  - targets:<br>      - localhost<br>    labels:<br>      job: containerlogs<br>      __path__: /var/lib/docker/containers/*/*log<br><br>  pipeline_stages:<br>  - json:<br>      expressions:<br>        output: log<br>        stream: stream<br>        attrs:<br><br>  - json:<br>      expressions:<br>        tag:<br>      source: attrs<br><br>  - regex:<br>      expression: (?P&lt;image_name&gt;(?:[^|]*[^|])).(?P&lt;container_name&gt;(?:[^|]*[^|])).(?P&lt;image_id&gt;(?:[^|]*[^|])).(?P&lt;container_id&gt;(?:[^|]*[^|]))<br>      source: tag<br><br>  - timestamp:<br>      format: RFC3339Nano<br>      source: time<br><br>  - labels:<br>      stream:<br>      container_name:<br><br>  - labeldrop:<br>    - filename<br><br>  - output:<br>      source: output</pre><p>Loki stores the logs collected by Promtail and also retrieves them when querying. For ease of use, you can use grafana frontend for querying both loki logs and Prometheus metrics.</p><p>Sample loki config</p><pre>auth_enabled: false<br><br>server:<br>  http_listen_port: 3100<br>  grpc_listen_port: 9096<br><br>common:<br>  path_prefix: /tmp/loki<br>  storage:<br>    filesystem:<br>      chunks_directory: /tmp/loki/chunks<br>      rules_directory: /tmp/loki/rules<br>  replication_factor: 1<br>  ring:<br>    instance_addr: 127.0.0.1<br>    kvstore:<br>      store: inmemory<br><br>schema_config:<br>  configs:<br>    - from: 2020-10-24<br>      store: boltdb-shipper<br>      object_store: filesystem<br>      schema: v11<br>      index:<br>        prefix: index_<br>        period: 24h</pre><h4>3. Grafana</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*DkAgeD1S9-_VWtz1.png" /><figcaption>Grafana dashboards</figcaption></figure><p>“Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.” <a href="https://en.wikipedia.org/wiki/Grafana">https://en.wikipedia.org/wiki/Grafana</a></p><p>Using grafana, you can query for the logs or metrics by adding Prometheus and Loki as a data source. You can also setup alerts and add dashboards as seen in image above.</p><h3>Setup of Monitoring Tools</h3><p><a href="https://github.com/gathecageorge/vps-monitoring-prometheus-loki">GitHub - gathecageorge/vps-monitoring-prometheus-loki: Allow monitoring a VPS and grafana for dashboards and logs</a></p><p>I have created a repo <a href="https://github.com/gathecageorge/vps-monitoring-prometheus-loki">here (https://github.com/gathecageorge/vps-monitoring-prometheus-loki)</a> that you can use to spin up a monitoring environment. Follow the ReadMe.md inside the repo to set it up on your own server.</p><p>Don&#39;t want to install docker yet, you can spin up a testing environment using play with docker labs: <a href="https://labs.play-with-docker.com/">https://labs.play-with-docker.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L2v9De8CdoQmu1xt.png" /><figcaption><a href="https://labs.play-with-docker.com/">https://labs.play-with-docker.com/</a></figcaption></figure><p>You can then just execute the commands below to spin up a test monitoring services.</p><pre>git clone https://github.com/gathecageorge/vps-monitoring-prometheus-loki.git<br>cd vps-monitoring-prometheus-loki<br>cp .env_template .env<br>docker-compose up -d</pre><p>As one long command. You can use the command in your machine if you have docker installed.</p><pre>git clone https://github.com/gathecageorge/vps-monitoring-prometheus-loki.git &amp;&amp; cd vps-monitoring-prometheus-loki &amp;&amp; cp .env_template .env &amp;&amp; docker-compose up -d</pre><p>Once this is done, 2 ports will be opened; 3000 for grafana and 9000 for portainer. Click on any of them to launch the respective app. You can then login to grafana using details below.</p><pre>Username: myadmin<br>Password: mypass</pre><p>You can checkout sample dashboard under dashboards menu in grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4WlAPAahQ8sCYFinz8vdA.png" /><figcaption>Access sample dashboards</figcaption></figure><p>Node exporter dashboard showing system resource usages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zklyca_py7-jLXxivNxgJQ.png" /><figcaption>Grafana node exporter dashboard</figcaption></figure><p>If you need to view logs or metrics, you can use the explore option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*pU16VDm-Zcty6jLbqgXE8Q.png" /><figcaption>Explore</figcaption></figure><p>You can then select data source; Prometheus for metrics and Loki for logs as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2_uLz0Wl2m2GPS1Xt6J7Q.png" /><figcaption>Choose data source</figcaption></figure><p>A sample query to get memory usage metrics from prometheus scraped from node exporter service is below. Use code option to have control on what you can do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VMSiDnk-9kVRE2ZKA1Dzpw.png" /><figcaption>Prometheus Query</figcaption></figure><p>For logs you can deploy a sample service. In this case its deploying a random logger container that will create random logs to be collected. This will run a random log generator generating logs randomly one for each time between 100 and 400 milliseconds.</p><pre>docker run -d --name randomlogger --log-driver json-file --log-opt tag=&quot;{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}&quot; chentex/random-logger:latest 100 400</pre><p>After that you can query the logs using the container name label randomlogger as below.</p><p>{container_name=”randomlogger”}</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fsG8V9wRQot9LKODQ82Uew.png" /><figcaption>Loki logs</figcaption></figure><p>To filter the log, you can use loki query language. Example instead of showing all logs, only show warning logs as below.</p><p>{container name=”randomlogger”} |= “WARN”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qN7WcikwQfbADEigcoUtlQ.png" /><figcaption>Loki logs filtered</figcaption></figure><h3>Summary</h3><p>We have learnt about services monitoring using Loki/Promtail/Prometheus and Grafana.</p><p>This tools provide a way to collect metrics and logs from running services and explore them in grafana. From there we can add more functionalities like dashboards, alerts etc.</p><p>On the next chapter link here (<a href="https://gatheca-george.medium.com/2-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-149d1b24f94">https://gatheca-george.medium.com/2-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-149d1b24f94</a>) we learn more about portainer to manage services running on the network.</p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b16e76f942fa" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/1-introduction-to-micro-services-monitoring-prometheus-grafana-loki-promtail-portainer-b16e76f942fa">2. Introduction to Micro services monitoring (Prometheus/Grafana/Loki/Promtail)</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6. Spring WebFlux: Load testing using JMeter]]></title>
            <link>https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0875b09fc25</guid>
            <category><![CDATA[jmeter]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 18:31:22 GMT</pubDate>
            <atom:updated>2023-03-09T00:48:46.744Z</atom:updated>
            <content:encoded><![CDATA[<p>For this tutorial we are going to learn how to use Jmeter to perform load test on out API.</p><p>You can download Jmeter from their official website <a href="https://jmeter.apache.org/">https://jmeter.apache.org/</a>. It is a java application meaning it will require java to run and can run in all environments where java is installed.</p><p>Full source code can be downloaded from this repository if you have not been following up with previous tutorials.</p><p><a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/5-tips-and-tricks">GitHub - gathecageorge/spring-webflux-rest-api-intro at 5-tips-and-tricks</a></p><p>Once you have the code, created the database and imported the SQL file in repository “db.sql”, changed the database connection details in “application.properties” file, then you can run the application.</p><p>To test one endpoint, you can invoke “<a href="http://localhost:8080/students/1">http://localhost:8080/students/1</a>” and expect to see such a response meaning API is working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Wp8fghwM4SBplsxjl_3Kg.png" /><figcaption>Sample response from API</figcaption></figure><h4>Load Test</h4><p>Start jmeter and follow steps below.</p><ol><li>Jmeter will launch with a new Test Plan already created. You need to add a “thread group” to the test plan as shown below. This will allow us to simulate the number of users who are accessing the website simultaneously. After added change number of thread(concurrent users) to 200, ramp up period(all users access concurrently) to 0, and Loop count to 10.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KciOzcPX4JbQtKj5Nvqq8Q.png" /></figure><p>2. Add a http request to thread group as shown below. This will allow us to make HTTP requests to the api endpoints that we want to test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvzrMvdkjtVxe_z0fktN3g.png" /></figure><p>3. To configure HTTP request, you need to fill all this information.</p><ul><li>Protocol is http, if you are testing on a server with SSL then you can use https.</li><li>Server is localhost or where API is running</li><li>Port is 8080 in localhost or 80 depending on how you are running application</li><li>For HTTP request method we can test any endpoint. Eg get all students which is a HTTP GET method endpoint so we select GET and path put “students”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ZHc9VvEPdlIgDg9eaJPsA.png" /></figure><p>4. Add a POST http request. For this just add another http request to thread group. For this choose post method and under “Body Data” add the following JSON from Line 1 to 3.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5914ba24eb8206f9c6872f87fe411b5f/href">https://medium.com/media/5914ba24eb8206f9c6872f87fe411b5f/href</a></iframe><p>Our post endpoint expects request to have header “Content-Type” with value “application/json” otherwise cannot perform the action. For this right click the http request that is post and add Config Element -&gt; HTTP Header Manager then add t header “Content-Type” to it with above value.</p><p>5. We are almost done. Now we need to be able to visualise the output of both the requests *GET students and *POST students. For this on each HTTP request right click and add listener “View Results Tree”. This will help us view every request sent for each endpoint. We also need aggregate report for both so on Thread Group add a listener “Aggregate Report”. At the end you should have the following structure.</p><p>Once all of this is done, we can now execute test plan and view results. Click on run button and wait for results. Below you can see sample results of the GET and POST endpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OjD7CI_p2Ukdy_4S4lTtZA.png" /><figcaption>Aggregated results summary</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W-XeaFsMzWQnoh6Hq6epkw.png" /><figcaption>Results for post method</figcaption></figure><h4>Understanding the results:</h4><p>The key values from above test are:</p><ul><li>Error % — This will tell us the percentage of request that are getting dropped or encountering an error. In our case you can see all requests are successful as error is 0%. You can increase the number of concurrent users in Thread Group and each time observe the error rate to determine the threshold hold of the API; ie the point where requests starts getting errors. If you have a non reactive API doing the same thing, you will observe that threshold is much lower than reactive meaning reactive is better at handling more users.</li><li>Min/Maximum/Average — This will tell us minimum, maximum and average response time for a single request. You can also compare with a non reactive API doing same thing and observe that its much lower for reactive meaning API is much more responsive. Even with simulated pause of 2 seconds on GET endpoints for each student in DB, we can see no request is getting dropped.</li></ul><h3>Summary</h3><p>Well this has been a rather long tutorial about spring reactive API development. I know it can be hard and challenging to learn a new method of programming when used to sequential design but as you have seen, the benefits are many. While this is not a comprehensive tutorial, it will form a very strong foundation for exploring and using reactive programming on your day to day activities.</p><p>Finally am really grateful for you taking your time to learn with me. There are always ways to improve and if you have any feedback, compliments or suggestions for new things you might want to learn then reach out via the following channels and will be sure to respond.</p><p>LinkedIn: <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">https://www.linkedin.com/in/gathecageorgenjoroge/</a></p><p>Medium: <a href="https://medium.com/@gatheca-george">@gatheca-george</a></p><p>Twitter: <a href="https://twitter.com/gatheca_george">https://twitter.com/gatheca_george</a></p><p>Happy Learning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*XmF68dKKquA9Zc5ZTis46A.png" /><figcaption>Happy Learning</figcaption></figure><h3>WHATS NEXT</h3><blockquote>So what’s next after this? Well checkout for the next articles on the series.</blockquote><ol><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">Spring WebFlux: Reactive micro service Introduction</a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226"><em>Spring WebFlux: Reactive micro service Practical guide Intro</em></a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f">Spring WebFlux: Using Relational Database (MySQL &amp; PostgreSQL)</a></li><li><a href="https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d">Spring WebFlux: Unit testing your REST API (Continuation)</a></li><li><a href="https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe">Spring WebFlux: Tips &amp; Tricks, Common mistakes</a></li><li><a href="https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25">Spring WebFlux: Load testing using JMeter</a> <strong>(You are here)</strong></li></ol><p>Want to learn more, have a topic in JAVA you might need articles on or have questions? Reach out to me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a></p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0875b09fc25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5. Spring WebFlux: Tips & Tricks, Common mistakes]]></title>
            <link>https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/123cb1bbf2fe</guid>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[senior-developer]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[tips]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 18:30:37 GMT</pubDate>
            <atom:updated>2023-03-09T00:49:00.858Z</atom:updated>
            <content:encoded><![CDATA[<p>We have learnt about creating micro services in a non blocking way using spring boot in previous articles.</p><p>In this tutorial, I want us to learn about common mistakes that one can make when starting out with Spring webflux. While there is no one single way of doing something, this will help you to prevent some runtime errors and create more maintainable code.</p><p><strong>1.New Mindset</strong></p><p>Even experienced programmers coming from writing sequential code can feel lost when learning reactive programming. Some very simple tasks that can be written as a sequence of blocking calls can be confusing and not clear at one go when expressed using flatMap/map. One thing to know is to never stop learning and exploring.</p><p><strong>2.Logging</strong></p><p>One of the critical requirements for any application is logging. In traditional approach of programming, its easy to trace logs for a certain request as its processed sequentially from start to end. This can be hard in reactive programming where a request can be handled together with other requests as the application reacts to different events.</p><p>To make log tracing easier in reactive programming, there needs to be a unique key/ID for each request that is been handled. That key will be added to the logs and together with log aggregators that collect all logs in one place, its easier to search for actions performed on a request using that ID.</p><p><strong>3.flatMap/map usage</strong></p><p>There is always a question, is there a difference between flatMap and map? Well yes there is a difference.</p><p>flatMap is used for non blocking operation in this case an operation that will return a Mono or Flux.</p><p>map is used for blocking operation that can be done in fixed time. An example would be transforming an object.</p><p>Example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a818e4b6654a64b035edb7094a7f0775/href">https://medium.com/media/a818e4b6654a64b035edb7094a7f0775/href</a></iframe><p><strong>4.Nesting flatMap</strong></p><p>Creating nested flatMap can be really hard to fix when there is a bug. At one glance its very hard to know what is going on. A better approach would be to chain the flatMap as shown in example below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df9ebb81d9b4fc73b9c421ecbed78095/href">https://medium.com/media/df9ebb81d9b4fc73b9c421ecbed78095/href</a></iframe><p>Summary:</p><p>We have learnt about some of the best practices of reactive programming using webflux. While this is not an exhaustive list; You can research and learn more about this.</p><p>I wish you happy coding.</p><h3>WHATS NEXT</h3><blockquote>So what’s next after this? Well checkout for the next articles on the series.</blockquote><ol><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">Spring WebFlux: Reactive micro service Introduction</a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226"><em>Spring WebFlux: Reactive micro service Practical guide Intro</em></a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f">Spring WebFlux: Using Relational Database (MySQL &amp; PostgreSQL)</a></li><li><a href="https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d">Spring WebFlux: Unit testing your REST API (Continuation)</a></li><li><a href="https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe">Spring WebFlux: Tips &amp; Tricks, Common mistakes</a> <strong>(You are here)</strong></li><li><a href="https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25">Spring WebFlux: Load testing using JMeter</a></li></ol><p>Want to learn more, have a topic in JAVA you might need articles on or have questions? Reach out to me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a></p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=123cb1bbf2fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4. Spring WebFlux: Unit testing your REST API (Continuation)]]></title>
            <link>https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2dd0298892d</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[mockito]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[spring]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 18:27:05 GMT</pubDate>
            <atom:updated>2023-03-09T00:49:13.558Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>On the previous article, </em><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f"><em>we improved our application by adding database support to it.</em></a><em> On this tutorial we will further improve on that by doing unit tests to the application.</em></blockquote><blockquote>Full code for this tutorial can be found on GitHub branch “4-unit-testing” by visiting this link. <a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/4-unit-testing">https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/4-unit-testing</a></blockquote><blockquote>Alternatively you can get code of previous tutorial and code along in this tutorial from the “3-using-relational-database” branch here <a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/3-using-relational-database">https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/3-using-relational-database</a></blockquote><h3>INTRODUCTION</h3><p>For this course we are going to use mockito and JUnit5 to do some testing for the StudentsController.java class. We will not go ahead to test every object or class in the application as its supposed to be done but concentrate on reactive part of the application only which is on the controller.</p><p>Other parts like Models, DTOs and SampleController.java class can be tested the same way using JUnit5.</p><p>To start create a class StudentsControllerTest.java on “test” workspace. For those using IntelliJ IDE, one can just use shortcut to create a skeleton of the test class by going to StudentsController.java file, and right click on class name, then choose generate Test. Select all the methods and also add a setup method.</p><h4>Setup Method</h4><p>The setup method will create an instance of controller for before running each test. We need 2 parameters when creating the controller which we mock using Mockito. We will then mock each action on a repository as required by the test methods.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f8e93bc8d4f62e52aba66ef8f22123f/href">https://medium.com/media/8f8e93bc8d4f62e52aba66ef8f22123f/href</a></iframe><h4>1. GetStudentTest</h4><p>To test this method, we need to call controller getStudent() method and pass a value like 1.</p><ul><li>We then need to check positive case that the result is okay and no error response.</li><li>We then need to test for negative case, i.e. pass a value not contained in the database and also test that expected error or message is returned.</li></ul><p>For this tests we need don’t need to connect to a real database which the repository uses to simulate a correct value and a not found value. For that we mock the repository findById() method and return a record when 1 is passed and empty when 2 is passed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4223e02ad621c88621eb3d6f756b7a25/href">https://medium.com/media/4223e02ad621c88621eb3d6f756b7a25/href</a></iframe><h4>2. GetStudentsTest</h4><p>To test this we use same concept as above. This time response expected is not of type Mono but Flux since we can get many. I have indicated using inline comments the code as necessary.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/493860d37ae69cdd72c6172bf8a08077/href">https://medium.com/media/493860d37ae69cdd72c6172bf8a08077/href</a></iframe><h4>3. AddStudentTest</h4><p>For this we introduce another concept which is simulate adding record to database and then set the DB id returned on an object then we can verify save was okay by checking object now has an ID.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08db9fb409f753b3e328e9fd948e6b73/href">https://medium.com/media/08db9fb409f753b3e328e9fd948e6b73/href</a></iframe><h4>4. UpdateStudentTest</h4><p>For this we introduce another concept of verifying that data has really changed. You can follow from inline comments in code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66aa4fd19ce566a2db73988dad1952e7/href">https://medium.com/media/66aa4fd19ce566a2db73988dad1952e7/href</a></iframe><h4>5. DeleteStudentTest</h4><p>As a learning lesson, we have intentionally left the test for this method and its up to you to implement. I would be happy to assist in case of any problem by reaching me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a> I wish you happy learning and making small improvements to how you code.</p><h3>CONCLUSION</h3><p>Well that was a lot of things in one lesson? What have we learnt?</p><ul><li>We have learnt to use mockito to mock classes and actions that would otherwise require using a database and making unit tests run faster</li><li>We have learnt how to assert and verify output of our methods using StepVerifier</li><li>We have learnt of how to have fun while coding. Its not usually as hard as it seems.</li></ul><h3>WHATS NEXT</h3><blockquote>So what’s next after this? Well checkout for the next articles on the series.</blockquote><ol><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">Spring WebFlux: Reactive micro service Introduction</a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226"><em>Spring WebFlux: Reactive micro service Practical guide Intro</em></a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f">Spring WebFlux: Using Relational Database (MySQL &amp; PostgreSQL)</a></li><li><a href="https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d">Spring WebFlux: Unit testing your REST API (Continuation)</a> <strong>(You are here)</strong></li><li><a href="https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe">Spring WebFlux: Tips &amp; Tricks, Common mistakes</a></li><li><a href="https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25">Spring WebFlux: Load testing using JMeter</a></li></ol><p>Want to learn more, have a topic in JAVA you might need articles on or have questions? Reach out to me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a></p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2dd0298892d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3. Spring WebFlux: Using Relational Database (MySQL & PostgreSQL)]]></title>
            <link>https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcc5e487f57f</guid>
            <category><![CDATA[crud]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[spring-data]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Sat, 08 May 2021 18:10:10 GMT</pubDate>
            <atom:updated>2023-03-09T00:49:29.121Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>On the previous article, <a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226">we learned how to create a rest API using spring WebFlux</a>. On this tutorial we will improve on that by adding database support.</blockquote><blockquote>Full code for this tutorial can be found on GitHub branch “3-using-relational-database” by visiting this link. <a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/3-using-relational-database">https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/3-using-relational-database</a></blockquote><blockquote>Alternatively you can get code of previous tutorial and code along in this tutorial from the “2-practical-guide-intro” branch here <a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/2-practical-guide-intro">https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/2-practical-guide-intro</a></blockquote><p>The first step will be adding extra dependencies to our existing project. Open pom.xml and add the following lines between the opening “&lt;dependencies&gt;” and closing “&lt;/dependencies&gt;” tags.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8656184faeb378b33bba6c424edc340/href">https://medium.com/media/b8656184faeb378b33bba6c424edc340/href</a></iframe><p>Now we can model our sample database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*vIK4zRZBOBWfnXdL1td_kg.png" /><figcaption>Our database design</figcaption></figure><p>You can create the database by running SQL statements on this file.</p><p><a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/blob/database/db.sql">https://github.com/gathecageorge/spring-webflux-rest-api-intro/blob/database/db.sql</a></p><h4>Model Our Database &amp; Repository</h4><p>We will use spring data to query for records from the database. The first step is to create models for the 3 tables as follows. We are using Lombok dependency to generate boiler plate code like setters, getters, constructors and builder pattern for our POJOs. @Table annotation comes from Spring Data and allows us to specify the table name of our model. Also each class has @Id denoting the primary key for the table and @Column specifying the column name for the attribute.</p><h4>Students Table</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfa83b9b52bab6b8ef42bf5180f52ce8/href">https://medium.com/media/dfa83b9b52bab6b8ef42bf5180f52ce8/href</a></iframe><p><strong>Courses Table</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a1d9254e3d796ec176643b108a18894/href">https://medium.com/media/0a1d9254e3d796ec176643b108a18894/href</a></iframe><p><strong>Coursework Table</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aecf7d075f647112523b5d29b6149213/href">https://medium.com/media/aecf7d075f647112523b5d29b6149213/href</a></iframe><h4>Query for a student</h4><p>Now that we have the database, our first task is to query for a single students information from the database in a reactive way. Create “StudentsController” on controller package and a method “getStudent”. Then create a repository for students table. See below;</p><p><strong>StudentsRepository.java</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9abd99bb27fa1204fcf7363119877e8e/href">https://medium.com/media/9abd99bb27fa1204fcf7363119877e8e/href</a></iframe><p>Note here the repository interface extends ReactiveCrudRepository to provide us with out of the box crud on table students in a reactive way.</p><h4>StudentsController.java</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/462d182ebb17446169ca978fd6279a76/href">https://medium.com/media/462d182ebb17446169ca978fd6279a76/href</a></iframe><p>As you can see, we are returning a Mono on our get student method. We can just as well return “Mono&lt;Student&gt;” but we want our API to be restful meaning we can modify http status codes in response. Thus we use ResponseEntity. We are also using spring and Lombok @AllArgsConstructor to auto inject an instance of StudentsRepository.java object to our controller.</p><p>The @GetMapping(“/students/{studentID}”) means we are expecting Http get method and a parameter which is the studentID on URL which will be put into studentID variable by @PathVariable. What remains is just to call repository method “findByID” passing the ID received and mapping(transforming) the response received to a ResponseEntity with correct status.</p><h4>Run the application</h4><p>One more step to run application, we need to update application.properties file with DB connection details as follows. Change username and password as required for location of your database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7403d06dd0525e3cb75a6deb7747304/href">https://medium.com/media/d7403d06dd0525e3cb75a6deb7747304/href</a></iframe><p>When all this is done and you run the application, you can get users by visiting “http://localhost:8080/students/1” to get details about student 1. Sample response below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuOOf1Lt_WjQrqGlMUhwlA.png" /><figcaption>Using tabbed postman to invoke endpoint created.</figcaption></figure><h4>Adding a student to database.(Saving data)</h4><p>To add a student create another method in StudentsController as below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76f302fa5280b22aebb8802bb5a9f527/href">https://medium.com/media/76f302fa5280b22aebb8802bb5a9f527/href</a></iframe><p>We are now using @PostMapping to indicate endpoint is only reachable via http POST method. We then accept json sent with @RequestBody and create a students object from it. We then set status to active and registered date to current time. We just then call repository save method and map saved object to the ResponseEntity and Http Status Created. These means we dont have to expect a client to send status and date. See sample response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_UneA_oMVH2qHmvtXzGXdQ.png" /><figcaption>Saving a user.</figcaption></figure><h4>Updating a student</h4><p>To update a student, we first need to find the student and if found we update the object and save to the database. If not found we need to return a specific message with student not found.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/372ddeb49dc8610b1200a1b14b668164/href">https://medium.com/media/372ddeb49dc8610b1200a1b14b668164/href</a></iframe><p>Sample response when a valid studentID is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kuAwZ1kMv_IffHXN6MmhGw.png" /></figure><p>When an invalid studentID is sent an error is thrown and client gets this response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*5XtlOvrvMPuzYT56yK9jiA.png" /></figure><p>That’s is not soo good response, we need to return a better message to the user indicating what occurred. Let do some modifications. Create a class to model a response as below. We are using generic programming to be able to reuse the class across the application.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf037dab2368c7a8275f822927bd301c/href">https://medium.com/media/cf037dab2368c7a8275f822927bd301c/href</a></iframe><p>Then we will modify our update method on controller as follows. NB we have introduced onErrorResume to catch the error thrown when no student was found and create a custom response. Now our response is always consistent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2de37184016f4cf67346e50fe4e8e279/href">https://medium.com/media/2de37184016f4cf67346e50fe4e8e279/href</a></iframe><p>Final response when studentID is found</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Mp57qEQFmAI_C4LuB9bYw.png" /></figure><p>Final response when studentID is NOT found</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UsVWPZZ4oAxNde3pUJYxEw.png" /></figure><p><strong>Deleting a student record</strong></p><p>For whatever reason, though not recommended(always update status to deleted) and a bad practice, you might need to delete a student record. This will involve deleting all students coursework since its not possible to delete if coursework exists. This id due to database constraints set on the database with a foreign key on coursework table referencing students table. We will need to create a coursework repository to provide CRUD on coursework table. Create one as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54fc1b81c6fff6f6ed782ad6b058d8b3/href">https://medium.com/media/54fc1b81c6fff6f6ed782ad6b058d8b3/href</a></iframe><blockquote>We are creating a new method that will delete coursework for a student given studentID. Good thing about spring, we dont need to provide implementation of the method since we have named the method intentionally as “deleteByStudentID”. We have a column named studentID on the model and using that we can create any combination crud operation using a special naming convenction and spring will provide the implementation. e.g “findByStudentID” or “findByStudentIDAndCourseID”.</blockquote><p>Then we need to create the object to be autowired on the controller as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/584eee11401b32192d4b6aed1479ab2f/href">https://medium.com/media/584eee11401b32192d4b6aed1479ab2f/href</a></iframe><p>Then we can create delete endpoint as follows. NOTE @Transactional this will make sure the delete operation happens inside a transaction. i.e. either pass all or fail all</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29af205d933e9edac153ad7319ba2168/href">https://medium.com/media/29af205d933e9edac153ad7319ba2168/href</a></iframe><p>Sample response on calling delete. Just as courtesy we are returning deleted model to the client. A message would be enough too. If student is not found similar message is returned as when we were checking on update.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cBmq7gtK7H2xbBJuvvEOiA.png" /></figure><h4>Querying Multiple Students from Database</h4><p>To query multiple students from database, remember from 1st tutorial, we need to return a flux of Students. We also annotate with @GetMapping and add produces to signify stream data is returned.</p><p>We are also accepting some parameters like page number and students per page(limit) and any filter params. If none is provided default values are used. We then call a method “findAllByStatusAndName” on student repository that we will create, then add a delay to simulate slow loading of responses.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f8c00b46f5045052687c387c046dd26/href">https://medium.com/media/6f8c00b46f5045052687c387c046dd26/href</a></iframe><p>Now lets add the method to students repository. This time we need to add custom query to allow with filter and same time pagination.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/633cd533e2b03e066fd977e64d5a5e3a/href">https://medium.com/media/633cd533e2b03e066fd977e64d5a5e3a/href</a></iframe><p>When we call the endpoint sample response can be seen here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*lEvCSyOaMwhlmaONxd26iQ.gif" /><figcaption>Students been returned in a stream</figcaption></figure><h3>Summary</h3><p>Okay that was a long lesson. I hope it was worth it. Well here lets summarize what we have learnt so far.</p><ol><li>We have learnt how to use spring data on our API. We have created table models, created repositories and auto wired them to a controller using Lombok @AllArgsConstructor annotation.</li><li>We have learnt how to retrieve data from database. Either a single record or multiple records.</li><li>We have learnt how to save and update a record</li><li>We have learnt how to delete a record and chain actions on the database ie delete dependencies before deleting main data for integrity checks.</li><li>We have learnt how to paginate responses from the database.</li></ol><h3>WHATS NEXT</h3><blockquote>So what’s next after this? Well checkout for the next articles on the series.</blockquote><ol><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">Spring WebFlux: Reactive micro service Introduction</a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226"><em>Spring WebFlux: Reactive micro service Practical guide Intro</em></a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f">Spring WebFlux: Using Relational Database (MySQL &amp; PostgreSQL)</a> <strong>(You are here)</strong></li><li><a href="https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d">Spring WebFlux: Unit testing your REST API (Continuation)</a></li><li><a href="https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe">Spring WebFlux: Tips &amp; Tricks, Common mistakes</a></li><li><a href="https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25">Spring WebFlux: Load testing using JMeter</a></li></ol><p>Want to learn more, have a topic in JAVA you might need articles on or have questions? Reach out to me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a></p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcc5e487f57f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2. Spring WebFlux: Reactive micro service Practical guide Intro]]></title>
            <link>https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226?source=rss-c761e483cd5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeadf733a226</guid>
            <category><![CDATA[spring]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[reactive]]></category>
            <dc:creator><![CDATA[George Gatheca]]></dc:creator>
            <pubDate>Sat, 08 May 2021 13:45:58 GMT</pubDate>
            <atom:updated>2023-03-09T00:49:42.972Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>On my previous article <a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">“Spring WebFlux: Reactive micro service Introduction”</a> we discussed why one should think of using reactive spring to build a micro service specifically a REST API. On this article we are going to create a sample reactive API. NB:</blockquote><blockquote>Full code for this tutorial can be found on GitHub branch “2-practical-guide-intro” by visiting this link. <a href="https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/2-practical-guide-intro/">https://github.com/gathecageorge/spring-webflux-rest-api-intro/tree/2-practical-guide-intro/</a></blockquote><p>To get started you will need java setup on your machine. I will assume basic knowledge of java and spring. First create a spring project by going to <a href="https://start.spring.io/">https://start.spring.io/</a>. You need to select the following dependencies. Alternatively you can click <a href="https://start.spring.io/#!type=maven-project&amp;language=java&amp;platformVersion=2.4.5.RELEASE&amp;packaging=jar&amp;jvmVersion=11&amp;groupId=com.gatheca&amp;artifactId=reactive-demo&amp;name=reactive-demo&amp;description=Reactive%20API%20demo&amp;packageName=com.gatheca.reactive-demo&amp;dependencies=lombok,webflux">this link for preset configs.</a></p><ol><li>For project type select maven.</li><li>For language select JAVA</li><li>For Spring boot select any latest spring version (Not SNAPSHOT)</li><li>For Project metadata enter your details as required. Checkout the one I used below:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*5tkberst-LoxWHuN-pjV3Q.png" /></figure><p>Once you select all this dependencies, click generate to get a project zip downloaded to your machine. Then you need to extract the ZIP to a location in your machine and open it with your favourite IDE. I recommend using Jetbrains Intellij. You can download the community edition by <a href="https://www.jetbrains.com/idea/download/">following this link.</a> Once all this is done you are ready to dive into code.</p><h3>Coding a simple Rest API Endpoint</h3><p>Once you open your project with Intellij, the first time will take some time while its downloading all dependencies required. Then you will be able to edit the code.</p><h4>Create a controller and endpoint</h4><p>First step will be to create a new controller and a http GET endpoint that will return a mono of string and invoke the endpoint using the browser.</p><p>Create a class “SampleController.java” on package “controller”.</p><pre>package com.gatheca.reactivedemo.controller;<br><br>import org.springframework.web.bind.annotation.GetMapping;<br>import org.springframework.web.bind.annotation.RestController;<br>import reactor.core.publisher.Mono;<br><br>@RestController<br>public class SampleController {<br>    @GetMapping(&quot;/sample1&quot;)<br>    Mono&lt;String&gt; sample1() {<br>        return Mono.<em>just</em>(&quot;Hello world&quot;);<br>    }<br>}</pre><p>So we have a class SampleController, we have annotated it with “@RestController” and also added a method that returns “Mono&lt;String&gt;”. The method is annotated with “@GetMapping” to mean its accessible via http get method on location “/sample1” from base URL. To create a mono instance we are using Mono.just() and passing the string we want returned.</p><p>To run our application, open the main method class in my case “ReactiveDemoApplication” and click on green run button next to main method and choose “Run ReactiveDemoAp…main()”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*OkVhIvsk2qHRiRlvYuOj7Q.png" /></figure><p>To invoke the application on browser, just type “http://localhost:8080/sample1” on the browser. By default the application starts using port 8080. The output is as expected, having a string “Hello World” on the browser. Nothing interesting but we have just made the 1st endpoint in reactive spring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*HGch_CrzMJHdCc55v3P5EA.png" /><figcaption>Sample response</figcaption></figure><h4>Make things interesting</h4><p>Lets say we want to create an endpoint that provides real time prices for a stock. We want to return a list of integers in a reactive way. To do that we create another endpoint on the controller like below. You will notice we use flux instead of mono since we are providing more than 1 response. Inside Flux.just() we provide a list of items we want returned comma separated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*NmnyVlPyZcZblxsUE0CAdw.png" /><figcaption>NOTE: The new endpoint is accessible through /sample2</figcaption></figure><p>Response from browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*XZY4QxY0u86idXWUq6EUeg.png" /><figcaption>The browser responds with a list of numbers we provided</figcaption></figure><p>Well this is still not interesting. The browser is waiting for all results to be computed and returned at once. That is why its even displayed as a list on one line. We want to stream the prices as each is calculated. For that we make some small changes. We introduce a delay when returning the prices to simulate process of calculation or database call, then we add “.log” to show logs as each item is streamed. Also we add produces value to the “@GetMapping” to mean method can produce a stream for supported clients. Those without support will just wait till all data is returned to begin processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h035n9LfM4T5UIJqgfANgA.png" /><figcaption>Allow streaming of values to client</figcaption></figure><p>Sample response from a browser that supports stream data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*_s9n_J6oT1WvZjWWSnDFuQ.gif" /><figcaption>Response from a browser or client that supports stream.</figcaption></figure><p>Sample response from <a href="https://chrome.google.com/webstore/detail/tabbed-postman-rest-clien/coohjcphdfgbiolnekdpbcijmhambjff?hl=en">tabbed postman</a> that does not support stream data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hdweZ0GhjRU2-aaNXAau1g.png" /><figcaption>If the client does not support stream then the output will be in a single line, its waiting to until all data is loaded to display.</figcaption></figure><p>Here we can see console on the IDE showing logs as values are emitted to the client. As you can see, when data is first requested, client subscribes to get data and passes the request with start value. Every time a new value is available then an onNext event is published with the value. When all data is sent then an onComplete event is sent telling client not to expect any other set of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LjAc3StbU5389tdil0kJAw.gif" /></figure><h3>WHATS NEXT</h3><blockquote>So what’s next after this? Well checkout for the next articles on the series.</blockquote><ol><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-introduction-3ad0659169df">Spring WebFlux: Reactive micro service Introduction</a></li><li><a href="https://gatheca-george.medium.com/spring-webflux-reactive-micro-service-practical-guide-intro-eeadf733a226"><em>Spring WebFlux: Reactive micro service Practical guide Intro</em></a><em> </em><strong>(You are here)</strong></li><li><a href="https://gatheca-george.medium.com/spring-webflux-using-relational-database-mysql-postgresql-fcc5e487f57f">Spring WebFlux: Using Relational Database (MySQL &amp; PostgreSQL)</a></li><li><a href="https://gatheca-george.medium.com/4-spring-webflux-unit-testing-your-rest-api-continuation-b2dd0298892d">Spring WebFlux: Unit testing your REST API (Continuation)</a></li><li><a href="https://gatheca-george.medium.com/5-spring-webflux-tips-tricks-common-mistakes-123cb1bbf2fe">Spring WebFlux: Tips &amp; Tricks, Common mistakes</a></li><li><a href="https://gatheca-george.medium.com/6-spring-webflux-load-testing-using-jmeter-b0875b09fc25">Spring WebFlux: Load testing using JMeter</a></li></ol><p>Want to learn more, have a topic in JAVA you might need articles on or have questions? Reach out to me on my profile or <a href="https://www.linkedin.com/in/gathecageorgenjoroge/">via LinkedIn here.</a></p><h3>Hello👋, Did you find this tutorial helpful, please click the clap 👏 button below to show your support for the author 👇. Also this will help the algorithm for others who are learning to see this story.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeadf733a226" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>