<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Murat Bilal on Medium]]></title>
        <description><![CDATA[Stories by Murat Bilal on Medium]]></description>
        <link>https://medium.com/@murat.bilal?source=rss-7946e5aff990------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZKKl70tDPrMpluSH</url>
            <title>Stories by Murat Bilal on Medium</title>
            <link>https://medium.com/@murat.bilal?source=rss-7946e5aff990------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:22:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@murat.bilal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Ping That Never Arrived: Debugging Linux rp_filter and Asymmetric Routing]]></title>
            <link>https://medium.com/@murat.bilal/the-ping-that-never-arrived-debugging-linux-rp-filter-and-asymmetric-routing-e93c7d054650?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/e93c7d054650</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[redhat-linux]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 08:37:05 GMT</pubDate>
            <atom:updated>2026-09-09T08:37:05.125Z</atom:updated>
            <content:encoded><![CDATA[<p>A real-world troubleshooting case where tcpdump seemed to show no traffic, Linux reported martian packets, and strict reverse-path filtering was quietly dropping valid connections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*DwF9ANlZt-Elg2BY" /></figure><p>I recently ran into a Linux networking issue that initially looked like a straightforward routing, firewall, ARP, or virtualization problem.</p><p>Two Linux VMs were reachable through the same network:</p><pre>192.0.2.28<br>192.0.2.29<br><br>From my Windows client:<br><br>ping 192.0.2.28   -&gt; works<br>ping 192.0.2.29   -&gt; timeout</pre><p>Both Linux machines could reach their default gateway.</p><p>At first, the usual suspects came to mind:</p><ul><li>firewall or ACL</li><li>stale ARP entry</li><li>incorrect VM MAC address</li><li>duplicate IP</li><li>virtualization bridge problem</li><li>incorrect VLAN</li><li>guest Linux firewall</li><li>routing issue</li></ul><p>Traffic was reaching the Linux server through a different interface than Linux expected, and strict reverse-path filtering was rejecting it.</p><p>Changing rp_filter made the ping work, but it did not necessarily fix the asymmetric network path that caused the problem.</p><p>The problematic Linux VM was multi-homed.</p><pre>ip -br addr<br><br>lo       UNKNOWN  127.0.0.1/8<br>enp1s0   UP       192.0.2.29/27<br>enp2s0   UP       198.51.100.237/27<br><br>Its default gateway was reachable through enp1s0 --&gt; 192.0.2.3<br><br>ip route<br><br>default via 192.0.2.3 dev enp1s0 proto static metric 100<br><br>192.0.2.0/27 dev enp1s0 proto kernel scope link<br>    src 192.0.2.29 metric 100<br><br>198.51.100.224/27 dev enp2s0 proto kernel scope link<br>    src 198.51.100.237 metric 101<br><br>At first glance, nothing looked obviously wrong.<br><br>The VM could ping its gateway,but a remote client could not ping 192.0.2.29<br><br></pre><p>Because 192.0.2.28 was reachable while 192.0.2.29 was not, I initially suspected an ARP or MAC-related problem.</p><p>I started capturing traffic on the interface where 192.0.2.29 was configured:</p><pre>tcpdump -ni enp1s0 &#39;icmp or arp&#39;<br><br>ARP, Request who-has 192.0.2.3 tell 192.0.2.29<br>ARP, Reply 192.0.2.3 is-at 00:00:0c:aa:bb:cc<br><br>But while the Windows client was continuously pinging 192.0.2.29, I saw no ICMP echo requests.</pre><p>The traffic is not reaching the vm.</p><p>I even changed the VM’s MAC address, but no luck.</p><p>It eliminated stale MAC/ARP behavior as the primary cause.</p><p>While investigating /var/log/messages, I noticed entries similar to:</p><pre>IPv4: martian source 192.0.2.29 from 203.0.113.132, on dev enp2s0.<br>IPv4: martian source 192.0.2.29 from 203.0.113.14, on dev enp2s0</pre><p>Linux was seeing the traffic. The destination IP 192.0.2.29 was configured on enp1s0 Yet the packets were arriving through enp2s0. That immediately pointed toward asymmetric routing.</p><p><strong>ip route get Revealed the Mismatch:</strong></p><p>Windows client is 203.0.113.132</p><pre>ip route get 203.0.113.132<br>203.0.113.132 via 192.0.2.3 dev enp1s0 src 192.0.2.29<br><br>The incoming packet arrived on enp2s0<br>but Linux&#39;s return route toward the source used enps10</pre><p>That is asymmetric routing.</p><p>And asymmetric routing becomes especially important when Linux reverse-path filtering is enabled.</p><p>I checked the Linux reverse-path filter settings:</p><pre>sysctl net.ipv4.conf.all.rp_filter<br>sysctl net.ipv4.conf.default.rp_filter<br>sysctl net.ipv4.conf.enp1s0.rp_filter<br>sysctl net.ipv4.conf.enp2s0.rp_filter<br><br>The result was:<br><br>net.ipv4.conf.all.rp_filter = 1<br>net.ipv4.conf.default.rp_filter = 1<br>net.ipv4.conf.enp1s0.rp_filter = 1<br>net.ipv4.conf.enp2s0.rp_filter = 1<br><br>The commonly used Linux rp_filter values are:<br><br>0 = disabled<br>1 = strict mode<br>2 = loose mode</pre><p>With strict reverse-path filtering enabled, Linux performs a reverse-path validation of incoming traffic. Linux considered the packet suspicious because Packet arrived from 203.0.113.132 on enp2s0 but ip route get 203.0.113.132 says dev enp1s0 and discarded it. That explained the martian messages.</p><p>Reverse-path filtering is primarily a source-address validation mechanism.</p><p>Imagine a packet arrives on one interface claiming to originate from an address that, according to the routing table, should normally be reachable through a completely different interface.</p><p>That can be an indication of IP spoofing.</p><p>Strict reverse-path filtering can therefore help protect systems against certain spoofed-source traffic.</p><p>In relatively simple symmetric networks, strict mode can work very well.</p><p>However, legitimate networks can also be asymmetric.</p><p>Examples include:</p><ul><li>multi-homed Linux servers</li><li>policy-based routing</li><li>firewalls with multiple paths</li><li>ECMP environments</li><li>complex virtual networks</li><li>redundant routers</li><li>multiple routing domains</li></ul><p>In those environments: incoming interface != return interface</p><p>To confirm the hypothesis, I temporarily disabled reverse-path filtering:</p><pre>sysctl -w net.ipv4.conf.all.rp_filter=0<br>sysctl -w net.ipv4.conf.enp2s0.rp_filter=0<br><br>The ping immediately started working.</pre><p>Changing rp_filter simply allowed Linux to accept that path.</p><p>Sometimes that is exactly what you want. Sometimes it is masking a routing problem. The network design needs to determine which one is true.</p><h4><strong>rp_filter=0 or rp_filter=2?</strong></h4><p>With loose mode, Linux does not require the best reverse route to use the exact interface on which the packet arrived.</p><p>Instead, the source generally needs to be reachable through some valid route.</p><pre>sysctl -w net.ipv4.conf.all.rp_filter=2<br>sysctl -w net.ipv4.conf.default.rp_filter=2<br>sysctl -w net.ipv4.conf.enp1s0.rp_filter=2<br>sysctl -w net.ipv4.conf.enp2s0.rp_filter=2</pre><p>In some multi-homed environments, loose mode can therefore be a better compromise than completely disabling reverse-path validation.</p><p>The correct setting depends on both:</p><ul><li>the intended routing architecture</li><li>the organization’s security requirements</li></ul><p>Changing the value is easy.</p><p>Understanding <strong>why the value matters in your network</strong> is the important part. Making traffic work is not always the same as fixing the underlying network architecture.</p><h4>Quick Reference</h4><pre># Interfaces<br>ip -br addr<br><br># Routing<br>ip route<br>ip rule<br><br># Reverse route toward problematic client<br>ip route get &lt;client-ip&gt;<br><br># Capture across every interface<br>tcpdump -ni any host &lt;client-ip&gt;<br><br># Kernel martian logs<br>journalctl -k | grep -i martian<br><br># Reverse-path filtering<br>sysctl net.ipv4.conf.all.rp_filter<br>sysctl net.ipv4.conf.default.rp_filter<br>sysctl net.ipv4.conf.&lt;interface&gt;.rp_filter</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e93c7d054650" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with GPU Infrastructure on Kubernetes: A Practical Guide Using NVIDIA GPU Operator]]></title>
            <link>https://medium.com/@murat.bilal/getting-started-with-gpu-infrastructure-on-kubernetes-a-practical-guide-using-nvidia-gpu-operator-ecf988baddf1?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecf988baddf1</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 12:34:56 GMT</pubDate>
            <atom:updated>2026-03-17T12:34:56.394Z</atom:updated>
            <content:encoded><![CDATA[<p>As AI workloads grow rapidly in 2026, managing GPUs efficiently has become a critical skill for infrastructure engineers. Running GPUs inside Kubernetes is no longer optional — it’s essential for scalable AI platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xbEyXkwyS7gedXSmAShlfg.png" /></figure><p>In this article, I’ll walk through a simple and practical approach to understanding how GPUs work in Kubernetes using the NVIDIA GPU Operator and some important commands.</p><h4>What is GPU Operator?</h4><p>The NVIDIA GPU Operator is a Kubernetes add-on that automates:</p><p>- GPU driver installation <br>- Container runtime configuration <br>- GPU device exposure to pods <br>- Monitoring via DCGM</p><p>Instead of manually configuring each node, the operator manages everything using Kubernetes-native resources.</p><h4>How GPUs are exposed to Kubernetes?</h4><p>Once the GPU Operator is installed, Kubernetes can see GPUs as resources:</p><pre>kubectl get nodes -o custom-columns=&quot;NAME:.metadata.name,GPU:.status.allocatable[&#39;nvidia\.com/gpu&#39;]&quot;<br>NAME   GPU<br>nodeA   8<br>nodeB   2</pre><p>This means one node has 8(because of time slicing), the other have 2 GPUs available for scheduling.</p><h4>Running your first GPU workload:</h4><p>A simple way to test GPUs is to run a pod using <em>nvidia-smi</em>:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: gpu-dedicated-test<br>spec:<br>  restartPolicy: Never<br>  nodeSelector:<br>    gpu-mode: dedicated<br>  containers:<br>    - name: cuda<br>      image: nvidia/cuda:12.4.1-base-ubuntu22.04<br>      command: [&quot;bash&quot;, &quot;-lc&quot;, &quot;nvidia-smi &amp;&amp; sleep 30&quot;]<br>      resources:<br>        limits:<br>          nvidia.com/gpu: 1</pre><p>I save it to gpu-dedicated-test.yaml file and apply then check logs.</p><p>If everything is working, you’ll see GPU details printed.</p><pre>kubectl apply -f gpu-dedicated-test.yaml<br>pod/gpu-dedicated-test created<br><br>kubectl logs gpu-dedicated-test<br><br>+-----------------------------------------------------------------------------------------+<br>| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0     |<br>+-----------------------------------------+------------------------+----------------------+<br>| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>|                                         |                        |               MIG M. |<br>|=========================================+========================+======================|<br>|   0  NVIDIA L40S                    On  |   00000000:61:00.0 Off |                    0 |<br>| N/A   30C    P8             36W /  350W |       0MiB /  46068MiB |      0%      Default |<br>|                                         |                        |                  N/A |<br>+-----------------------------------------+------------------------+----------------------+<br><br>+-----------------------------------------------------------------------------------------+<br>| Processes:                                                                              |<br>|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |<br>|        ID   ID                                                               Usage      |<br>|========================================================================================</pre><h4>Understanding GPU topology:</h4><p>One of the most overlooked topics in AI infrastructure is <strong>NUMA and topology awareness.</strong></p><p>You can check GPU topology with:</p><pre>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi topo -m<br>        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID<br>GPU0     X      NODE    0-63,128-191    0               N/A<br>GPU1    NODE     X      0-63,128-191    0               N/A<br><br>Legend:<br><br>  X    = Self<br>  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)<br>  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node<br>  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)<br>  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)<br>  PIX  = Connection traversing at most a single PCIe bridge<br>  NV#  = Connection traversing a bonded set of # NVLinks</pre><p>This tells you how GPUs are connected and how close they are to CPUs and NICs.</p><p>In high-performance AI workloads:</p><ul><li>GPUs should be close to CPUs</li><li>GPUs should be close to NICs</li><li>Memory access should be local</li></ul><p>If not, performance drops due to cross-socket communication.</p><h4>GPU and NIC alignment</h4><p>In real-world setups:</p><p>GPUs are attached to a specific NUMA node. NICs are also attached to NUMA nodes</p><p>For best performance:</p><p>GPU ↔ NIC ↔ CPU should be on the same NUMA node</p><p>This becomes critical for:</p><ul><li>distributed training</li><li>large dataset streaming</li><li>RDMA workloads</li></ul><p>For NUMA Mapping You can use below:</p><pre>for bdf in $(lspci -D | grep &#39;BCM57608&#39; | awk &#39;{print $1}&#39;); do   echo &quot;PCI:  $bdf&quot;;   echo &quot;DESC: $(lspci -s $bdf)&quot;;   echo &quot;IFACE: $(ls -1 /sys/bus/pci/devices/$bdf/net 2&gt;/dev/null)&quot;;   echo &quot;NUMA: $(cat /sys/bus/pci/devices/$bdf/numa_node)&quot;;   echo; done<br>PCI:  0000:41:00.0<br>DESC: 41:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet (rev 11)<br>IFACE: ens1np0<br>NUMA: 0<br><br>PCI:  0000:c2:00.0<br>DESC: c2:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet (rev 11)<br>IFACE: ens20np0<br>NUMA: 1</pre><p>Both GPUs are on NUMA 0. If you want best performance for AI training/inference the 400G NICs used for GPU traffic should ideally also be on NUMA 0.</p><p>Extra useful host checks for pci:</p><pre> lscpu | egrep &#39;NUMA|Socket|CPU\(s\)&#39;<br>CPU(s):                                  256<br>On-line CPU(s) list:                     0-255<br>Socket(s):                               2<br>CPU(s) scaling MHz:                      41%<br>NUMA node(s):                            2<br>NUMA node0 CPU(s):                       0-63,128-191<br>NUMA node1 CPU(s):                       64-127,192-255<br><br>lspci -nnk | egrep -A4 &#39;Ethernet|Network&#39;<br>25:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)<br>        Subsystem: Lenovo BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [17aa:4050]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>25:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)<br>        Subsystem: Lenovo BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [17aa:4050]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>25:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)<br>        Subsystem: Lenovo BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [17aa:4050]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>25:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1751] (rev 12)<br>        Subsystem: Lenovo BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [17aa:4050]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>26:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ac] (rev 01)<br>        Subsystem: Lenovo Device [17aa:781d]<br>--<br>41:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet [14e4:1760] (rev 11)<br>        Subsystem: Broadcom Inc. and subsidiaries Device [14e4:a120]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>42:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ac] (rev 01)<br>--<br>c2:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet [14e4:1760] (rev 11)<br>        Subsystem: Broadcom Inc. and subsidiaries Device [14e4:a120]<br>        Kernel driver in use: bnxt_en<br>        Kernel modules: bnxt_en<br>c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ac] (rev 01)<br><br><br>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi --query-gpu=index,pci.bus_id,name --format=csv<br>index, pci.bus_id, name<br>0, 00000000:01:00.0, NVIDIA L40S<br>1, 00000000:61:00.0, NVIDIA L40S</pre><h4><strong>Check GPU usage in real-time:</strong></h4><pre>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi dmon<br># gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk<br># Idx      W      C      C      %      %      %      %      %      %    MHz    MHz<br>    0     35     29      -      0      0      0      0      0      0    405    210<br>    1     35     31      -      0      0      0      0      0      0    405    210<br>    0     35     29      -      0      0      0      0      0      0    405    210<br>    1     35     31      -      0      0      0      0      0      0    405    210<br>    0     35     29      -      0      0      0      0      0      0    405    210<br>    1     35     31      -      0      0      0      0      0      0    405    210<br><br>## You can also use:<br><br>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi pmonon<br># gpu         pid   type     sm    mem    enc    dec    jpg    ofa    command<br># Idx           #    C/G      %      %      %      %      %      %    name<br>    0          -     -      -      -      -      -      -      -    -<br>    1          -     -      -      -      -      -      -      -    -<br>    0          -     -      -      -      -      -      -      -    -<br>    1          -     -      -      -      -      -      -      -    -<br>    0          -     -      -      -      -      -      -      -    -<br>    1          -     -      -      -      -      -      -      -    -<br>    0          -     -      -      -      -      -      -      -    -<br><br><br></pre><h4>Check device plugin:</h4><p>If GPUs are not visible → check here first</p><pre>kubectl logs -n gpu-operator ds/nvidia-device-plugin-daemonset</pre><h4>Check DCGM exporter (metrics):</h4><pre>kubectl logs -n gpu-operator ds/nvidia-dcgm-exporter</pre><h4>Check GPU Feature Discovery:</h4><pre>kubectl get nodes --show-labels | grep nvidia</pre><h4>Check MIG / time-slicing config:</h4><pre>kubectl get configmap -n gpu-operator<br>NAME                                              DATA   AGE<br>default-gpu-clients                               1      148d<br>default-mig-parted-config                         1      148d<br>gpu-operator-node-feature-discovery-master-conf   1      148d<br>gpu-operator-node-feature-discovery-worker-conf   1      148d<br>kube-root-ca.crt                                  1      148d<br>nvidia-container-toolkit-entrypoint               1      141d<br>nvidia-device-plugin-entrypoint                   1      148d<br>nvidia-mig-manager-entrypoint                     1      148d<br>time-slicing-config                               1      148d</pre><h4>Check GPU allocation per pod:</h4><pre>kubectl describe pod &lt;pod-name&gt;<br><br>Containers:<br>  cuda:<br>    Container ID:  containerd://12716d8b72a1c57081e818a87b51370295ee171844a195d00506643e830ac2ac<br>    Image:         nvidia/cuda:12.4.1-base-ubuntu22.04<br>    Image ID:      docker.io/nvidia/cuda@sha256:0f6bfcbf267e65123bcc2287e2153dedfc0f24772fb5ce84afe16ac4b2fada95<br>    Port:          &lt;none&gt;<br>    Host Port:     &lt;none&gt;<br>    Command:<br>      bash<br>      -lc<br>      nvidia-smi &amp;&amp; sleep 30<br>    State:          Terminated<br>      Reason:       Completed<br>      Exit Code:    0<br>      Started:      Tue, 17 Mar 2026 12:21:00 +0100<br>      Finished:     Tue, 17 Mar 2026 12:21:30 +0100<br>    Ready:          False<br>    Restart Count:  0<br>    Limits:<br>      nvidia.com/gpu:  1</pre><h4>Check driver container status:</h4><pre>kubectl logs -n gpu-operator ds/nvidia-driver-daemonset<br>Found 2 pods, using pod/nvidia-driver-daemonset-4wg5n<br>DRIVER_ARCH is x86_64<br><br>========== NVIDIA Software Installer ==========<br><br>Starting installation of NVIDIA driver version 580.82.07 for Linux kernel version 6.8.0-101-generic<br><br>Unloading NVIDIA driver kernel modules...<br>Unmounting NVIDIA driver rootfs...<br>Updating the package cache...<br>Resolving Linux kernel version...<br>Proceeding with Linux kernel version 6.8.0-101-generic<br>Installing Linux kernel headers...<br>Installing Linux kernel module files...<br>Generating Linux kernel version string...<br>Verifying archive integrity... OK<br>Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 580.82.07...............................................................................................................................................................................................</pre><h4>Verify GPU resources from Kubernetes perspective:</h4><pre>kubectl describe node &lt;node-name&gt; | grep -A5 nvidia</pre><h4>Tiny cheat sheet:</h4><pre># Operator health<br>kubectl get pods -n gpu-operator -o wide<br><br># GPUs advertised to K8s<br>kubectl get nodes -o custom-columns=&quot;NAME:.metadata.name,GPU:.status.allocatable[&#39;nvidia\.com/gpu&#39;]&quot;<br><br># Driver visibility<br>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi<br><br># GPU topology<br>kubectl -n gpu-operator exec -it ds/nvidia-driver-daemonset -- nvidia-smi topo -m<br><br># Device plugin logs<br>kubectl logs -n gpu-operator ds/nvidia-device-plugin-daemonset --tail=100<br><br># Time-slicing config<br>kubectl get configmap time-slicing-config -n gpu-operator -o yaml<br><br># Full operator config<br>kubectl get clusterpolicy -o yaml</pre><h4>Summary:</h4><ul><li>GPU Operator simplifies GPU management in Kubernetes</li><li>GPUs are exposed as schedulable resources</li><li>Testing GPU access is easy using a simple pod</li><li>NUMA and topology awareness are critical for performance</li><li>Proper GPU ↔ NIC alignment can significantly impact throughput</li></ul><p>Working with GPUs in Kubernetes is not just about making things run , it’s about making them run efficiently.</p><p>Understanding topology, NUMA, and resource scheduling is what separates basic setups from production-grade AI infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecf988baddf1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DJANGO WEB APP FOR IP TRACKING AND MANAGEMENT]]></title>
            <link>https://medium.com/@murat.bilal/django-web-app-for-ip-tracking-and-management-a1888b0770d3?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1888b0770d3</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Sat, 07 Feb 2026 08:40:22 GMT</pubDate>
            <atom:updated>2026-02-07T08:40:22.290Z</atom:updated>
            <content:encoded><![CDATA[<p>I have created a tiny web application for lab IP management and tracking. I am using Django 6 and postgresql 16 for this project. My purpose is tracking IP usage and recording all of these in postgresql database. Users can search used IPs , stale IP adressess which have been using since 30 days, also Find free IP using Linux arping tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cy9mjWoKRbIN-bAq9vXCPw.png" /></figure><p>Here is a basic Usage Guide for this tool and you can find github repo at the end of this article including the setup of the project.</p><p>For demonstration purposes I am using Django framework own web server.</p><p>Running server with below command</p><pre>poetry run python manage.py runserver 0.0.0.0:8000</pre><p>For admin tasks like creating subnet, user, group…. go to the URL</p><p><a href="http://127.0.0.1:8000/admin">http://127.0.0.1:8000/admin</a></p><p>Then Create your first subnet by clicking Subnets — -+Add</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3BdfNgFYN33CPpNo812_w.png" /><figcaption>Adding Subnet</figcaption></figure><p>You can give a name to your subnet, then define CIDR, exclude necessary IP addresses and save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtF21vMcGv3NW1gJgPSy4w.png" /><figcaption>Add subnet with necessary input.</figcaption></figure><p>For authentication and authorization, I create a group called ip-users, and select only IP address allocation permissons, so that users in this group cannot create subnet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDphgYvXzExr5UVC3jVE2A.png" /><figcaption>Creating a Group for Authentication and Authorization.</figcaption></figure><p>I added a user bob, deselect staff status, I do not want Bob to login to admin site, and put bob inside our ip-users group then save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGK1mMfqLwBBoKmmHNe-xg.png" /><figcaption>Creating a user.</figcaption></figure><p>Now when Bob goes to link <a href="http://127.0.0.1:8000/accounts/login">http://127.0.0.1:8000/accounts/login</a> , he must change his password at his first login.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kfdbptkBCgW3SEe2YhSlOg.png" /><figcaption>Change password page.</figcaption></figure><p>Once Bob changes his password, he can login and see the defined subnet, click Open for Ip address allocation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_GcxYRowAtrI-LZBkUa9RQ.png" /><figcaption>IP address allocation</figcaption></figure><p>If he can click on Find Free IP, system returns first free IP, and Bob can use it by just clicking claim(You can define hostname and description in here)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OrtG8tk1ZIira4tPJAKc0w.png" /><figcaption>Claim an IP.</figcaption></figure><p>Also you can give an IP which you already know it is free, without using Find free IP button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rmfg3HqUY5M7vrtMjBLQ_Q.png" /><figcaption>Claim an IP without using Find Free IP</figcaption></figure><p>You can also search for other users and their IP allocations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkq-Cyk3YMYUeDakGWNVYQ.png" /><figcaption>Search for a user IP allocation</figcaption></figure><p>You can also release IP if you do not use anymore by clicking release button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1DtU5W1pcNkp92aWdlW0jw.png" /><figcaption>Release an IP.</figcaption></figure><p>From the admin page you can see the IP release details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITlEbGfUAJRVMH19xNQ88Q.png" /></figure><p>This is the basic intro for how to use this tool, I hope you will use and enjoy it. For github repo you can find it below</p><p><a href="https://github.com/muratbill/django-ip-manager.git">GitHub - muratbill/django-ip-manager: Lab IP tracker build with Django6 and Postgres 16</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1888b0770d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPU Operator on Kubernetes]]></title>
            <link>https://medium.com/@murat.bilal/gpu-operator-on-kubernetes-608a84f1d177?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/608a84f1d177</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Fri, 07 Nov 2025 09:48:43 GMT</pubDate>
            <atom:updated>2025-11-07T09:48:43.619Z</atom:updated>
            <content:encoded><![CDATA[<p>The<strong> </strong>GPU Operator on Kubernetes is an essential tool for projects that involve high-performance computing (HPC), Artificial Intelligence (AI), or Machine Learning (ML).</p><p>Think of it as a “robotic system administrator” that automatically handles all the complex setup required to use Graphics Processing Units (GPUs) in a Kubernetes cluster.</p><p>Running GPU-accelerated workloads on Kubernetes is complicated. You need to manually install and manage multiple software components on every server with a GPU, including the GPU drivers, container runtimes, and a Kubernetes “device plugin” so the cluster knows the GPUs exist. This is tedious, error-prone, and slow, especially when you need to scale.</p><p>The GPU Operator automates the entire process. It uses the Kubernetes Operator pattern to monitor your cluster nodes and automatically install, configure, and manage <strong>all</strong> the necessary NVIDIA software components as containers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cQHSr5O2OegbXWhb4CA5MA.jpeg" /></figure><p>In short, the GPU Operator <strong>abstracts away the complexity</strong> of GPU hardware management, making it as easy to manage a GPU server as a standard CPU server in Kubernetes, which directly translates to <strong>faster time-to-market</strong> and <strong>lower operational risk</strong> for your AI/ML initiatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EXVLrzE6xq93-yLu_LkB5g.jpeg" /></figure><p>You can get an overview of the operator and its benefits from this <a href="https://www.youtube.com/watch?v=KER0dbfmAqQ">NVIDIA GPU Operator Overview</a> video. This video explains the components of a Kubernetes cluster with GPUs and how the GPU Operator simplifies the management challenges.</p><p>This article uses Ubuntu VERSION=”24.04.3 LTS (Noble Numbat)” for installing GPU Operator:</p><p>First you need to get helm if it is not already installed. Here are the steps:</p><pre>curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash<br><br>## Check helm version after installed<br><br>helm version<br>version.BuildInfo{Version:&quot;v3.19.0&quot;, GitCommit:&quot;3d8990f0836691f0229297773f3524598f46bda6&quot;, GitTreeState:&quot;clean&quot;, GoVersion:&quot;go1.24.7&quot;}<br><br>helm repo add nvidia https://nvidia.github.io/gpu-operator<br>helm repo update<br><br>## Install GPU Operator<br>## set the driver.enabled true and toolkit.enabled to true to give<br>## full control for installling nvidia drivers to GPU Operator<br><br>helm install --wait gpu-operator nvidia/gpu-operator -n gpu-operator --create-namespace --version v25.3.4 --set driver.enabled=true --set toolkit.enabled=true<br><br>## Check pods(Output may vary according to your dep)<br><br>kubectl get pods -n gpu-operator<br>NAME                                                          READY   STATUS      RESTARTS      AGE<br>curl                                                          0/1     Completed   0             12d<br>gpu-feature-discovery-tppgf                                   2/2     Running     0             10d<br>gpu-operator-797b5df7dc-vtlnz                                 1/1     Running     2 (10d ago)   18d<br>gpu-operator-node-feature-discovery-gc-86f6495b55-jbcqj       1/1     Running     2 (10d ago)   18d<br>gpu-operator-node-feature-discovery-master-694467d5db-cqf6l   1/1     Running     2 (10d ago)   18d<br>gpu-operator-node-feature-discovery-worker-fzzvq              1/1     Running     2 (10d ago)   18d<br>nvidia-container-toolkit-daemonset-qh4xk                      1/1     Running     0             10d<br>nvidia-cuda-validator-q2847                                   0/1     Completed   0             10d<br>nvidia-dcgm-exporter-4wwnc                                    1/1     Running     0             10d<br>nvidia-device-plugin-daemonset-cfrnh                          2/2     Running     0             10d<br>nvidia-driver-daemonset-wtw2g                                 1/1     Running     0             10d<br>nvidia-operator-validator-qfdl6                               1/1     Running     0             10d<br><br>kubectl get deploy -n gpu-operator<br>NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE<br>gpu-operator                                 1/1     1            1           18d<br>gpu-operator-node-feature-discovery-gc       1/1     1            1           18d<br>gpu-operator-node-feature-discovery-master   1/1     1            1           18d</pre><p>The <strong>GPU Operator</strong> doesn’t just install drivers; it also manages and exposes advanced GPU features to the Kubernetes scheduling system.The ability to use both Multi-Instance GPU (MIG) and Time-Slicing within a Kubernetes cluster is directly enabled and managed by the GPU Operator.</p><p>The GPU Operator deploys and configures the NVIDIA Container Toolkit and the Kubernetes Device Plugin.</p><p>To enable Time-Slicing, you configure the Device Plugin (which the Operator manages) to report fractional GPU resources. For example, instead of reporting “1 GPU,” the plugin is configured to report “10 virtual devices,” allowing up to 10 jobs to share the GPU via time-slicing.</p><p>MIG is more complex because it involves physically partitioning the hardware.</p><ul><li>The GPU Operator includes a <strong>MIG Manager</strong> component. When deployed on a compatible GPU (like the A100 or H100), the Operator automatically detects the card and can be configured to:</li><li><strong>Dynamically create</strong> specific MIG profiles (e.g., split a GPU into three 2g.10gb partitions).</li><li><strong>Expose these specific hardware-isolated partitions</strong> as distinct, schedulable resources within Kubernetes (e.g., nvidia.com/mig-2g.10gb).</li></ul><p><strong>Time-Slicing</strong> and <strong>Multi-Instance GPU (MIG):</strong></p><p>Time-slicing lets multiple pods share one physical GPU (oversubscription). You enable it by providing a config to the device plugin via the Operator; pods keep using <a href="http://nvidia.com/gpu">nvidia.com/gpu</a>.</p><p>Time-Slicing is a <strong>software-based</strong> method that allows multiple containers (workloads) to take <strong>turns</strong> using the full GPU. Think of it like a single-lane road where all the cars (workloads) must take turns. The GPU driver rapidly switches between the running jobs, giving each one a short “slice” of processing time.</p><p>MIG is a hardware-based feature that allows you to partition a single physical GPU into multiple, fully isolated “mini-GPUs.” Think of it like a multi-lane highway. The GPU is physically split, giving each workload its own dedicated lane with guaranteed resources (compute cores, memory, and cache). Each partition is completely isolated, ensuring that one workload’s performance or crash will not affect any other workload on the GPU. This provides predictable performance (Quality of Service).</p><p>For your project, the choice depends on your priorities: use <strong>MIG</strong> if you need <strong>guaranteed, consistent performance</strong> and isolation for critical workloads; use <strong>Time-Slicing</strong> if you prioritize <strong>maximum utilization</strong> for non-critical or development work on any available GPU.</p><p>Here is a small demo for time slicing:</p><pre>cat time-slicing-config.yaml<br><br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: time-slicing-config<br>  namespace: gpu-operator<br>data:<br>  any: |<br>    version: v1<br>    flags:<br>      migStrategy: none<br>    sharing:<br>      timeSlicing:<br>        renameByDefault: false<br>        failRequestsGreaterThanOne: false<br>        resources:<br>          - name: nvidia.com/gpu<br>            replicas: 4</pre><p>Apply yaml file:</p><pre>kubectl apply- f time-slicing-config.yaml<br>kubectl label node &quot;$NODE&quot; nvidia.com/device-plugin.config=any --overwrite<br>kubectl -n gpu-operator rollout restart daemonset/nvidia-device-plugin-daemonset<br>kubectl -n gpu-operator rollout status daemonset/nvidia-device-plugin-daemonset<br><br>## Check logs for time slicing:<br>kubectl -n gpu-operator logs ds/nvidia-device-plugin-daemonset | grep -i timeslicing -A 10<br><br>Defaulted container &quot;nvidia-device-plugin&quot; out of: nvidia-device-plugin, config-manager, toolkit-validation (init), config-manager-init (init)<br>    &quot;timeSlicing&quot;: {<br>      &quot;resources&quot;: [<br>        {<br>          &quot;name&quot;: &quot;nvidia.com/gpu&quot;,<br>          &quot;devices&quot;: &quot;all&quot;,<br>          &quot;replicas&quot;: 4<br>        }<br>      ]<br>    }<br>  },<br>  &quot;imex&quot;: {}</pre><p>Create namespace for gpu1 and define quotas:</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: gpu1<br>---<br># Limit namespace to 4 GPU for the demo<br>apiVersion: v1<br>kind: ResourceQuota<br>metadata:<br>  name: gpu-quota<br>  namespace: gpu1<br>spec:<br>  hard:<br>    requests.nvidia.com/gpu: &quot;4&quot;<br>    limits.nvidia.com/gpu: &quot;4&quot;<br></pre><p>Create demo pod under namespace gpu1:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: timeslice-demo1<br>  namespace: gpu1<br>spec:<br>  replicas: 4<br>  selector:<br>    matchLabels:<br>      app: timeslice-demo1<br>  template:<br>    metadata:<br>      labels:<br>        app: timeslice-demo1<br>    spec:<br>      runtimeClassName: nvidia<br>      containers:<br>        - name: cuda<br>          image: nvidia/cuda:12.6.0-base-ubuntu22.04<br>          command: [&quot;/bin/bash&quot;,&quot;-lc&quot;]<br>          args:<br>            - |<br>              echo &quot;Pod: $(hostname) starting…&quot;<br>              nvidia-smi -L || true<br>              while true; do<br>                nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader || true<br>                sleep 5<br>              done<br>          resources:<br>            requests:<br>              nvidia.com/gpu: 1<br>            limits:<br>              nvidia.com/gpu: 1</pre><p>Verification for demo pods and GPU allocations:</p><pre>kubectl get pods -n gpu1<br>NAME                               READY   STATUS    RESTARTS   AGE<br>timeslice-demo1-7764b4d67d-76kql   1/1     Running   0          127m<br>timeslice-demo1-7764b4d67d-dppqx   1/1     Running   0          127m<br>timeslice-demo1-7764b4d67d-dvjjq   1/1     Running   0          127m<br>timeslice-demo1-7764b4d67d-ggvmv   1/1     Running   0          127m<br><br>kubectl get deploy -n gpu1<br>NAME              READY   UP-TO-DATE   AVAILABLE   AGE<br>timeslice-demo1   4/4     4            4           128m<br><br>kubectl -n gpu1 exec -it  timeslice-demo1-7764b4d67d-dppqx -- nvidia-smi --query-gpu=uuid,index,name<br>uuid, index, name<br>GPU-ef551c88-0853-3dae-223b-5238fa1e2a17, 0, NVIDIA L40S<br><br>kubectl -n gpu1 exec -it  timeslice-demo1-7764b4d67d-dvjjq  -- nvidia-smi --query-gpu=uuid,index,name<br>uuid, index, name<br>GPU-526864ff-2588-8519-324b-f93e5ccda7d7, 0, NVIDIA L40S<br><br>kubectl -n gpu1 exec -it  timeslice-demo1-7764b4d67d-ggvmv -- nvidia-smi --query-gpu=uuid,index,name<br>uuid, index, name<br>GPU-ef551c88-0853-3dae-223b-5238fa1e2a17, 0, NVIDIA L40S</pre><p>In summary, the <strong>GPU Operator is the bridge</strong> that takes these advanced, low-level NVIDIA features (<strong>Time-Slicing</strong> and <strong>MIG</strong>) and transforms them into standard, easy-to-use <strong>Kubernetes resources</strong> that your project workloads can request. Without the Operator, setting up and managing these capabilities manually would be a complex and time-consuming administrative task.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=608a84f1d177" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Brief Introduction to Java Record Classes]]></title>
            <link>https://medium.com/@murat.bilal/brief-introduction-to-java-record-classes-7603d803caa2?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/7603d803caa2</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 19:05:17 GMT</pubDate>
            <atom:updated>2024-12-30T19:05:17.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*wzv5WrLK_LN0JTfHI3703Q.png" /></figure><p>Beginning with JDK 16, Java supports a special-purpose class called a <em>record</em>. A record is designed to provide an efficient, easy-to-use way to hold a group of values. For example, you might use a record to hold a set of coordinates; bank account numbers and balances; the length, width, and height of a shipping container; and so on. Because it holds a group of values, a record is commonly referred to as an <em>aggregate</em> type. However, the record is more than simply a means of grouping data, because records also have some of the capabilities of a class. In addition, a record has unique features that simplify its declaration and streamline access to its values. As a result, records make it much easier to work with groups of data. Records are supported by the new context-sensitive keyword <strong>record.</strong></p><p>One of the central motivations for records is the reduction of the effort required to create a class whose primary purpose is to organize two or more values into a single unit. Although it has always been possible to use <strong>class</strong> for this purpose, doing so can entail writing a number of lines of code for constructors, getter methods, and possibly (depending on use) overriding one or more of the methods inherited from <strong>Object</strong>. As you will see, by creating a data aggregate by using <strong>record</strong>, these elements are handled automatically for you, greatly simplifying your code. Another reason for the addition of records is to enable a program to clearly indicate that the intended purpose of a class is to hold a grouping of data, rather than act as a full-featured class. Because of these advantages, records are a much welcomed addition to Java.</p><p>Here is an example of a simple <strong>record</strong> declaration:</p><pre>record Point(int x, int y) {}</pre><p>The key characteristics of this example:</p><p><strong>Automatic Generation:</strong> The compiler automatically generates:</p><ul><li><strong>Constructor:</strong> A canonical constructor that initializes all components.</li><li><strong>Accessors:</strong> Getter methods for each component (e.g., x(), y()).</li><li><strong>toString():</strong> A human-readable string representation of the record.</li><li><strong>equals() and </strong><strong>hashCode():</strong> Methods for comparing records based on their components.</li></ul><p><strong>Immutability:</strong> Record components are implicitly final, ensuring that their values cannot be changed after the record is created.</p><p><strong>Readability:</strong> Records promote a more concise and readable way to represent data structures.</p><p>In general, each record component will have a corresponding private final field and a read-only public getter method automatically created by the compiler.</p><pre>public class MyPoint {<br>    record Points(int x, int y){}<br><br>    public static void main(String[] args) {<br>        Points poi = new Points(9 ,8);<br>        System.out.println(poi.x);<br>        System.out.println(poi.y);<br>    }<br>}</pre><p>Output of this code:</p><pre>9<br>8</pre><p>A <strong>record</strong> is instantiated by use of <strong>new</strong>, just the way you create an instance of a <strong>class </strong>Points poi = new Points(9 ,8);</p><p><strong>Create Record Constructors</strong></p><p>Although you will often find that the automatically supplied canonical constructor is precisely what you want, you can also declare one or more of your own constructors. You can define your own constructors within a record. 1 These constructors must call the canonical constructor using this(...) within their body.</p><p>Custom constructors are useful for:</p><ul><li><strong>Validation:</strong> Performing input validation before creating the record object.</li><li><strong>Initialization:</strong> Initializing derived values or performing calculations based on the input parameters.</li><li><strong>Complex Logic:</strong> Handling more intricate creation scenarios that cannot be easily expressed within the record’s component list.</li></ul><p>A canonical constructor example:</p><pre>public class Lectures {<br>    record MyLectures(String lecture, int lectureCode) {<br>        public MyLectures(String lecture, int lectureCode) {<br>            this.lecture = lecture.toUpperCase();<br>            this.lectureCode = lectureCode;<br>        }<br>    }<br><br>        public static void main(String[] args) {<br>            MyLectures myLectures = new MyLectures(&quot;Physics&quot;, 101);<br>            System.out.print(STR.&quot;My lecture is: \{myLectures.lecture()} My lecture code is : \{myLectures.lectureCode()}&quot;);<br><br><br><br>        }<br><br>}</pre><p>Output of this code:</p><pre>My lecture is: PHYSICS My lecture code is : 101</pre><p>A compact canonical example:</p><pre>public class Lectures {<br>    record MyLectures(String lecture, int lectureCode) {<br>        public MyLectures {<br>            lecture = lecture.toUpperCase();<br><br>        }<br>    }<br><br>        public static void main(String[] args) {<br>            MyLectures myLectures = new MyLectures(&quot;Physics&quot;, 101);<br>            System.out.print(STR.&quot;My lecture is: \{myLectures.lecture()} My lecture code is : \{myLectures.lectureCode()}&quot;);<br><br><br><br>        }<br><br>}</pre><p>Output is same. Here the result of toUpperCase() is called on the lecture parameter(which is implicitly declared by the compact constructor) and the result is assigned back to the lecture parameter.At the end of the compact constructor, the value of lecture is automatically assigned to its corresponding field. The value of the implicit lectureCode parameter is also assigned to its corresponding field at the end of the constructor. Because the parameters are implicitly assigned to their corresponding fields when the constructor ends, there is no need to initialize the fields explicitly.</p><p>Non Canonical Constructor: The following program declares a non-canonical constructor for MyLectures that initializes the name to a known value, but gives the lectureCode field the special value id(which is –1) to indicate an ID value is not available when the record is created:</p><pre> record MyLectures(String lecture, int lectureCode) {<br>        static int id = -1;<br>        public MyLectures {<br>            lecture = lecture.toUpperCase();<br><br>        }<br>        public MyLectures(String lecture){<br>            this(lecture,id);<br>        }<br>    }<br><br>        public static void main(String[] args) {<br>            MyLectures[] lecturesList = new MyLectures[3];<br>            lecturesList[0] = new MyLectures(&quot;Physics&quot;, 101);<br>            lecturesList[1] = new MyLectures(&quot;Chemistry&quot;, 103);<br>            lecturesList[2] = new MyLectures(&quot;Mathematics&quot;);<br>            for(MyLectures m: lecturesList){<br>                System.out.println(&quot;My lectures are: &quot; + m.lecture());<br>                if(m.lectureCode() == MyLectures.id) System.out.println(&quot;Pending&quot;);<br>                else System.out.println(m.lectureCode());<br>            }<br><br><br><br>        }<br><br>}</pre><p>Output is:</p><pre>My lectures are: PHYSICS<br>101<br>My lectures are: CHEMISTRY<br>103<br>My lectures are: MATHEMATICS<br>Pending</pre><p>Instance fields are not allowed in a <strong>record</strong> declaration, but a <strong>static</strong> field is legal. That is why id field is static.</p><p><strong>Create Record Getter Methods</strong></p><p>Although it is seldom necessary, it is possible to create your own implementation of a getter method. When you declare the getter, the implicit version is no longer supplied. One possible reason you might want to declare your own getter is to throw an exception if some condition is not met. For example, if a record holds a filename and a URL, the getter for the filename might throw a FileNotFoundException if the file is not present at the URL. If you do declare a getter implementation, there are a number of rules that apply. A getter must have the same return type and name as the component that it obtains. No throws clause is allowed in a getter declaration. Finally, a getter must be non-generic and non-static.</p><p>A better alternative to overriding a getter in cases in which you want to obtain a modified value of a component is to declare a separate method with its own name</p><p>For example If I want to get only capital cities from this format countryInfos[0] = new CountryInfo(“Rome, Italy”);</p><pre>public class Countries {<br>    record CountryInfo(String coun){<br>        String capital(){<br>            return coun.substring(0, coun.indexOf(&#39;,&#39;));<br>        }<br>    }<br>    public static void main(String[] args) {<br>        CountryInfo[] countryInfos = new CountryInfo[3];<br>        countryInfos[0] = new CountryInfo(&quot;Rome, Italy&quot;);<br>        countryInfos[1] = new CountryInfo(&quot;London, U.K&quot;);<br>        countryInfos[2] = new CountryInfo(&quot;Paris, France&quot;);<br>        for(CountryInfo e: countryInfos)<br>            System.out.println(e.coun());<br>        System.out.println(&quot;Capital cities are : &quot;);<br>        for(CountryInfo m: countryInfos)<br>            System.out.println(m.capital());<br><br><br>    }<br><br><br><br>}</pre><p>Result:</p><pre>Rome, Italy<br>London, U.K<br>Paris, France<br>Capital cities are : <br>Rome<br>London<br>Paris</pre><p><strong>Summary:</strong></p><p>Record classes in Java offer several key advantages:</p><p>Records significantly reduce the amount of code needed to define simple data classes. You no longer need to manually write constructors, getters, equals(), hashCode(), and toString() methods. The compiler automatically generates these for you.</p><p>Records are inherently immutable. Once a record is created, its fields cannot be modified. This ensures data integrity and makes it easier to reason about the state of your objects.</p><p>Immutability simplifies concurrent programming by eliminating the need to worry about thread-safety issues that can arise with mutable objects.</p><p>Their conciseness, immutability, and automatic generation of essential methods make them a valuable addition to the Java language, enhancing code readability, maintainability, and overall productivity.</p><p><strong>Sources:</strong></p><p>1- Java: The Complete Reference, Thirteenth Edition, 13th Edition<br>Herbert Schildt, Dr. Danny Coward.</p><p>2- Google Gemini.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7603d803caa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CONNECTING TO POSTGRES WITH JAVA USING JDBI]]></title>
            <link>https://medium.com/@murat.bilal/connecting-to-postgres-with-java-using-jdbi-a5a1e1095ad8?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5a1e1095ad8</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 11:46:30 GMT</pubDate>
            <atom:updated>2024-07-18T11:46:30.795Z</atom:updated>
            <content:encoded><![CDATA[<p>JDBI is a high-level API built on top of JDBC. It simplifies database interactions by reducing boilerplate code and providing a more user-friendly and declarative approach to database operations. JDBI manages resources automatically, reducing the risk of resource leaks. It abstracts away the explicit handling of connections, statements, and result sets, making the code cleaner and easier to maintain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0yjVywB97aT71GMtaVUluA.png" /></figure><p>All Jdbi modules are available through Maven central repository. Before starting our Maven project we should add postgresql and jdbi dependency to our pom.xml file.My project is using coretto-22</p><pre>&lt;dependency&gt;<br>            &lt;groupId&gt;org.postgresql&lt;/groupId&gt;<br>            &lt;artifactId&gt;postgresql&lt;/artifactId&gt;<br>            &lt;version&gt;42.7.3&lt;/version&gt;<br>        &lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>            &lt;groupId&gt;org.jdbi&lt;/groupId&gt;<br>            &lt;artifactId&gt;jdbi3-core&lt;/artifactId&gt;<br>            &lt;version&gt;3.45.2&lt;/version&gt;<br>        &lt;/dependency&gt;</pre><p>I am using postgresql in my docker VirtualBox VM so I created a JDBIOrnek.java class only for postgresql connection setup.</p><pre>import java.sql.Connection;<br>import java.sql.DriverManager;<br>import java.sql.SQLException;<br><br>public class JDBIOrnek {<br>    private static JDBCOrnek instance = null;<br>    private  Connection connection;<br>    private final String url = &quot;jdbc:postgresql://127.0.0.1:5432/personel&quot;;<br>    private final String username = &quot;postgres&quot;;<br>    private final String password = &quot;mypassword&quot;;<br>    private JDBIOrnek() {<br>        try {<br>            Connection connection = DriverManager.getConnection(url, username, password);<br>            if (connection != null) {<br>                System.out.println(&quot;Connected to Postgresql!&quot;);<br>                jdbi = Jdbi.create(connection);<br>            }<br>        } catch (SQLException e) {<br>            System.out.println(STR.&quot;Connection failed: \{e.getMessage()}&quot;);<br>        }<br>    }<br><br>    public static JDBIOrnek getInstance() {<br>        if (instance == null) {<br>            instance = new JDBIOrnek();<br>        }<br>        return instance;<br>    }<br><br>    public Jdbi getJdbi() {<br>        return jdbi;<br>    }<br>}</pre><p>This example demonstrates the singleton pattern to ensure that only one instance of the database connection is created and used throughout the application.</p><p>Now let’s create an another class for simple Database CRUD operations. I called this class DBOperations.java and I created a database called personel</p><pre>import org.jdbi.v3.core.Jdbi;<br>import org.jdbi.v3.core.statement.UnableToExecuteStatementException;<br>import java.util.List;<br>import java.util.Map;<br>public class DBOperations {<br><br>    private final Jdbi jdbi;<br><br>    public DBOperations(Jdbi jdbi) {<br>        this.jdbi = jdbi;<br>    }<br>    // Method to insert data into the table<br>    public void insertData(String adsoyad, String gorev, Integer yas, String bolum, int sicilno) {<br><br><br><br>            jdbi.useHandle(handle -&gt; {<br>                handle.execute(&quot;INSERT INTO ad_soyad (adsoyad, gorev, yas, bolum, sicilno) VALUES (?, ?, ?, ?, ?)&quot;,<br>                        adsoyad, gorev, yas, bolum, sicilno);<br>            });<br>            System.out.println(&quot;Data inserted.&quot;);<br>        }<br><br><br>    // Method to update data in the table<br>    public void updateData(String newad, String newgorev, Integer newyas, String newbolum, int sicilno) {<br><br><br><br>            StringBuilder updateQuery = new StringBuilder(&quot;UPDATE ad_soyad SET &quot;);<br>            boolean first = true;<br><br>            if (newad != null) {<br>                updateQuery.append(&quot;adsoyad = :newad&quot;);<br>                first = false;<br>            }<br>            if (newgorev != null) {<br>                if (!first) updateQuery.append(&quot;, &quot;);<br>                updateQuery.append(&quot;gorev = :newgorev&quot;);<br>                first = false;<br>            }<br>            if (newyas != null) {<br>                if (!first) updateQuery.append(&quot;, &quot;);<br>                updateQuery.append(&quot;yas = :newyas&quot;);<br>                first = false;<br>            }<br>            if (newbolum != null) {<br>                if (!first) updateQuery.append(&quot;, &quot;);<br>                updateQuery.append(&quot;bolum = :newbolum&quot;);<br>            }<br>            updateQuery.append(&quot; WHERE sicilno = :sicilno&quot;);<br>            try {<br><br>            int affected_rows = jdbi.withHandle(handle -&gt;<br>            handle.createUpdate(updateQuery.toString())<br>                    .bind(&quot;newad&quot;, newad)<br>                    .bind(&quot;newgorev&quot;, newgorev)<br>                    .bind(&quot;newyas&quot;, newyas)<br>                    .bind(&quot;newbolum&quot;, newbolum)<br>                    .bind(&quot;sicilno&quot;, sicilno)<br>                    .execute()<br>            );<br>                if (affected_rows == 0) {<br>                    System.err.println(STR.&quot;Herhangi bir update islemi olmadi \{sicilno}&quot;);<br>                } else {<br>                    System.out.println(&quot;Update basarili..&quot;);<br>                }<br>            } catch (UnableToExecuteStatementException e) {<br>                System.err.println(STR.&quot;Error updating data: \{e.getMessage()}&quot;);<br>            }<br><br>             }<br>    // Method to delete data from the table<br>    public void deleteData(int sicilno) {<br>        try {<br>            int affected_rows = jdbi.withHandle(handle -&gt;<br>                handle.createUpdate(&quot;DELETE FROM ad_soyad WHERE sicilno = :sicilno&quot;)<br>                        .bind(&quot;sicilno&quot;,sicilno)<br>                        .execute()<br>            );<br><br>            if (affected_rows == 0) {<br>                System.err.println(STR.&quot;Veri bulunamadi: \{sicilno}&quot;);<br>            } else {<br>                System.out.println(STR.&quot;Silinen sicilno: \{sicilno}&quot;);<br>            }<br>        } catch (UnableToExecuteStatementException e) {<br>            System.out.println(&quot;Veri silinemedi &quot;);<br>        }<br>    }<br><br>    public void createTable(){<br>        jdbi.useHandle(handle -&gt; handle.execute(&quot;CREATE TABLE IF NOT EXISTS ad_soyad (&quot; +<br>                &quot;adsoyad VARCHAR(50), &quot; +<br>                &quot;gorev VARCHAR(50), &quot; +<br>                &quot;yas INT, &quot; +<br>                &quot;bolum VARCHAR(50), &quot; +<br>                &quot;sicilno int PRIMARY KEY)&quot;));<br>        System.out.println(&quot;Table created or already exists.&quot;);<br>    }<br>    public List&lt;Map&lt;String, Object&gt;&gt; fetchDataBySicilNo(int sicilno) {<br>        return jdbi.withHandle(handle -&gt;<br>                handle.createQuery(&quot;SELECT * FROM ad_soyad  WHERE sicilno = :sicilno&quot;)<br>                        .bind(&quot;sicilno&quot;, sicilno)<br>                        .mapToMap()<br>                        .list());<br><br>    }<br>}</pre><p>Main class for testing CRUD operations;</p><p>Let’s first create our table</p><pre>import org.jdbi.v3.core.Jdbi;<br><br>import java.util.List;<br>import java.util.Map;<br><br>public class Main {<br>    public static void main(String[] args) {<br>        JDBIOrnek instance = JDBIOrnek.getInstance();<br>        Jdbi jdbi = instance.getJdbi();<br>        DBOperations dbops = new DBOperations(jdbi);<br>        if (jdbi == null) {<br>            System.err.println(&quot;Failed to establish jdbi. Exiting.&quot;);<br>            return;<br>        }<br>        dbops.createTable();<br> }<br><br>}</pre><p>Output of this Main class execution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*KEaPCXMDCqdEIJiN0n8Xmg.png" /></figure><p>Now let’s insert some data and expand our Main class</p><pre>import org.jdbi.v3.core.Jdbi;<br><br>import java.util.List;<br>import java.util.Map;<br><br>public class Main {<br>    public static void main(String[] args) {<br>        JDBIOrnek instance = JDBIOrnek.getInstance();<br>        Jdbi jdbi = instance.getJdbi();<br>        DBOperations dbops = new DBOperations(jdbi);<br>        if (jdbi == null) {<br>            System.err.println(&quot;Failed to establish jdbi. Exiting.&quot;);<br>            return;<br>        }<br>        dbops.createTable();<br>        dbops.insertData(&quot;John Wayne&quot;,&quot;Engineer&quot;,43,&quot;Engineering&quot;,1234);<br>        dbops.insertData(&quot;Emma Watson&quot;,&quot;Accountant&quot;,43,&quot;Finance&quot;,1235);<br>        dbops.insertData(&quot;Mehmet Can&quot;,&quot;Project manager&quot;,45,&quot;R&amp;D&quot;,1236);<br>        dbops.insertData(&quot;Tim Burton&quot;,&quot;Security&quot;,24,&quot;Security&quot;,1237);<br> }<br><br>}</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*lPCpxro6acE9mCSt6GVb5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*GFEygMT9yFdrrg6T9hAYkw.png" /></figure><p>Now expand main class and fetch data according to sicil_no which is PK</p><pre>import org.jdbi.v3.core.Jdbi;<br><br>import java.util.List;<br>import java.util.Map;<br><br>public class Main {<br>    public static void main(String[] args) {<br>        JDBIOrnek instance = JDBIOrnek.getInstance();<br>        Jdbi jdbi = instance.getJdbi();<br>        DBOperations dbops = new DBOperations(jdbi);<br>        if (jdbi == null) {<br>            System.err.println(&quot;Failed to establish jdbi. Exiting.&quot;);<br>            return;<br>        }<br>     //   dbops.createTable();<br>     //   dbops.insertData(&quot;John Wayne&quot;,&quot;Engineer&quot;,43,&quot;Engineering&quot;,1234);<br>     //   dbops.insertData(&quot;Emma Watson&quot;,&quot;Accountant&quot;,43,&quot;Finance&quot;,1235);<br>     //   dbops.insertData(&quot;Mehmet Can&quot;,&quot;Project manager&quot;,45,&quot;R&amp;D&quot;,1236);<br>     //   dbops.insertData(&quot;Tim Burton&quot;,&quot;Security&quot;,24,&quot;Security&quot;,1237);<br>        List&lt;Map&lt;String, Object&gt;&gt; dataBySicilNo = dbops.fetchDataBySicilNo(1237);<br>        for (Map&lt;String, Object&gt; row : dataBySicilNo) {<br>            System.out.println(STR.&quot;Isim: \{row.get(&quot;adsoyad&quot;)}&quot;);<br>            System.out.println(STR.&quot;Gorev: \{row.get(&quot;gorev&quot;)}&quot;);<br>            System.out.println(STR.&quot;yas: \{row.get(&quot;yas&quot;)}&quot;);<br>            System.out.println(STR.&quot;Bolum: \{row.get(&quot;bolum&quot;)}&quot;);<br>            System.out.println(STR.&quot;Sicil No: \{row.get(&quot;sicilno&quot;)}&quot;);<br>        } <br>}<br><br>}</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*-IaBYOVlCqwx4Xmn4NTIKg.png" /></figure><p>Now updating data which has sicil_no=1237</p><pre>import org.jdbi.v3.core.Jdbi;<br><br>import java.util.List;<br>import java.util.Map;<br><br>public class Main {<br>    public static void main(String[] args) {<br>        JDBIOrnek instance = JDBIOrnek.getInstance();<br>        Jdbi jdbi = instance.getJdbi();<br>        DBOperations dbops = new DBOperations(jdbi);<br>        if (jdbi == null) {<br>            System.err.println(&quot;Failed to establish jdbi. Exiting.&quot;);<br>            return;<br>        }<br>        dbops.updateData(&quot;Tim Cooking&quot;,null,null,null,1237);<br>        List&lt;Map&lt;String, Object&gt;&gt; dataBySicilNo = dbops.fetchDataBySicilNo(1237);<br>        for (Map&lt;String, Object&gt; row : dataBySicilNo) {<br>            System.out.println(STR.&quot;Isim: \{row.get(&quot;adsoyad&quot;)}&quot;);<br>            System.out.println(STR.&quot;Gorev: \{row.get(&quot;gorev&quot;)}&quot;);<br>            System.out.println(STR.&quot;yas: \{row.get(&quot;yas&quot;)}&quot;);<br>            System.out.println(STR.&quot;Bolum: \{row.get(&quot;bolum&quot;)}&quot;);<br>            System.out.println(STR.&quot;Sicil No: \{row.get(&quot;sicilno&quot;)}&quot;);<br>        } <br>}<br><br>}</pre><p>Output is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*9qL11NyVXsDH7PBG6znMTQ.png" /></figure><p>Now finally delete data which has sicil_no=1237</p><pre>import org.jdbi.v3.core.Jdbi;<br><br>import java.util.List;<br>import java.util.Map;<br><br>public class Main {<br>    public static void main(String[] args) {<br>        JDBIOrnek instance = JDBIOrnek.getInstance();<br>        Jdbi jdbi = instance.getJdbi();<br>        DBOperations dbops = new DBOperations(jdbi);<br>        if (jdbi == null) {<br>            System.err.println(&quot;Failed to establish jdbi. Exiting.&quot;);<br>            return;<br>        }<br>        dbops.deleteData(1237);<br>        List&lt;Map&lt;String, Object&gt;&gt; dataBySicilNo = dbops.fetchDataBySicilNo(1237);<br>        for (Map&lt;String, Object&gt; row : dataBySicilNo) {<br>            System.out.println(STR.&quot;Isim: \{row.get(&quot;adsoyad&quot;)}&quot;);<br>            System.out.println(STR.&quot;Gorev: \{row.get(&quot;gorev&quot;)}&quot;);<br>            System.out.println(STR.&quot;yas: \{row.get(&quot;yas&quot;)}&quot;);<br>            System.out.println(STR.&quot;Bolum: \{row.get(&quot;bolum&quot;)}&quot;);<br>            System.out.println(STR.&quot;Sicil No: \{row.get(&quot;sicilno&quot;)}&quot;);<br>        } <br>}<br><br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*Ff5iYpnrWq0BY0Fo9V-L1g.png" /></figure><h4>Conclusion:</h4><p>JDBI offers powerful result mapping capabilities, allowing developers to map query results directly to Java objects or collections without manually iterating through result sets. It supports various mapping strategies, including annotations and custom mappers.</p><p>While JDBI abstracts away much of the boilerplate, it still allows for flexibility and customization. Developers can use raw SQL queries when needed and extend JDBI’s functionality with custom mappers and handlers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5a1e1095ad8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JENKINS DIARIES-1]]></title>
            <link>https://medium.com/@murat.bilal/jenkins-diaries-1-8a9382769a6b?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a9382769a6b</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Sun, 18 Feb 2024 17:50:21 GMT</pubDate>
            <atom:updated>2024-02-18T17:50:21.700Z</atom:updated>
            <content:encoded><![CDATA[<p>Jenkins is an open source continous integration/continuous delivery and deployment (CI/CD) automation software DevOps tool written in the Java programming language. It is used to implement CI/CD workflows, called pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPY5jDh2n_PSS57CH6INFg.png" /></figure><p>It is designed to automate various tasks involved in the software development process, such as building, testing, and deploying code. Jenkins is extensible through a wide variety of plugins, which allows users to integrate it with different tools and technologies.</p><p>Key features of Jenkins and advantages of using it in software development include:</p><ol><li><strong>Continuous Integration (CI):</strong> Jenkins automates the process of integrating code changes from multiple contributors into a shared repository. This helps identify and fix integration issues early in the development cycle, reducing the chances of bugs and conflicts.</li><li><strong>Continuous Delivery/Deployment (CD):</strong> Jenkins supports the automation of the entire software delivery process, from code integration to testing and deployment. This leads to faster and more reliable release cycles.</li><li><strong>Extensibility: </strong>Jenkins has a large and active plugin ecosystem, allowing users to extend its functionality and integrate it with various tools and services. This flexibility makes Jenkins suitable for a wide range of development environments.</li></ol><p>This article series is all about installation and configuration of Jenkins including:</p><ol><li><strong>Install Jenkins:</strong></li></ol><ul><li>Download and install Jenkins on your server or local machine. You can find the latest version on the official Jenkins website.</li></ul><p>2. <strong>Access Jenkins Web Interface:</strong></p><ul><li>Once Jenkins is installed, access its web interface by navigating to <a href="http://localhost:8080/">http://localhost:8080</a> (by default) in your web browser. Follow the instructions to unlock Jenkins and retrieve the initial admin password from the specified location on your machine.</li></ul><p>3. <strong>Install Plugins:</strong></p><ul><li>During the initial setup, Jenkins will prompt you to install plugins. Choose the recommended plugins or select plugins based on your specific requirements. These plugins extend Jenkins’ functionality and integrate it with various tools.</li></ul><p>4. <strong>Create a Jenkins Job:</strong></p><ul><li>After installation, create a new Jenkins job by selecting “New Item” from the Jenkins dashboard.</li><li>Choose the type of job you want to create (e.g., Freestyle project, Pipeline).</li><li>Configure the job settings, such as source code repository, build triggers, and build steps.</li></ul><p>5. <strong>Configure Source Code Management:</strong></p><ul><li>Connect your Jenkins job to your version control system (e.g., Git). Configure the repository URL, credentials, and branch to monitor.</li></ul><p>6. <strong>Set Build Triggers:</strong></p><ul><li>Define the conditions that trigger a build, such as code commits, scheduled builds, or manual triggers.</li></ul><p>7. <strong>Configure Build Steps:</strong></p><ul><li>Define the build steps to be executed during the job. This can include compiling code, running tests, static code analysis, and creating artifacts.</li></ul><p>8. <strong>Post-Build Actions:</strong></p><ul><li>Specify post-build actions, such as archiving artifacts, triggering downstream jobs, or sending notifications.</li></ul><p>9. <strong>Save and Run the Job:</strong></p><ul><li>Save the job configuration and manually run the job to verify that it executes correctly. This helps identify and address any configuration issues.</li></ul><p>10. <strong>Set Up Build Pipelines:</strong></p><ul><li>For more complex workflows, consider setting up build pipelines using Jenkins Pipeline. Pipelines provide a way to define, manage, and visualize entire deployment processes.</li></ul><p>11. <strong>Configure Authentication and Authorization:</strong></p><ul><li>Set up user authentication and define access controls to secure your Jenkins instance. This ensures that only authorized users can access and modify Jenkins configurations.</li></ul><p>We are starting with Installation of Jenkins. There are various ways of installing Jenkins(i.e vm, containerized..), in this article series I am installing Jenking on a Virtual Box Ubuntu VM.</p><h4>Install Jenkins:</h4><p>I am using a Virtual Box Ubuntu VM. Here are the specs of my vm:</p><p>OS: “22.04.3 LTS (Jammy Jellyfish)”</p><p>MEMORY: 2GB</p><p>CPU: 1vcpu</p><p>DISK: 25GB</p><p>Run the commands below in order to install Jenkins:</p><pre>alcalab@wolf:~$ curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc &gt; /dev/null<br><br>alcalab@wolf:~$ echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list &gt; /dev/null<br><br>alcalab@wolf:~$ sudo apt-get update -y<br><br>alcalab@wolf:~$ sudo apt-get -y install openjdk-11-jdk<br><br>alcalab@wolf:~$ sudo apt-get -y install jenkins<br><br>// Run the following command and copy the output:<br><br>alcalab@wolf:~$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword<br>2b58b28b2abf4737873f514a95bebffd<br><br></pre><p>Open a browser to <a href="http://localhost:8080/">localhost:8080</a> and enter the initial admin password and press continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDsVb8FAZ7BvM_R06ereVA.png" /></figure><p>Now install suggested Plugins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4-WkVhIj5p_KrFJObEwNw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sY1u80sRl2F7YhU9p7Gmaw.png" /></figure><p>Create your first admin user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*6YWOZJEzhIPhuWPCWvfhgA.png" /></figure><p>Save and Finish with your instance URL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*bwW8EfVCS2wLgYPiW8EMKw.png" /></figure><p>Jenkins is ready for use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*JcV5BHieMgST9ElBvMpvIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAZd1EVrcZhKt3WTXTig6Q.png" /></figure><h4>Install/Uninstall Plugins:</h4><p>For installing a plugin(For this article I will install golang) Go to “Manage Jenkins →Plugins → Available Plugins →Type Golang and Press Install</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2GtevvWMSvaYHDFeobfDgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nr_7aEVWJXxtd2PPm1Y19A.png" /></figure><p>Now Let’s uninstall Golang:</p><p>Manage Jenkins → Plugins →Installed plugins →Type Go , then click on red cross sign</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*peM-Jor06mDAh2hj1xiNXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ah6JA2QQK0yUD7uEdu4Sg.png" /></figure><p>If you are seeing Uninstallation pending, it means restart required</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3R4NSPZVZuQOgWogOqH8g.png" /></figure><p>Go to available plugins and install “Safe Restart” plugin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AcEQVG3ROfCqALe8i_3FUg.png" /></figure><p>After that go to Dashboard and you should see Safe Restart icon, press for restarting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZh5ChhXMImPq_qmy-ReGg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpfxIxSNdou5VA8PMqnLow.png" /></figure><p>After some time should see the login page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yw1d91hawOvm0Da3OutVMg.png" /></figure><p>Go to Installed Plugins and verify that no Golang plugin is installed.</p><h4>First Jenkins Job — Hello World:</h4><p>For our first Jenkins Job , let’s create a new Item and named as Hello- Jenkins. For this when you login your Jenkins web page click on +New Item</p><p>It is better to use with Hello-Jenkins, because spaces will be problematic.Choose the Freestyle project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*URDUJxW8RWvtdc-E4mKMOg.png" /></figure><p>Leave all settings as default now and Add Built Step Exexute Shell, then save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eJja5Se7b-py9ZVoYgKH5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8KdcFvquAphR891wxx9PQ.png" /></figure><p>Click on the Job and then Build Now, then check your build history and Console Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*wMFw-O67NhErPF_zQqKpnw.png" /></figure><p>To view the console output real time, create a new job called realtimeMonitor and choose Execute Shell:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*nLBnUifegksg40FR1jT1uw.png" /></figure><p>Check the console output after click on Build Now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*Ut6TKFY0FAWyx-tkFrVVLw.png" /></figure><p>If you want to stop it Just click on red cross at the status page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FprFezQRVXZozOD56M4SA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0asLRDWWtqxzkxkX2J3THg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*iAzYiHn7erWgsXgjejmQrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*bvJ_P9PRyeDdxSG5uFMB-w.png" /></figure><p>Now let’s click on trend and open Timeline to see a graphical representation of our builds with success and aborted states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*50zTN5-T5VM6cUsnWgWbDA.png" /></figure><p>If you click and drag on the top lighter gray field, you can scroll by day. And if you click and drag on the bottom darker gray field, you can scroll by hour. Clicking a link on the timeline gives a report on that build with links to the log and the time the build started and finished. Below the timeline, is report showing how long each build took to run and which bills were successful or failed. The time trend can help you track that information. The graph also gives a visualization of the same information. And in each of these graphs, failures are shown in red to make them easy to identify.</p><p>To be continued……..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a9382769a6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MONITORING POSTGRESQL WITH GRAFANA AND PROMETHEUS IN DOCKER]]></title>
            <link>https://medium.com/@murat.bilal/monitoring-postgresql-with-grafana-and-prometheus-in-docker-7fe6a36ef7b1?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fe6a36ef7b1</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 19:48:44 GMT</pubDate>
            <atom:updated>2023-12-28T19:48:44.902Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article I setup an environment including grafana and prometheus in my docker Virtualbox VM, you can find details about my setup from this <a href="https://medium.com/p/faeffc4aa18d">link</a>.</p><p>In this article I start to monitor my Postgresql Database with prometheus and graphana which is on the same docker host.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQ-cT1L0Z0iFgEySHjzVtw.png" /></figure><p>This is the databases found in my Postgresql:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*lOGJ4Lk2OmgTiP_aoEAOJA.png" /></figure><p>I am using postgres-exporter from prometheus-community, which can be found from this <a href="https://github.com/prometheus-community/postgres_exporter#configuration-file">link</a>.</p><p>The following quickstart provides setup instructions and preconfigured dashboards and alerting rules for the <a href="https://grafana.com/oss/prometheus/exporters/postgres-exporter/">Postgres Exporter</a>.</p><p>Let’s start with running docker image for postgresql-exporter</p><pre>alcalab@wolf:~$ sudo docker run --net host -d -e DATA_SOURCE_NAME=&quot;postgresql://postgres:1qaz2wsx@@localhost:5432/postgres?sslmode=disable&quot; quay.io/prometheuscommunity/postgres-exporter:v0.11.0<br>6505d1ec15f0b4680d6c5fa3f8774a0d881859c9c6fc7d681342bce7038fe1b9</pre><p>Now I have postgresql-exporter run as a docker container</p><pre>alcalab@wolf:~$ sudo docker ps<br>CONTAINER ID   IMAGE                                                   COMMAND                  CREATED          STATUS          PORTS                                       NAMES<br>6505d1ec15f0   quay.io/prometheuscommunity/postgres-exporter:v0.11.0   &quot;/bin/postgres_expor…&quot;   38 minutes ago   Up 38 minutes                                               intelligent_swanson<br>afd21974a135   grafana/grafana                                         &quot;/run.sh&quot;                11 days ago      Up 2 hours      0.0.0.0:3000-&gt;3000/tcp, :::3000-&gt;3000/tcp   stoic_neumann2<br>d380f074ee78   quay.io/prometheus/node-exporter                        &quot;/bin/node_exporter&quot;     11 days ago      Up 2 hours                                                  confident_cartwright2<br>05552735f5ad   prom/prometheus                                         &quot;/bin/prometheus --c…&quot;   11 days ago      Up 46 minutes   0.0.0.0:9090-&gt;9090/tcp, :::9090-&gt;9090/tcp   jovial_spence2<br>41a816217096   postgres:15.2                                           &quot;docker-entrypoint.s…&quot;   2 months ago     Up 2 hours      0.0.0.0:5432-&gt;5432/tcp, :::5432-&gt;5432/tcp   postgresql1</pre><p>Here is my network configs for docker containers:</p><pre>alcalab@wolf:~$ sudo docker inspect 6505d1ec15f0 -f &quot;{{json .NetworkSettings.Networks }}&quot;<br>{&quot;host&quot;:{&quot;IPAMConfig&quot;:null,&quot;Links&quot;:null,&quot;Aliases&quot;:null,&quot;NetworkID&quot;:&quot;16d7dce47598bcc1791088e76b5fcc4395c281879da1ceaa2823cf7b0d6acbef&quot;,&quot;EndpointID&quot;:&quot;dfbc1f33b0db770a5a1ca99e50647bd031f1e9ef92afb34b0ec0e7812199fd48&quot;,&quot;Gateway&quot;:&quot;&quot;,&quot;IPAddress&quot;:&quot;&quot;,&quot;IPPrefixLen&quot;:0,&quot;IPv6Gateway&quot;:&quot;&quot;,&quot;GlobalIPv6Address&quot;:&quot;&quot;,&quot;GlobalIPv6PrefixLen&quot;:0,&quot;MacAddress&quot;:&quot;&quot;,&quot;DriverOpts&quot;:null}}<br>alcalab@wolf:~$ sudo docker inspect afd21974a135 -f &quot;{{json .NetworkSettings.Networks }}&quot;<br>{&quot;bridge&quot;:{&quot;IPAMConfig&quot;:null,&quot;Links&quot;:null,&quot;Aliases&quot;:null,&quot;NetworkID&quot;:&quot;6ebcbc3202942ed47b36f1ce219cb735a725dd8c41fd48ccfd182eba718daeb1&quot;,&quot;EndpointID&quot;:&quot;deeac5d398c8bc58ce8aa2e0c0bd2b8e3b811b79234830dee0e5c7a96aef75c1&quot;,&quot;Gateway&quot;:&quot;172.17.0.1&quot;,&quot;IPAddress&quot;:&quot;172.17.0.2&quot;,&quot;IPPrefixLen&quot;:16,&quot;IPv6Gateway&quot;:&quot;&quot;,&quot;GlobalIPv6Address&quot;:&quot;&quot;,&quot;GlobalIPv6PrefixLen&quot;:0,&quot;MacAddress&quot;:&quot;02:42:ac:11:00:02&quot;,&quot;DriverOpts&quot;:null}}<br>alcalab@wolf:~$ sudo docker inspect d380f074ee78 -f &quot;{{json .NetworkSettings.Networks }}&quot;<br>{&quot;host&quot;:{&quot;IPAMConfig&quot;:null,&quot;Links&quot;:null,&quot;Aliases&quot;:null,&quot;NetworkID&quot;:&quot;16d7dce47598bcc1791088e76b5fcc4395c281879da1ceaa2823cf7b0d6acbef&quot;,&quot;EndpointID&quot;:&quot;d152d9975076c5f7f2607c4675c1183f04354d0748b88b7fb29d87fc558e5199&quot;,&quot;Gateway&quot;:&quot;&quot;,&quot;IPAddress&quot;:&quot;&quot;,&quot;IPPrefixLen&quot;:0,&quot;IPv6Gateway&quot;:&quot;&quot;,&quot;GlobalIPv6Address&quot;:&quot;&quot;,&quot;GlobalIPv6PrefixLen&quot;:0,&quot;MacAddress&quot;:&quot;&quot;,&quot;DriverOpts&quot;:null}}<br>alcalab@wolf:~$ sudo docker inspect 05552735f5ad -f &quot;{{json .NetworkSettings.Networks }}&quot;<br>{&quot;bridge&quot;:{&quot;IPAMConfig&quot;:null,&quot;Links&quot;:null,&quot;Aliases&quot;:null,&quot;NetworkID&quot;:&quot;6ebcbc3202942ed47b36f1ce219cb735a725dd8c41fd48ccfd182eba718daeb1&quot;,&quot;EndpointID&quot;:&quot;bbf6067e1b7959a68880fdb9e3bea855914c9f8d4c4bb1347959199375767ff8&quot;,&quot;Gateway&quot;:&quot;172.17.0.1&quot;,&quot;IPAddress&quot;:&quot;172.17.0.5&quot;,&quot;IPPrefixLen&quot;:16,&quot;IPv6Gateway&quot;:&quot;&quot;,&quot;GlobalIPv6Address&quot;:&quot;&quot;,&quot;GlobalIPv6PrefixLen&quot;:0,&quot;MacAddress&quot;:&quot;02:42:ac:11:00:05&quot;,&quot;DriverOpts&quot;:null}}<br>alcalab@wolf:~$ sudo docker inspect 41a816217096 -f &quot;{{json .NetworkSettings.Networks }}&quot;<br>{&quot;bridge&quot;:{&quot;IPAMConfig&quot;:{},&quot;Links&quot;:null,&quot;Aliases&quot;:[],&quot;NetworkID&quot;:&quot;6ebcbc3202942ed47b36f1ce219cb735a725dd8c41fd48ccfd182eba718daeb1&quot;,&quot;EndpointID&quot;:&quot;582cbd04172bec16f6799260a0510bb72a2129de0a51dba5e24228d0e4d1ebfb&quot;,&quot;Gateway&quot;:&quot;172.17.0.1&quot;,&quot;IPAddress&quot;:&quot;172.17.0.4&quot;,&quot;IPPrefixLen&quot;:16,&quot;IPv6Gateway&quot;:&quot;&quot;,&quot;GlobalIPv6Address&quot;:&quot;&quot;,&quot;GlobalIPv6PrefixLen&quot;:0,&quot;MacAddress&quot;:&quot;02:42:ac:11:00:04&quot;,&quot;DriverOpts&quot;:{}}}</pre><p>Add postgresql-exporter to prometheus.yaml and restart prometheus container.10.0.2.15 is the IP of my Oracle VirtualBox VM</p><pre>alcalab@wolf:~$ cat prometheus.yml<br>global:<br>  scrape_interval: 15s<br><br>scrape_configs:<br>  - job_name: &#39;prometheus&#39;<br>    static_configs:<br>      - targets: [&#39;localhost:9090&#39;]<br><br>  - job_name: &#39;node-exporter&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9100&#39;]<br>  - job_name: &#39;docker&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9323&#39;]<br>  - job_name: &#39;postgres-exporter&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9187&#39;]<br><br>alcalab@wolf:~$ sudo docker stop jovial_spence2<br>alcalab@wolf:~$ sudo docker start jovial_spence2</pre><p>You should see postgresql-exporter UP from prometheus targets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JW2tEN4lRr-licBU8erJdA.png" /></figure><p>Prometheus data source is already added to Grafana, now it is time to import PostgreSQL Dashboard for Prometheus.</p><p>For this, Login to Grafana, I forgot password for admin , so If you want to reset admin password please follow below steps:</p><pre># First login to grafana docker container<br><br>alcalab@wolf:~$ sudo docker exec -it afd21974a135 bash<br><br>afd21974a135:/usr/share/grafana$ grafana-cli admin reset-admin-password admin<br>Deprecation warning: The standalone &#39;grafana-cli&#39; program is deprecated and will be removed in the future. Please update all uses of &#39;grafana-cli&#39; to &#39;grafana cli&#39;<br>INFO [12-28|18:42:27] Starting Grafana                         logger=settings version= commit= branch= compiled=1970-01-01T00:00:00Z<br>INFO [12-28|18:42:27] Config loaded from                       logger=settings file=/usr/share/grafana/conf/defaults.ini<br>INFO [12-28|18:42:27] Config overridden from Environment variable logger=settings var=&quot;GF_PATHS_DATA=/var/lib/grafana&quot;<br>INFO [12-28|18:42:27] Config overridden from Environment variable logger=settings var=&quot;GF_PATHS_LOGS=/var/log/grafana&quot;<br>INFO [12-28|18:42:27] Config overridden from Environment variable logger=settings var=&quot;GF_PATHS_PLUGINS=/var/lib/grafana/plugins&quot;<br>INFO [12-28|18:42:27] Config overridden from Environment variable logger=settings var=&quot;GF_PATHS_PROVISIONING=/etc/grafana/provisioning&quot;<br>INFO [12-28|18:42:27] Target                                   logger=settings target=[all]<br>INFO [12-28|18:42:27] Path Home                                logger=settings path=/usr/share/grafana<br>INFO [12-28|18:42:27] Path Data                                logger=settings path=/var/lib/grafana<br>INFO [12-28|18:42:27] Path Logs                                logger=settings path=/var/log/grafana<br>INFO [12-28|18:42:27] Path Plugins                             logger=settings path=/var/lib/grafana/plugins<br>INFO [12-28|18:42:27] Path Provisioning                        logger=settings path=/etc/grafana/provisioning<br>INFO [12-28|18:42:27] App mode production                      logger=settings<br>INFO [12-28|18:42:28] Connecting to DB                         logger=sqlstore dbtype=sqlite3<br>INFO [12-28|18:42:28] Starting DB migrations                   logger=migrator<br>INFO [12-28|18:42:28] migrations completed                     logger=migrator performed=0 skipped=517 duration=234.651µs<br>INFO [12-28|18:42:28] Envelope encryption state                logger=secrets enabled=true current provider=secretKey.v1<br><br>Admin password changed successfully ✔</pre><p>Now password is resetted to admin again, you can change with new password when try to login with admin/admin:</p><p>For importing Postgresql dashboard</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SHLSmQqiyrvU95Ir-47kvg.jpeg" /><figcaption>Click on Import a dashboard</figcaption></figure><p>You have to type 9628 for JSON code <a href="https://grafana.com/grafana/dashboards/9628">https://grafana.com/grafana/dashboards/9628</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZzXTkv3AZtR5GWurWARcw.jpeg" /><figcaption>Enter 9628 and click load on the right</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnxbBEM5IiGx5UmyrWeCKw.jpeg" /><figcaption>Click Import with default settings</figcaption></figure><p>Now Postgresql dashboard is ready:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78oqIxT4DkSK4Z6A_ibPDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IuvD-btlcpw3X2bdp8Tjdg.png" /></figure><p>Now let’s do some load tests using pgbench.</p><p>Connecting to our docker postgresql server</p><pre>alcalab@wolf:~$ sudo docker exec -it postgresql1 bash<br>root@41a816217096:/# which pgbench<br>/usr/bin/pgbench<br><br># If pgbench is not found you need to install postgresql-contrib<br><br>root@41a816217096:/# su - postgres<br>postgres@41a816217096:~$ psql<br>psql (15.2 (Debian 15.2-1.pgdg110+1))<br>Type &quot;help&quot; for help.<br><br>postgres=# \l<br>                                                   List of databases<br>      Name      |  Owner   | Encoding |  Collate   |   Ctype    | ICU Locale | Locale Provider |   Access privileges<br>----------------+----------+----------+------------+------------+------------+-----------------+-----------------------<br> baseball       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> dvdrental      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> exercises      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> postgres       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> samples_nullif | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> school         | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> teams          | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> template0      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            | =c/postgres          +<br>                |          |          |            |            |            |                 | postgres=CTc/postgres<br> template1      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            | =c/postgres          +<br>                |          |          |            |            |            |                 | postgres=CTc/postgres<br> örnekler      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br>(10 rows)<br><br>postgres=# CREATE DATABASE deneme;<br>CREATE DATABASE<br>postgres=# \l<br>                                                   List of databases<br>      Name      |  Owner   | Encoding |  Collate   |   Ctype    | ICU Locale | Locale Provider |   Access privileges<br>----------------+----------+----------+------------+------------+------------+-----------------+-----------------------<br> baseball       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> deneme         | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> dvdrental      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> exercises      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> postgres       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> samples_nullif | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> school         | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> teams          | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br> template0      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            | =c/postgres          +<br>                |          |          |            |            |            |                 | postgres=CTc/postgres<br> template1      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            | =c/postgres          +<br>                |          |          |            |            |            |                 | postgres=CTc/postgres<br> örnekler      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |            | libc            |<br>(11 rows)<br><br>postgres=# \q<br>postgres@41a816217096:~$ pgbench -i -s 50 deneme<br>dropping old tables...<br>NOTICE:  table &quot;pgbench_accounts&quot; does not exist, skipping<br>NOTICE:  table &quot;pgbench_branches&quot; does not exist, skipping<br>NOTICE:  table &quot;pgbench_history&quot; does not exist, skipping<br>NOTICE:  table &quot;pgbench_tellers&quot; does not exist, skipping<br>creating tables...<br>generating data (client-side)...<br>5000000 of 5000000 tuples (100%) done (elapsed 4.43 s, remaining 0.00 s)<br>vacuuming...<br>creating primary keys...<br>pgbench: error: query failed: ERROR:  could not extend file &quot;base/17897/17918&quot;: wrote only 4096 of 8192 bytes at block 9834<br>HINT:  Check free disk space.<br>pgbench: detail: Query was: alter table pgbench_accounts add primary key (aid)<br><br># I have not enough disk space but let&#39;s continue</pre><p>Now from grafana you should see the database named “deneme” stats</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*bQssq3JwnEcoXPplAOWfKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsSi4OPgwT90pNKzlUn3yQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFaz4YKBOFEVN0SSMP-Xfw.png" /></figure><h4><strong>Conclusion:</strong></h4><p>Note that depending on its configuration, Postgres Exporter may collect and publish far more metrics than this default set. To learn more about configuring Postgres Exporter and toggling its collectors, please see the Postgres Exporter <a href="https://github.com/prometheus-community/postgres_exporter">Github Repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fe6a36ef7b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MONITORING HOST VM WITH GRAFANA AND PROMETHEUS IN DOCKER]]></title>
            <link>https://medium.com/@murat.bilal/monitoring-host-vm-with-grafana-and-prometheus-in-docker-faeffc4aa18d?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/faeffc4aa18d</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 18:12:47 GMT</pubDate>
            <atom:updated>2023-12-17T18:12:47.365Z</atom:updated>
            <content:encoded><![CDATA[<p>Grafana is a multi-platform open source analytical and visualization tool consisting of multiple individual panels arranged in a grid. The panels interact with configured data sources like AWS CloudWatch, Microsoft SQL server, Prometheus, MySQL, InfluxDB, and many others. The design of Grafana is such that each panel is tied to a data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_mAsQ1TPf3zZ5crKmiBng.png" /></figure><p>Prometheus is an open-source tool that enables you to monitor databases, VMs, and basically anything else. With its cloud-native monitoring capabilities, there’s a time-series database that sweeps data from a bunch of exporters that you define. In order to export all of the resource-monitoring metrics that you need to properly monitor your database servers the most popular exporter is node-exporter for Prometheus at system-level resource monitoring.</p><p>Monitoring Host VM with Grafana and Prometheus in Docker involves setting up the following components:</p><ol><li>Host VM : The Ubuntu VM that you want to monitor.</li><li>Prometheus: A monitoring and alerting toolkit.</li><li>Grafana: A platform for monitoring and observability.</li><li>Node Exporter (Optional): A Prometheus exporter for system metrics.</li></ol><p>I am using VirtualBox in this demo and all my related docker containers are running on this Ubuntu VirtualBox VM</p><h3>Step 1: Set Up Prometheus</h3><p>Create a prometheus.yml file for Prometheus configuration. Here&#39;s a simple example:</p><pre>alcalab@wolf:~$ cat prometheus.yml<br>global:<br>  scrape_interval: 15s<br><br>scrape_configs:<br>  - job_name: &#39;prometheus&#39;<br>    static_configs:<br>      - targets: [&#39;localhost:9090&#39;]<br></pre><p>I created a network rather than using bridge network :</p><pre>docker network create mynetwork<br>docker network connect mynetwork &lt;prometheus_container_id&gt;<br>docker network connect mynetwork &lt;grafana_container_id&gt;<br><br><br></pre><p>Run prometheus using docker:</p><p>docker run -d -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus</p><pre>  alcalab@wolf:~$ sudo docker run -d -p 9090:9090 -v /home/alcalab/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus</pre><h3>Step 2: Set Up Grafana</h3><p>Run Grafana using Docker:</p><pre>alcalab@wolf:~$ sudo docker run -d -p 3000:3000 grafana/grafana<br>Unable to find image &#39;grafana/grafana:latest&#39; locally<br>latest: Pulling from grafana/grafana<br>96526aa774ef: Pull complete<br>932a65841cb5: Pull complete<br>a486d8a79b5f: Pull complete<br>9f17a73c904a: Pull complete<br>c1ade82e0f62: Pull complete<br>882619e0a642: Pull complete<br>2f0808654570: Pull complete<br>df5835f957f7: Pull complete<br>b94284e881a5: Pull complete<br>3e944d3294a3: Pull complete<br>Digest: sha256:e3e9c2b5776fe3657f4954dfa91579224f98a0316f51d431989b15425e95530f<br>Status: Downloaded newer image for grafana/grafana:latest<br>bd7595b6d65bc0eec05e8ff6de5101856b83dd36a25ccaefefbe4168d9acf315</pre><p>Access Grafana at <a href="http://localhost:3000/">http://localhost:3000</a> and log in with the default credentials (admin/admin). Change the password immediately.</p><p>Before that, we need Port forwarding on Oracle Virtual Box for both grafana and prometheus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgUGYSFX7CAL3gUHm3RUDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WiubfIVJ8GZowjmeSUIDQw.png" /></figure><h3>Step 3: Monitoring Node Metrics</h3><pre>alcalab@wolf:~$ sudo docker run -d -p 9100:9100 --net=&quot;host&quot; quay.io/prometheus/node-exporter<br></pre><p>With the host network mode, the Node Exporter is binding to the host’s network namespace, and it can be accessed on localhost from other containers on the same host.</p><p>Add a new job to Prometheus yaml</p><pre>global:<br>  scrape_interval: 15s<br><br>scrape_configs:<br>  - job_name: &#39;prometheus&#39;<br>    static_configs:<br>      - targets: [&#39;localhost:9090&#39;]<br><br> -  job_name: &#39;node-exporter&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9100&#39;] #This my VirtualBox VM</pre><p>Restart prometheus for the changes to take effect.</p><pre>alcalab@wolf:~$ sudo docker ps<br>CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS          PORTS                                       NAMES<br>95676490d997   quay.io/prometheus/node-exporter   &quot;/bin/node_exporter&quot;     4 minutes ago    Up 4 minutes                                                confident_cartwright<br>bd7595b6d65b   grafana/grafana                    &quot;/run.sh&quot;                44 minutes ago   Up 44 minutes   0.0.0.0:3000-&gt;3000/tcp, :::3000-&gt;3000/tcp   stoic_neumann<br>b00228fc1cd9   prom/prometheus                    &quot;/bin/prometheus --c…&quot;   44 minutes ago   Up 18 seconds   0.0.0.0:9090-&gt;9090/tcp, :::9090-&gt;9090/tcp   jovial_spence</pre><h3>Step 4: Configure Grafana</h3><p>Add Prometheus as a data source:</p><ul><li>Navigate to “Settings” &gt; “Data Sources” in Grafana.</li><li>Click “Add your first data source.”</li><li>Choose “Prometheus” from the list.</li><li>Set the URL to http://localhost:9090 or the address where your Prometheus instance is running. I am using my VM address</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwMhF-CDQzqeBBvd5C4R3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FM0G4bhZke44k98_0OaPqw.png" /></figure><p>My prometheus address is:</p><pre>alcalab@wolf:~$ sudo docker ps<br>CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                                       NAMES<br>bd7595b6d65b   grafana/grafana   &quot;/run.sh&quot;                23 minutes ago   Up 23 minutes   0.0.0.0:3000-&gt;3000/tcp, :::3000-&gt;3000/tcp   stoic_neumann<br>b00228fc1cd9   prom/prometheus   &quot;/bin/prometheus --c…&quot;   24 minutes ago   Up 24 minutes   0.0.0.0:9090-&gt;9090/tcp, :::9090-&gt;9090/tcp   jovial_spence<br>41a816217096   postgres:15.2     &quot;docker-entrypoint.s…&quot;   2 months ago     Up 29 minutes   0.0.0.0:5432-&gt;5432/tcp, :::5432-&gt;5432/tcp   postgresql1<br><br>lcalab@wolf:~$ sudo docker inspect b00228fc1cd9 |grep IPAdd<br>            &quot;SecondaryIPAddresses&quot;: null,<br>            &quot;IPAddress&quot;: &quot;&quot;,<br>                    &quot;IPAddress&quot;: &quot;172.18.0.3&quot;,</pre><p>Create a Dashboard:</p><ul><li>Use the Grafana dashboard</li><li>Navigate to “Home” &gt; “Dashboards” and New Dashboard.</li><li>For Average CPU you can use a metric like node_cpu_seconds_total from Node Exporter.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Zr2aOYvWn6QP4GGVqcH3w.png" /></figure><p>Press Apply</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qrFaoqGSGl-7-SFy8c8_A.png" /></figure><p>Now For Memory related examples:</p><p>Total Memory:<br>Metric: node_memory_MemTotal_bytes<br>Description: Total amount of installed physical memory.</p><p>Free Memory:<br>Metric: node_memory_MemFree_bytes<br>Description: Amount of physical memory left unused by the system.</p><p>Used Memory:<br>Metric: node_memory_MemUsed_bytes<br>Description: Total physical memory minus free memory.</p><p>Cached Memory:<br>Metric: node_memory_Cached_bytes<br>Description: Amount of physical memory used as cache memory.</p><p>Buffers Memory:<br>Metric: node_memory_Buffers_bytes<br>Description: Amount of physical memory used as buffers.</p><p>You can configure all of this from grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOcJGXKvzvGTSM1Lb9mz_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*OWt-N1445lkmCBUXLvPAbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*ERPjE7LBHRjP0rrWel5ZOA.png" /></figure><p>To enable the Docker daemon metrics endpoint, you need to modify the Docker daemon configuration. The specific location of the Docker daemon configuration file can vary depending on your operating system.For Ubuntu22.04 You may need to create the override.conf file if it doesn&#39;t exist.</p><pre>alcalab@wolf:~$ cat /etc/systemd/system/docker.service.d/override.conf<br>cat: /etc/systemd/system/docker.service.d/override.conf: No such file or directory<br>alcalab@wolf:~$ sudo mkdir -p /etc/systemd/system/docker.service.d<br>[sudo] password for alcalab:<br>alcalab@wolf:~$ sudo vi /etc/systemd/system/docker.service.d/override.conf<br># Add this on override.conf<br>[Service]<br>ExecStart=<br>ExecStart=/usr/bin/dockerd --metrics-addr=0.0.0.0:9323<br><br>alcalab@wolf:~$ sudo systemctl daemon-reload<br>alcalab@wolf:~$ sudo systemctl restart docker</pre><p>Now configure prometheus and restart:</p><pre># ADD this to prometheus.yaml file<br><br> - job_name: &#39;docker&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9323&#39;]<br><br>alcalab@wolf:~$ cat prometheus.yml<br>global:<br>  scrape_interval: 15s<br><br>scrape_configs:<br>  - job_name: &#39;prometheus&#39;<br>    static_configs:<br>      - targets: [&#39;localhost:9090&#39;]<br><br>  - job_name: &#39;node-exporter&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9100&#39;]<br>  - job_name: &#39;docker&#39;<br>    static_configs:<br>      - targets: [&#39;10.0.2.15:9323&#39;]  # Replace with the IP of your Docker daemon<br><br></pre><p>In Grafana, add panels that query Prometheus for Docker daemon metrics. Some example metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*577tuEhkjzaBx1SWvfXsRg.png" /></figure><p>If you are using a custom setup or a different exporter, the metric names might differ. You can explore the available metrics by browsing the /metrics endpoint of your Node Exporter or checking the exporter&#39;s documentation.</p><p>It is important to note that docker containers should be labeled as docker. This label is crucial for Prometheus to identify and scrape metrics from Docker containers.</p><p>Check labels with below command:</p><pre>sudo docker ps --format &#39;{{.Names}}: {{.Labels}}&#39;</pre><p>To add a label to an existing Docker container, you would need to stop the container, create a new one with the desired labels, and then remove the old container. Here are the steps:</p><pre>alcalab@wolf:~$ sudo docker stop jovial_spence<br>alcalab@wolf:~$ sudo docker run --name jovial_spence2 --label job=docker -d prom/prometheus<br>a5b1ac51bae72fc4ea32ed970dfc8d328968f73c5aeda53a3ce387bb9666cfe0<br>alcalab@wolf:~$ sudo docker rm jovial_spence<br><br>alcalab@wolf:~$ sudo docker ps<br>[sudo] password for alcalab:<br>CONTAINER ID   IMAGE                              COMMAND                  CREATED             STATUS             PORTS                                       NAMES<br>afd21974a135   grafana/grafana                    &quot;/run.sh&quot;                44 minutes ago      Up 44 minutes      0.0.0.0:3000-&gt;3000/tcp, :::3000-&gt;3000/tcp   stoic_neumann2<br>d380f074ee78   quay.io/prometheus/node-exporter   &quot;/bin/node_exporter&quot;     50 minutes ago      Up 50 minutes                                                  confident_cartwright2<br>05552735f5ad   prom/prometheus                    &quot;/bin/prometheus --c…&quot;   About an hour ago   Up About an hour   0.0.0.0:9090-&gt;9090/tcp, :::9090-&gt;9090/tcp   jovial_spence2</pre><p>Here is an example about the states of containers 3 of them are running 2 of them are stopped</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Oxy-oAcaEsQuhHeYZTPfA.png" /></figure><p>Now I want to show one example about Active TCP Opens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZPd58PSpaYSpxuEZwAvqbw.png" /></figure><p>And My Prometheus Targets and job labels;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1cooaQ5yD4gnGRreoTN1uQ.png" /></figure><p>My Node Network Assign Type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1q8LbV81fzxVZjSEOXF3NA.png" /></figure><p>You can find your node exporter metrics by querying <a href="http://127.0.0.1:9100/metrics">http://127.0.0.1:9100/metrics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*6LNaL-hKuiuQqtxmBpJu-Q.png" /></figure><p>For example choose last one node_network_transmit_bytes_total{device=”enp0s3&quot;}</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IENJ0WGLkaPin3FYyOiwtA.png" /></figure><p>Or change the view as stat</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfVhf1HlyYmqrj1oJWPI6A.png" /></figure><h4>Conclusion:</h4><p>This a very brief introduction for Prometheus and Graphana in Docker. You can easily create these image files on a Virtual Box Ubuntu VM and test for Graphana and Prometheus features.</p><p>Prometheus and Grafana are both great observability solutions. However, they were created with different priorities. Prometheus focuses on data acquisition, allowing users to select and aggregate time series data in real time. Grafana, on the other hand, focuses on data visualization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=faeffc4aa18d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SETTING UP A POSTGRESQL HA CLUSTER]]></title>
            <link>https://medium.com/@murat.bilal/setting-up-a-postgresql-ha-cluster-0a4348fca444?source=rss-7946e5aff990------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a4348fca444</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[ubuntu-server]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Murat Bilal]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 18:19:46 GMT</pubDate>
            <atom:updated>2023-11-17T18:19:46.539Z</atom:updated>
            <content:encoded><![CDATA[<p>Setting up a High Availability (HA) cluster for PostgreSQL typically involves configuring multiple nodes to ensure data availability and reliability. There are various ways to achieve this, in this tutorial I deploy 3 PostgreSQL nodes with Patroni for HA setting up etcd for consensus, and using HAProxy to manage client connections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*Ii0QLnH06AmWof6hNAwfwg.png" /></figure><p>The need for shared storage in a PostgreSQL high availability (HA) cluster depends on the specific architecture and requirements of your deployment. PostgreSQL itself doesn’t require shared storage for a basic HA setup, but shared storage might be beneficial for certain configurations. Here are some considerations:</p><h3>Local Disks:</h3><p>Advantages:</p><ul><li>Simplicity: Setting up a PostgreSQL cluster with local disks is usually simpler and easier to manage.</li><li>Cost: Local disks are often more cost-effective, especially in smaller deployments.</li></ul><p>Considerations:</p><ul><li>Data Replication: If you are using tools like Patroni for HA, each node typically manages its own local copy of the database. Replication mechanisms are used to keep the data in sync between nodes.</li></ul><p>Use Cases:</p><ul><li>Small to medium-sized deployments.</li><li>When simplicity and cost are critical factors.</li></ul><h3>Shared Storage:</h3><p>Advantages:</p><ul><li>Shared Data: Shared storage allows multiple nodes to access the same data. This can simplify data consistency and reduce the need for complex replication mechanisms.</li></ul><p>Considerations:</p><ul><li>Complexity: Setting up and maintaining shared storage can be more complex, requiring additional configuration and management.</li></ul><p>Cost: Shared storage solutions can be more expensive than local disks.</p><p>Use Cases:</p><ul><li>Large-scale deployments where data consistency and central management are critical.</li><li>When shared storage is already part of the existing infrastructure.</li></ul><h3>Hybrid Approaches:</h3><p>Advantages:</p><ul><li>Flexibility: You can use a combination of local storage and shared storage to balance cost, complexity, and performance.</li></ul><p>Considerations:</p><ul><li>Data Distribution: Carefully plan how data is distributed and replicated between nodes to ensure high availability.</li></ul><p>Use Cases:</p><ul><li>Deployments with specific requirements that benefit from a hybrid approach.</li></ul><h3>Prerequisites:</h3><p>1 . 3 postgresql cluster node (ubuntu 22.04 minimal install) and postgres14</p><p>2. 1 etcd host (ubuntu 22.04 minimal install)</p><p>3. 1 haproxy host (ubuntu 22.04 minimal install)</p><p>My IP Plan :</p><p>Postgresql and patroni node1 10.252.55.129</p><p>Postgresql and patroni node2 10.252.54.125</p><p>Postgresql and patroni node3 10.252.54.87</p><p>etcd database node4 10.252.54.174</p><p>HAproxy 10.252.54.81</p><h3>Steps:</h3><ol><li>Upgrade all nodes</li></ol><pre>sudo apt update<br>sudo apt upgrade</pre><p>2. Edit all /etc/hosts For example for node1:</p><pre>10.252.55.129  node1<br>10.252.54.125  node2<br>10.252.54.87   node3<br>10.252.54.174  node4<br>10.252.54.81   haproxy</pre><p>3. Install postgresql server software on node1,node2 and node3. Also stop postgresql service after installation:</p><pre>alcalab@node1:~$ sudo apt install postgresql postgresql-contrib -y<br>alcalab@node1:~$ sudo systemctl stop postgresql</pre><p>4. Create a symlink on all 3 nodes between /usr/lib/postgresql/14/bin/ and /usr/sbin for patroni requirement:</p><pre>sudo ln -s /usr/lib/postgresql/14/bin/* /usr/sbin</pre><p>5. Install patroni on node1,node2,node3</p><pre>sudo apt -y install python3-pip python3-dev libpq-dev<br>pip3 install --upgrade pip<br>sudo pip install patroni<br>sudo pip install python-etcd<br>sudo pip install psycopg2</pre><p>6. Install etcd on node4</p><pre>sudo apt -y install etcd</pre><p>7. Install haproxy on node named as haproxy</p><pre>sudo apt -y install haproxy</pre><p>8. Configuration of etcd:</p><ul><li>Edit etcd config file / add below on /etc/default/etcd</li></ul><pre>ETCD_LISTEN_PEER_URLS  =  &quot;http://10.252.54.174:2380,http://10.252.54.174:7001&quot;<br>ETCD_LISTEN_CLIENT_URLS  =  &quot;http://127.0.0.1:2379, http://10.252.54.174:3379&quot;<br>ETCD_INITIAL_ADVERTISE_PEER_URLS  =  &quot;http://10.252.54.174:2380&quot;<br>ETCD_INITIAL_CLUSTER  =  &quot;etcd0=http://10.252.54.174:2380&quot;<br>ETCD_ADVERTISE_CLIENT_URLS  =  &quot;http://10.252.54.174:2379&quot;<br>ETCD_INITIAL_CLUSTER_TOKEN  =  &quot;node1&quot;<br>ETCD_INITIAL_CLUSTER_STATE  =  &quot;new&quot;<br><br>alcalab@node4:~$ sudo systemctl restart etcd<br>alcalab@node4:~$ sudo systemctl status etcd</pre><p>9. Configuration of patroni on node1:</p><p>. Create /etc/patroni.yml and add below lines to patroni.yml</p><pre>alcalab@node1:~$ sudo vi /etc/patroni.yml<br><br># Add below lines<br><br>scope: postgres<br>namespace: /db/<br>name: node1<br>restapi:<br>  listen: 10.252.55.129:8008<br>  connect_address: 10.252.55.129:8008<br>etcd:<br>  host: 10.252.54.174:2379<br>bootstrap:<br>  dcs:<br>    ttl: 30<br>    loop_wait: 10<br>    retry_timeout: 10<br>    maximum_lag_on_failover: 1048576<br>    postgresql:<br>    use_pg_rewind: true<br>  initdb:<br>    - encoding: UTF8<br>    - data-checksums<br>  pg_hba:<br>    - host replication replicator   127.0.0.1/32 md5<br>    - host replication replicator   10.252.55.129/0   md5<br>    - host replication replicator   10.252.54.125/0   md5<br>    - host replication replicator   10.252.54.87/0   md5<br>    - host all all   0.0.0.0/0   md5<br>  users:<br>    admin:<br>       password: admin<br>       options:<br>       - createrole<br>       - createdb<br>postgresql:<br>   listen: 10.252.55.129:5432<br>   connect_address: 10.252.55.129:5432<br>   data_dir:     /data/patroni<br>   pgpass:     /tmp/pgpass<br>   authentication:<br>    replication:<br>      username:   replicator<br>      password:     &quot;A1qaz2wsx3edc&quot;<br>    superuser:<br>      username:   postgres<br>      password:     &quot;B1qaz2wsx3edc&quot;<br>      parameters:<br>      unix_socket_directories:  &#39;.&#39;<br>tags:<br>   nofailover:   false<br>   noloadbalance:   false<br>   clonefrom:   false<br>   nosync:   false<br></pre><p>10. Configuration of patroni on node2:</p><p>. Create /etc/patroni.yml and add below lines to patroni.yml</p><pre>alcalab@node2:~$ sudo vi /etc/patroni.yml<br><br># Add below lines<br><br>scope: postgres<br>namespace: /db/<br>name: node2<br>restapi:<br>  listen: 10.252.54.125:8008<br>  connect_address: 10.252.54.125:8008<br>etcd:<br>  host: 10.252.54.174:2379<br>bootstrap:<br>  dcs:<br>    ttl: 30<br>    loop_wait: 10<br>    retry_timeout: 10<br>    maximum_lag_on_failover: 1048576<br>    postgresql:<br>    use_pg_rewind: true<br>  initdb:<br>    - encoding: UTF8<br>    - data-checksums<br>  pg_hba:<br>    - host replication replicator   127.0.0.1/32 md5<br>    - host replication replicator   10.252.55.129/0   md5<br>    - host replication replicator   10.252.54.125/0   md5<br>    - host replication replicator   10.252.54.87/0   md5<br>    - host all all   0.0.0.0/0   md5<br>  users:<br>    admin:<br>       password: admin<br>       options:<br>       - createrole<br>       - createdb<br>postgresql:<br>   listen: 10.252.54.125:5432<br>   connect_address: 10.252.54.125:5432<br>   data_dir:     /data/patroni<br>   pgpass:     /tmp/pgpass<br>   authentication:<br>    replication:<br>      username:   replicator<br>      password:   &quot;A1qaz2wsx3edc&quot;<br>    superuser:<br>      username:   postgres<br>      password:   &quot;B1qaz2wsx3edc&quot;<br>      parameters:<br>      unix_socket_directories:  &#39;.&#39;<br>tags:<br>   nofailover:   false<br>   noloadbalance:   false<br>   clonefrom:   false<br>   nosync:   false<br></pre><p>11. Configuration of patroni on node3:</p><p>. Create /etc/patroni.yml and add below lines to patroni.yml</p><pre>alcalab@node3:~$ sudo vi /etc/patroni.yml<br><br># Add below lines<br><br>scope: postgres<br>namespace: /db/<br>name: node3<br>restapi:<br>  listen: 10.252.54.87:8008<br>  connect_address: 10.252.54.87:8008<br>etcd:<br>  host: 10.252.54.174:2379<br>bootstrap:<br>  dcs:<br>    ttl: 30<br>    loop_wait: 10<br>    retry_timeout: 10<br>    maximum_lag_on_failover: 1048576<br>    postgresql:<br>    use_pg_rewind: true<br>  initdb:<br>    - encoding: UTF8<br>    - data-checksums<br>  pg_hba:<br>    - host replication replicator   127.0.0.1/32 md5<br>    - host replication replicator   10.252.55.129/0   md5<br>    - host replication replicator   10.252.54.125/0   md5<br>    - host replication replicator   10.252.54.87/0   md5<br>    - host all all   0.0.0.0/0   md5<br>  users:<br>    admin:<br>       password: admin<br>       options:<br>       - createrole<br>       - createdb<br>postgresql:<br>   listen: 10.252.54.87:5432<br>   connect_address: 10.252.54.87:5432<br>   data_dir:     /data/patroni<br>   pgpass:     /tmp/pgpass<br>   authentication:<br>    replication:<br>      username:   replicator<br>      password:   &quot;A1qaz2wsx3edc&quot;<br>    superuser:<br>      username:   postgres<br>      password:   &quot;B1qaz2wsx3edc&quot;<br>      parameters:<br>      unix_socket_directories:  &#39;.&#39;<br>tags:<br>   nofailover:   false<br>   noloadbalance:   false<br>   clonefrom:   false<br>   nosync:   false</pre><p>12. Create patroni data directory on node1,node2 and node3:</p><pre>alcalab@node1:~$ sudo mkdir -p  /data/patroni<br>alcalab@node1:~$ sudo chown postgres:postgres /data/patroni/<br>alcalab@node1:~$ sudo chmod 700 /data/patroni/</pre><p>13. Create systemd file for patroni on node1,node2,node3:</p><pre>alcalab@node1:~$ sudo vi  /etc/systemd/system/patroni.service<br><br># Add below lines<br>[Unit]<br>Description=Patroni Orchestration<br>After=syslog.target network.target<br>[Service]<br>Type=simple<br>User=postgres<br>Group=postgres<br>ExecStart=/usr/local/bin/patroni /etc/patroni.yml<br>KillMode=process<br>TimeoutSec=30<br>Restart=no<br>[Install]<br>WantedBy=multi-user.targ<br></pre><p>14. Start only patroni services on node1,node2,node3</p><pre>  # Create systemd file <br><br>alcalab@node1:~$ cat /etc/systemd/system/patroni.service<br>[Unit]<br>Description=Patroni Orchestration<br>After=syslog.target network.target<br>[Service]<br>Type=simple<br>User=postgres<br>Group=postgres<br>ExecStart=/usr/local/bin/patroni /etc/patroni.yml<br>KillMode=process<br>TimeoutSec=30<br>Restart=no<br>[Install]<br>WantedBy=multi-user.targ<br><br>alcalab@node1:~$ sudo systemctl daemon-reload<br>alcalab@node1:~$ sudo systemctl start patroni</pre><p>15. Add below lines in /etc/haproxy/haproxy.cfg on haproxy node:</p><pre>alcalab@haproxy:~$ vi /etc/haproxy/haproxy.cfg<br># Add lines to this config<br><br>global<br>      maxconn 100<br>defaults<br>      log global<br>      mode tcp<br>      retries 2<br>      timeout client 30m<br>      timeout connect 4s<br>      timeout server 30m<br>      timeout   check   5s<br>listen stats<br>      mode http<br>      bind *:7000<br>      stats enable<br>      stats uri /<br>listen postgres<br>      bind *:5000<br>      option httpchk<br>      http-check expect status 200<br>      default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>      server node1 10.252.55.129:5432 maxconn 100   check   port 8008<br>      server node2 10.252.54.125:5432 maxconn 100   check   port 8008<br>      server node3 10.252.54.87:5432 maxconn 100   check   port 8008<br><br></pre><p>16. Restart haproxy service</p><pre>alcalab@haproxy:~$ sudo systemctl restart haproxy</pre><p>17. Test from url http://&lt;ha_proxy&gt;:7000</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_2LxKuMQrLT6GKtGObizw.png" /></figure><p>18. Active node is node1 , let’s stop patroni service on node1 and check failover:</p><pre>alcalab@node1:~$ sudo systemctl stop patroni<br><br># Check node2 patroni service:<br><br>alcalab@node2:~$ sudo systemctl status patroni<br>● patroni.service - Patroni Orchestration<br>     Loaded: loaded (/etc/systemd/system/patroni.service; disabled; vendor preset: enabled)<br>     Active: active (running) since Fri 2023-11-17 17:12:23 UTC; 4min 58s ago<br>   Main PID: 4958 (patroni)<br>      Tasks: 16 (limit: 2219)<br>     Memory: 179.1M<br>        CPU: 1.709s<br>     CGroup: /system.slice/patroni.service<br>             ├─4958 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni.yml<br>             ├─4990 postgres -D /data/patroni --config-file=/data/patroni/postgresql.conf --listen_addresses=10.252.54.125 --port=5432 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=&gt;<br>             ├─4997 &quot;postgres: postgres: checkpointer &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─4998 &quot;postgres: postgres: background writer &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&gt;<br>             ├─4999 &quot;postgres: postgres: stats collector &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5006 &quot;postgres: postgres: postgres postgres 10.252.54.125(51052) idle&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5009 &quot;postgres: postgres: postgres postgres 10.252.54.125(51060) idle&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5155 &quot;postgres: postgres: walwriter &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5156 &quot;postgres: postgres: autovacuum launcher &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot;&gt;<br>             ├─5157 &quot;postgres: postgres: logical replication launcher &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot;&gt;<br>             └─5165 &quot;postgres: postgres: walsender replicator 10.252.54.87(50028) streaming 0/50003B8&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&gt;<br><br>Nov 17 17:16:57 node2 patroni[4996]: 2023-11-17 17:16:57.872 UTC [4996] LOG:  received promote request<br>Nov 17 17:16:57 node2 patroni[4996]: 2023-11-17 17:16:57.872 UTC [4996] LOG:  redo done at 0/5000228 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 270.71 s<br>Nov 17 17:16:57 node2 patroni[4996]: 2023-11-17 17:16:57.883 UTC [4996] LOG:  selected new timeline ID: 4<br>Nov 17 17:16:58 node2 patroni[5154]: 2023-11-17 17:16:58.097 UTC [5154] ERROR:  replication slot &quot;node3&quot; does not exist<br>Nov 17 17:16:58 node2 patroni[5154]: 2023-11-17 17:16:58.097 UTC [5154] STATEMENT:  START_REPLICATION SLOT &quot;node3&quot; 0/5000000 TIMELINE 3<br>Nov 17 17:16:58 node2 patroni[4996]: 2023-11-17 17:16:58.323 UTC [4996] LOG:  archive recovery complete<br>Nov 17 17:16:58 node2 patroni[4990]: 2023-11-17 17:16:58.400 UTC [4990] LOG:  database system is ready to accept connections<br>Nov 17 17:16:59 node2 patroni[4958]: 2023-11-17 17:16:59,442 INFO: no action. I am (node2), the leader with the lock<br>Nov 17 17:17:08 node2 patroni[4958]: 2023-11-17 17:17:08,895 INFO: no action. I am (node2), the leader with the lock<br>Nov 17 17:17:18 node2 patroni[4958]: 2023-11-17 17:17:18,933 INFO: no action. I am (node2), the leader with the lock<br><br># Check node3<br><br>alcalab@node3:~$ sudo systemctl status patroni<br>● patroni.service - Patroni Orchestration<br>     Loaded: loaded (/etc/systemd/system/patroni.service; disabled; vendor preset: enabled)<br>     Active: active (running) since Fri 2023-11-17 17:14:39 UTC; 3min 23s ago<br>   Main PID: 5405 (patroni)<br>      Tasks: 14 (limit: 2220)<br>     Memory: 82.7M<br>        CPU: 2.015s<br>     CGroup: /system.slice/patroni.service<br>             ├─5405 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni.yml<br>             ├─5437 postgres -D /data/patroni --config-file=/data/patroni/postgresql.conf --listen_addresses=10.252.54.87 --port=5432 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=1&gt;<br>             ├─5439 &quot;postgres: postgres: startup recovering 000000040000000000000005&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5443 &quot;postgres: postgres: checkpointer &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5444 &quot;postgres: postgres: background writer &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&gt;<br>             ├─5445 &quot;postgres: postgres: stats collector &quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br>             ├─5455 &quot;postgres: postgres: postgres postgres 10.252.54.87(46316) idle&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&gt;<br>             ├─5459 &quot;postgres: postgres: postgres postgres 10.252.54.87(46326) idle&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&gt;<br>             └─5558 &quot;postgres: postgres: walreceiver streaming 0/50003B8&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &gt;<br><br>Nov 17 17:17:03 node3 patroni[5558]: 2023-11-17 17:17:03.466 UTC [5558] LOG:  replication terminated by primary server<br>Nov 17 17:17:03 node3 patroni[5558]: 2023-11-17 17:17:03.466 UTC [5558] DETAIL:  End of WAL reached on timeline 3 at 0/50002A0.<br>Nov 17 17:17:03 node3 patroni[5439]: 2023-11-17 17:17:03.467 UTC [5439] LOG:  new target timeline is 4<br>Nov 17 17:17:03 node3 patroni[5558]: 2023-11-17 17:17:03.469 UTC [5558] LOG:  restarted WAL streaming at 0/5000000 on timeline 4<br>Nov 17 17:17:09 node3 patroni[5405]: 2023-11-17 17:17:09,252 INFO: no action. I am (node3), a secondary, and following a leader (node2)<br>Nov 17 17:17:19 node3 patroni[5405]: 2023-11-17 17:17:19,284 INFO: no action. I am (node3), a secondary, and following a leader (node2)<br>Nov 17 17:17:29 node3 patroni[5405]: 2023-11-17 17:17:29,251 INFO: no action. I am (node3), a secondary, and following a leader (node2)<br>Nov 17 17:17:39 node3 patroni[5405]: 2023-11-17 17:17:39,262 INFO: no action. I am (node3), a secondary, and following a leader (node2)<br>Nov 17 17:17:49 node3 patroni[5405]: 2023-11-17 17:17:49,249 INFO: no action. I am (node3), a secondary, and following a leader (node2)<br>Nov 17 17:17:59 node3 patroni[5405]: 2023-11-17 17:17:59,249 INFO: no action. I am (node3), a secondary, and following a leader (node2)</pre><p>19. Now the new leader is node2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*baQ5LS8wXwXUmvSMT6-K_g.png" /></figure><p>20. Start patroni service on node1 and check with patroni list command:</p><pre>alcalab@node1:~$ sudo systemctl start patroni<br><br>alcalab@node1:~$ patronictl -c /etc/patroni.yml list<br>+ Cluster: postgres (7302469623088403237) -----+----+-----------+<br>| Member | Host          | Role    | State     | TL | Lag in MB |<br>+--------+---------------+---------+-----------+----+-----------+<br>| node1  | 10.252.55.129 | Replica | streaming |  4 |         0 |<br>| node2  | 10.252.54.125 | Leader  | running   |  4 |           |<br>| node3  | 10.252.54.87  | Replica | streaming |  4 |         0 |<br>+--------+---------------+---------+-----------+----+-----------+</pre><p>21. Connect postgres database from haproxy:</p><pre>alcalab@node1:~$ psql -h haproxy -p 5000 -U postgres<br>Password for user postgres:<br>psql (14.9 (Ubuntu 14.9-0ubuntu0.22.04.1))<br>Type &quot;help&quot; for help.<br><br>postgres=# \dt;<br>Did not find any relations.<br>postgres=# \l<br>                                  List of databases<br>   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges<br>-----------+----------+----------+-------------+-------------+-----------------------<br> postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |<br> template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +<br>           |          |          |             |             | postgres=CTc/postgres<br> template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +<br>           |          |          |             |             | postgres=CTc/postgres<br><br></pre><p>22. Add cluster postgress on pgadmin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ya_LeiV-nnGb-gzC5sTnUQ.png" /></figure><p>23. Initiate a switchover, which gracefully switches the roles of the current primary and a selected secondary. This is a controlled operation</p><pre>alcalab@node1:~$ patronictl -c /etc/patroni.yml switchover<br>Current cluster topology<br>+ Cluster: postgres (7302469623088403237) -----+----+-----------+<br>| Member | Host          | Role    | State     | TL | Lag in MB |<br>+--------+---------------+---------+-----------+----+-----------+<br>| node1  | 10.252.55.129 | Replica | streaming |  4 |         0 |<br>| node2  | 10.252.54.125 | Leader  | running   |  4 |           |<br>| node3  | 10.252.54.87  | Replica | streaming |  4 |         0 |<br>+--------+---------------+---------+-----------+----+-----------+<br>Primary [node2]:<br>Candidate [&#39;node1&#39;, &#39;node3&#39;] []: node1<br>When should the switchover take place (e.g. 2023-11-17T18:43 )  [now]:<br>Are you sure you want to switchover cluster postgres, demoting current leader node2? [y/N]: <br>y<br><br>alcalab@node1:~$ patronictl -c /etc/patroni.yml list<br>+ Cluster: postgres (7302469623088403237) -----+----+-----------+<br>| Member | Host          | Role    | State     | TL | Lag in MB |<br>+--------+---------------+---------+-----------+----+-----------+<br>| node1  | 10.252.55.129 | Leader  | running   |  5 |           |<br>| node2  | 10.252.54.125 | Replica | streaming |  5 |         0 |<br>| node3  | 10.252.54.87  | Replica | streaming |  5 |         0 |<br>+--------+---------------+---------+-----------+----+-----------+</pre><h4>CONCLUSION:</h4><p>This is a simplified guide, and the specific details may vary based on your operating system, PostgreSQL version, and the tools you choose.</p><p>Patroni is primarily responsible for managing the high availability aspects of PostgreSQL clusters, including automated failover and promotion of new primary nodes. However, Patroni itself does not handle logical replication directly. Logical replication is a feature provided by PostgreSQL itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a4348fca444" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>