<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yunus emre on Medium]]></title>
        <description><![CDATA[Stories by Yunus emre on Medium]]></description>
        <link>https://medium.com/@ygormus173?source=rss-46bc7b096d2e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t_yz51M64lPC-phQodUbjw.jpeg</url>
            <title>Stories by Yunus emre on Medium</title>
            <link>https://medium.com/@ygormus173?source=rss-46bc7b096d2e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:16:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ygormus173/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Redis at Trendyol: The Evolution from Sentinel to Native Cluster Architecture]]></title>
            <link>https://medium.com/trendyol-tech/scaling-redis-at-trendyol-the-evolution-from-sentinel-to-native-cluster-architecture-939c424b5c5c?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/939c424b5c5c</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[redis-cluster]]></category>
            <category><![CDATA[stretch-redis]]></category>
            <category><![CDATA[redis-sentinel]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 07:37:23 GMT</pubDate>
            <atom:updated>2026-07-10T06:33:22.771Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>“Scale” is not a buzzword at Trendyol, it is our everyday life. Our systems power one of the largest e-commerce platforms, handling millions of requests per minute with extremely low latency, high throughput and consistent availability. Our caching database layer is a critical part of how we achieve these metrics.<br>In this article, we will discuss the main differences between a Sentinel-managed Redis architecture and a Native Redis Cluster approach, and walk you through our journey of migrating from the former to the latter to support our ever-expanding workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/0*sXGi0PKOhDQgLToP" /></figure><p>In this article, we’ll discuss key points between sentinel managed redis cluster vs native redis cluster method.</p><p><strong><em>Issue</em></strong></p><p>As your requests(concurrent and multi-app configured) is enlarging, infra systems need to response. The problematic here is request types; read and writes. For the reads, you can add one more slave but what if your inserts going higher, absolute bottleneck. Redis manages writes via master roles, so sentinel stucks at one master and its followers (aka slaves or replicas).</p><p>Very briefly, we can consider sentinel as an extra layer of cluster management of redis service. It depends on some set of rules likely postgresql as one node is master(let’s say primary) and has slaves(replicas). Master asynchronously keeps identical it’s slaves connected to itself. If master is unreachable for any reason, one of the eligible slaves get voted for leadership and become master. Actually that’s all with some important keynotes below.</p><p><strong><em>How It Works</em></strong></p><ol><li><strong>Master-Slave Replication</strong>: All data is replicated from master to all slaves</li><li><strong>Sentinel Monitoring</strong>: Sentinel processes monitor the health of Redis nodes</li><li><strong>Quorum</strong>: It is the minimum number of Sentinel nodes that must agree a master node is unreachable. <br> — An odd number of total sentinel nodes is best practise to set healthy quorum. Generally (n-1)/2 +1 where n is total number of nodes.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K0evCWjd5tBMTHSuB3rhLQ.png" /><figcaption>Seninel Architecture</figcaption></figure><h4>Native Clustering</h4><p><strong><em>How It Works</em></strong></p><ol><li><strong>Hash Slot Partitioning</strong>: The keyspace is divided into 16,384 hash slots</li><li><strong>Key Distribution</strong>: Each key is assigned to a slot using CRC16(key) mod 16384</li><li><strong>Built-in Failover</strong>: If a master fails, its replica is promoted automatically</li><li><strong>Gossip Protocol</strong>: Nodes communicate via cluster bus (port 16379)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSSjlzIcjtnEoAKvxgDQ6g.png" /><figcaption>Base Redis-Cluster Architecture</figcaption></figure><h4>Comparison Table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*2PODS2oGpQ1lG2g-ftAWgw.png" /><figcaption>Comparison Table between Sentinel-Cluster</figcaption></figure><h4>Comparison with another NoSQL database Couchbase</h4><p><em>Total 7 data nodes &lt;- 2xlarge (16CPU-32GB RAM) flavor Couchbase Cluster</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loHx6-yxAD5kaFZ3pnsBuQ.png" /></figure><p><em>Total 9 nodes &lt;- medium (4CPU-16GBRAM) flavor<br>3 master 6 replica</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*P-Xusn29R0MLRZAZa6nPSg.png" /></figure><p><strong>Load Test Metrics Comparison</strong></p><p>The key performance metrics from the load tests comparing the Redis Cluster (Playground) configuration with the existing Couchbase (Production) setup are summarized in the referenced spreadsheet.</p><p>In brief:</p><ul><li><strong>Latency (p99/p90/p50):</strong> Despite a slight difference, with Redis at (114/22.9/6.4 ms) and Couchbase at (108/21.8/4.6 ms), the latency performance of the Redis cluster remains strong and comparable to the production Couchbase environment, especially considering the smaller allocated resources for Redis.</li><li><strong>Throughput:</strong> Couchbase maintained higher throughput (340k req/m) than Redis (280k req/m).</li><li><strong>Error Rate:</strong> Redis had a 0% error rate, while Couchbase showed a negligible 0.004%.</li></ul><p><strong>Cache Hit Rate:</strong> Both systems performed comparably, with Redis at 70% and Couchbase at 70.2%.</p><blockquote>We’ve compared with one of the most scalable databases Couchbase because compared to Redis-Sentinel, throughput only climbed to 70k req/m.</blockquote><h4>When to Use Which</h4><p>✅ Your dataset is <strong>too large for a single node</strong><br> ✅ You need <strong>horizontal scaling</strong> for writes<br> ✅ You have <strong>high throughput requirements</strong><br> ✅ Your application can handle <strong>cluster-aware clients</strong><br> ✅ You don’t need many multi-key operations across different slots</p><p><strong><em>Choose Redis Sentinel When:</em></strong></p><p>✅ Your dataset <strong>fits on a single node</strong><br> ✅ You need <strong>simple high availability</strong><br> ✅ Your application uses <strong>standard Redis clients</strong><br> ✅ You need full <strong>multi-key operation support</strong><br> ✅ You prefer <strong>simpler operational complexity</strong><br> ✅ Read scaling via replicas is sufficient</p><h4>Summary</h4><p><strong><em>Bottom Line:</em></strong></p><ul><li><strong>Redis Cluster</strong> = When you need to <strong>scale beyond one node</strong></li><li><strong>Redis Sentinel</strong> = When you need <strong>HA for a single dataset</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*6RZT6tUFlT_yh1ppuuZVSg.png" /><figcaption>Summary</figcaption></figure><h3><em>How Trendyol Manages Redis at Scale Across 3 Availability Zones</em></h3><h4>Stretch Strategy</h4><p>A <strong>Stretched Redis Cluster</strong> is a Redis Cluster deployment that spans <strong>multiple data centers or availability zones</strong>. This provides:</p><ul><li>🛡️ <strong>Disaster Recovery</strong> — If one datacenter goes down, data survives</li><li>⚡ <strong>Low Latency</strong> — Local reads from nearby replicas</li><li>🔄 <strong>Automatic Failover</strong> — Masters fail over to replicas in healthy zones</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*CXLvsyeZ_H_Y8gfUitD-_w.png" /><figcaption>Master Nodes Matrix</figcaption></figure><h4>Architecture Overview (Maximum HA Topology)</h4><p><strong>Key Design: Masters distributed across ALL 3 locations</strong> — No single point of failure!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExsXxAJuW64UougYheEv-Q.png" /><figcaption>HA Stretch Architecture</figcaption></figure><ul><li>So ,what does that mean and what let us gain?</li><li>&lt;- Each availability zone has only one master to prevent manual interventions upon any cluster fail detection.</li><li>&lt;- Split write requests between availability zones for better performance and reliably network transmissions. Also having likely active-active connection ability.</li></ul><blockquote>Wait, what if any failover triggers and balanced matrix breaks up? In this kind of large inventory pool, that’s very possible to have some node unreachable or restart required operations on infra or service level. So we deploy inside terraform module a <em>cluster-zone-checker.py </em>by a 5 minute interval cronjob to check and run failover to balance inter-az sections.</blockquote><p>Here is example screenshot after dummy service interruption. If any node auto-failed over and as in example mars-az2 has more than one master, a promote triggered to balance master node formations across cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BHSr7pxkDM9gGgc4QUiEg.png" /><figcaption>Promote a node to balance inter-az sections</figcaption></figure><h4>Scalability</h4><p>In short if there is no concern to get running on single dc, we provide scalability for redis cluster. In a seperate module version within unique branch we have such tasks added.</p><ul><li>Terraform validation for odd number counts to ensure better quorum elections. √</li><li>Redis deploys automatically keys to new master-replica combines equally. √</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*L4_NlkRH8Vz2wF3THzhaKg.png" /><figcaption>rebalance</figcaption></figure><p>Here is how:</p><pre>- name: Add new master nodes to existing cluster<br>      shell: &gt;<br>        redis-cli --cluster add-node<br>        {{ hostvars[item].ansible_host }}:{{ redis_port }}<br>        {{ first_master_ip }}:{{ redis_port }}<br>        --cluster-yes<br>        --user {{ redis_auth_user }}<br>        -a {{ redis_auth_pass }}<br>      loop: &quot;{{ new_master_hosts_list | default([]) }}&quot;<br>      run_once: true<br>      when:<br>        - cluster_exists | bool<br>        - new_master_hosts_list | default([]) | length &gt; 0<br><br>    - name: Rebalance hash slots across all masters (distributes slots to new empty masters)<br>      shell: &gt;<br>        redis-cli --cluster rebalance<br>        {{ first_master_ip }}:{{ redis_port }}<br>        --cluster-yes<br>        --cluster-use-empty-masters<br>        --user {{ redis_auth_user }}<br>        -a {{ redis_auth_pass }}<br>      run_once: true<br>      register: rebalance_result<br>      when:<br>        - cluster_exists | bool<br>        - new_master_hosts_list | default([]) | length &gt; 0</pre><pre># --- MASTERS ---<br>[redis_masters]<br>%{ for node in redis_masters ~}<br>${node.name} ansible_host=${node.ip} redis_role=${node.redis_role} dc=${node.dc} az=${node.az}<br>%{ endfor ~}<br><br># --- REPLICAS<br>[redis_replicas]<br>%{ for node in redis_replicas ~}<br>${node.name} ansible_host=${node.ip} redis_role=${node.redis_role} dc=${node.dc} az=${node.az} replica_of=${node.replica_of}<br>%{ endfor ~}<br><br>[redis_nodes:children]<br>redis_masters<br>redis_replicas</pre><blockquote>As you can see above a part of our dynamic inventory template, each created node pair follow as their master-replica match for healthy rebalance and scale out policy. More clearly, <br>[M1,R1, M2,R2, M3,R3] → scale to 5 → [M1,R1, M2,R2, M3,R3, M4,R4, M5,R5].</blockquote><h3>Conclusion</h3><p>At Trendyol, managing extreme traffic spikes and maintaining system reliability is non-negotiable. While Redis Sentinel provided a solid foundation for simple, highly available caching, its single-master limitation became a strict bottleneck.</p><p>By evolving to a Native Redis Cluster Architecture, we shattered this ceiling, achieving scalable requests per minute with sub-millisecond median latencies. Coupling this native partitioning with our 3-Availability Zone “Stretch Strategy” and automated failover scripts, we have built a resilient, scalable, and self-healing data layer.</p><p>I would like to thank my colleagues on <strong><em>datastore nosql</em></strong> team who couraged me in this journey, <strong><em>SRE</em></strong> <strong><em>team</em></strong> for helping us for better data reliability as well as <strong><em>development staff </em></strong>for load and chaos tests scenarios.</p><p>If you are interested in this world, you can join Trendyol’s rich technology world.</p><p><a href="https://careers.trendyol.com/">https://careers.trendyol.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=939c424b5c5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/scaling-redis-at-trendyol-the-evolution-from-sentinel-to-native-cluster-architecture-939c424b5c5c">Scaling Redis at Trendyol: The Evolution from Sentinel to Native Cluster Architecture</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Way to MultiAZ-Stretch Redis]]></title>
            <link>https://medium.com/trendyol-tech/way-to-multiaz-stretch-redis-2f2a0eafe45a?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f2a0eafe45a</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[redis-sentinel]]></category>
            <category><![CDATA[terraform-modules]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 12:19:14 GMT</pubDate>
            <atom:updated>2025-02-19T06:25:19.843Z</atom:updated>
            <content:encoded><![CDATA[<p>We are at trendyol running our services in different zones as expected to have availability. Not going too much detailed but in each of three datacenter we have two provider ( Openstack and VMWare also public cloud solutions). So doesn’t matter the hypervisor type, we need to enable our infra codes suitable for both. So let’s begin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*xIu8dfmMkJOpm3ub.png" /><figcaption>Redis-on-Terraform</figcaption></figure><p>If you need to work in a memory persistent key-value database, <a href="https://redis.io/docs/latest/operate/rc/rc-quickstart/">redis</a> is the one of the most used one; most probably the eldest as well. But here is the issue which you need to copy your keys in realtime or nearly realtime (as we are using community edition which enterprise has that feature) just like Couchbase XDCR methodology. So without needing our developers feeds the data upon any problematic on datacenter/infra side, we needed to hire a new way to manage our clusters and of course multiAZ adoptation.</p><p>Firstly, we were using <a href="https://github.com/DavidWittman/ansible-redis?tab=readme-ov-file#installation">ansible role</a> for setting up redis (mostly on a 3 memory optimized virtual machine) clusters with sentinel services as well.<br>Not: prometheus exporter has been used for observability within its <a href="https://galaxy.ansible.com/ui/standalone/roles/mesaguy/prometheus/">roles</a> as well. These roles and of course ansible tasks are specialized for trendyol requirements.</p><h3>So how can we set up a stretch redis cluster?</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FIrdFKldNXQ6SA%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F1965-wile-e-coyote-chaser-on-the-rocks-IrdFKldNXQ6SA&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExMTE2ZjJ0YnZtMTR4MDlvc2gxanlrdXRhbGpvbnl0ZmhubXAzeGJqciZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FIrdFKldNXQ6SA%2Fgiphy.webp&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3efef6bcacae6a4decb7bf40782126b/href">https://medium.com/media/e3efef6bcacae6a4decb7bf40782126b/href</a></iframe><p>Very shortly, running vms on both DCs and merge the cluster in the inventory.</p><pre>[redis_master]<br>rds-p-test-ha-01-data-1-m ansible_host=IP1M<br>[redis_slave]<br>rds-p-test-ha-01-data-2-m ansible_host=IP2M<br>rds-p-test-ha-01-data-3-m ansible_host=IP3M<br>rds-p-test-ha-01-data-4-v ansible_host=IP2V<br>rds-p-test-ha-01-data-v ansible_host=IP2V<br>rds-p-test-ha-01-data-6-v ansible_host=IP3V<br>[redis_sentinel]<br>rds-p-test-ha-01-data-1-m  ansible_host=IP1M<br>rds-p-test-ha-01-data-2-m ansible_host=IP2M<br>rds-p-test-ha-01-data-m ansible_host=IP3M<br>rds-p-test-ha-01-data-v ansible_host=1IP1V<br>rds-p-test-ha-01-data-v ansible_host=IP2V<br>rds-p-test-ha-01-data-v ansible_host=IP3V<br><br>[all:vars]<br>ansible_connection=ssh<br>ansible_python_interpreter=/usr/bin/python3<br>ansible_os_family=Debian<br>redis_port=6379<br>redis_pidfile=/run/redis/{{ redis_port }}.pid<br>redis_sentinel_port=26379<br>redis_sentinel_pidfile=/run/redis/sentinel_{{ redis_sentinel_port }}.pid<br>redis_nofile_limit=65535<br># redis_save is the How frequently to snapshot the database to disk<br># by default role variable it is like below but you can set it empty for performance reasons<br># redis_save=[]<br>redis_save=[&#39;900 1&#39;, &#39;300 10&#39;, &#39;60 10000&#39;]<br>redis_version=7.2.4<br>redis_sentinel_logfile=/var/log/redis/sentinel.log<br>redis_logfile=/var/log/redis/redis.log<br>redis_maxmemory_policy=volatile-ttl</pre><p>Ok, it works. But for us we would like to manage it via terraform.</p><p><strong><em>Before starts, we already set up redis-terraform infrastructure. Here we focused on multi-az enabling features and examples are explained on openstack modules but does not change the logic ever.</em></strong></p><p>Firstly we developed our redis-node.tf</p><pre>module &quot;redis_node_zone_1&quot; {<br>  source = &quot;git::ssh://git@gitlab.com:/xyz/yzx.git//modules/openstack?ref=x.y.z&quot;<br><br>  providers = {<br>    openstack = openstack.zone_1<br>  }<br><br>  availability_zone      = var.zone_1_az_name<br>  network_name           = var.zone_1_network_name<br>  image                  = var.image<br>  key_pair_name          = var.key_pair_name<br>  security_groups        = var.security_groups<br>  server_groups_policies = var.server_groups_policies<br><br>  nodes_count = var.redis_nodes_count<br>  flavor_name = var.redis_node_flavor_name<br><br>  instance_tags = merge(local.redis_node_instance_tags, {<br>    zone = var.zone_1_name<br>  })<br>}<br><br>module &quot;redis_node_zone_2&quot; {<br>  count  = var.multi_az_enabled ? 1 : 0<br>  source = &quot;git::ssh://git@gitlab.com:/xyz/yzx.git//modules/openstack?ref=x.y.z&quot;<br><br>  providers = {<br>    openstack = openstack.zone_2<br>  }<br><br>  availability_zone      = var.zone_2_az_name<br>  network_name           = var.zone_2_network_name<br>  image                  = var.image<br>  key_pair_name          = var.key_pair_name<br>  security_groups        = var.security_groups<br>  server_groups_policies = var.server_groups_policies<br><br>  nodes_count = var.redis_nodes_count<br>  flavor_name = var.redis_node_flavor_name<br><br>  instance_tags = merge(local.redis_node_instance_tags, {<br>    zone = var.zone_2_name<br>  })<br>}</pre><h4>Explanation:</h4><p>module <em>‘redis_node_zone_1/2 (</em><strong><em>which we have node_3 as well</em></strong><em>) </em>are where we set up our virtual machines.<br>But here the important point:</p><pre>module &quot;redis_node_zone_2&quot; {<br>  count  = var.multi_az_enabled ? 1 : 0</pre><p>we are using <a href="https://developer.hashicorp.com/terraform/language/modules/develop/providers#implicit-provider-inheritance">terraform provider alias</a> and if the <strong><em>multi_az_enabled</em></strong> is true the zone<em>_2 </em>module is being activated.</p><p>Secondly we aggregating our vms within our inventory such below:</p><pre>resource &quot;local_file&quot; &quot;ansible_inventory&quot; {<br>  count    = var.redis_nodes_count == 0 ? 0 : 1<br>  filename = local.inventory_file_name<br>  content = templatefile(&quot;${path.module}/../../files/templates/inventory.tpl&quot;, {<br>    redis_nodes = local.redis_nodes<br>  })<br>}</pre><p>Let’s see how we dynamically manage our inventory template:</p><pre>[redis_master]<br>%{ if length(redis_nodes) &gt; 0 ~}<br>${redis_nodes[0].name} ansible_host=${redis_nodes[0].ip}<br>%{ endif ~}<br><br>[redis_slave]<br>%{ if length(redis_nodes) &gt; 1 ~}<br>%{ for i in range(1, length(redis_nodes)) ~}<br>${redis_nodes[i].name} ansible_host=${redis_nodes[i].ip}<br>%{ endfor ~}<br>%{ endif ~}<br><br>[redis_sentinel]<br>%{ if length(redis_nodes) &gt; 0 ~}<br>%{ for i, node in redis_nodes ~}<br>${redis_nodes[i].name} ansible_host=${redis_nodes[i].ip}<br>%{ endfor ~}<br>%{ endif ~}<br><br>[all:vars]<br>ansible_connection=ssh<br><br>redis_port=6379<br>redis_pidfile=/run/redis/{{ redis_port }}.pid<br>redis_sentinel_port=26379<br>redis_sentinel_pidfile=/run/redis/sentinel_{{ redis_sentinel_port }}.pid<br>redis_sentinel_logfile=/var/log/redis/sentinel.log<br>redis_logfile=/var/log/redis/redis.log<br><br>redis_nofile_limit=65535<br>redis_save=[&#39;900 1&#39;, &#39;300 10&#39;, &#39;60 10000&#39;]<br>redis_maxmemory_policy=volatile-ttl</pre><h4>Setting up cluster:</h4><pre>resource &quot;null_resource&quot; &quot;cluster&quot; {<br><br>  triggers = {<br>    redis_nodes                = join(&quot;\n&quot;, formatlist(&quot;%s&quot;, local.redis_nodes[*].name))<br>    tf_reapply_config_revision = var.tf_reapply_config_revision<br>  }<br>  provisioner &quot;local-exec&quot; {<br>    command     = file(format(&quot;%s/../../files/scripts/cluster_formation.sh&quot;, path.module))<br>    interpreter = [&quot;bash&quot;, &quot;-c&quot;]<br>    environment = {<br>      MODULE_PATH        = path.module<br>      INVENTORY_FILENAME = local.inventory_file_name<br>      REDIS_NODE_COUNT   = var.redis_nodes_count<br>      REDIS_VERSION      = var.redis_version<br>    }<br>  }<br><br>  depends_on = [<br>    module.redis_node_zone_1.servers,<br>    module.redis_node_zone_2.servers,<br>  ]<br>}</pre><ul><li><em>locals.tf</em> to creating tags, naming conventions ,merging &amp;structuring instances:</li></ul><pre>locals {<br>  module_tags = {<br>    app               = &quot;redis&quot;<br>    service           = &quot;redis&quot;<br>    tf_module_version = &quot;0.1.0&quot;<br>    image             = var.image<br>  }<br><br>  redis_cluster_name = join(&quot;-&quot;,<br>    compact([<br>      var.instance_tags.team,<br>      var.instance_tags.cluster,<br>    ])<br>  )<br><br>  inventory_file_name = format(&quot;%s-inventory.ini&quot;, local.redis_cluster_name)<br><br>  redis_node_instance_tags = merge(local.module_tags, var.instance_tags, {<br>    role = &quot;data&quot;<br>  })<br><br>  redis_nodes = concat(<br>    [<br>      for s in module.redis_node_zone_1.servers : {<br>        name = &quot;${s.name}-${var.zone_1_name}&quot;<br>        ip   = s.ip<br>        zone = var.zone_1_name<br>      }<br>    ],<br>    var.multi_zone_enabled ? [<br>      for s in(module.redis_node_zone_2[0].servers) : {<br>        name = &quot;${s.name}-${var.zone_2_name}&quot;<br>        ip   = s.ip<br>        zone = var.zone_2_name<br>      }<br>    ] : []<br>  )<br>}</pre><ul><li>example requirements.yml to call within terraform module without any error in gitlab-ci pipelines:</li></ul><pre>- src: git@gitlab.com:xyz/xyz/mesaguy.prometheus-ty.git<br>  name: mesaguy.prometheus-ty<br>  version: 0.0.2<br>  scm: git<br><br>- src: git@gitlab.com:xyz/xyz/davidwittman.redis-ty.git<br>  name: davidwittman.redis-ty<br>  version: 0.0.2<br>  scm: git</pre><p><em>Here we also add our zone_3 servers as well to concatenate.</em></p><ul><li>Lastly we would like to share our cluster_formation.sh</li></ul><pre>#!/usr/bin/env bash<br><br># shellcheck disable=SC2086<br>set -o errexit<br>set -o nounset<br>set -o pipefail<br><br>export MAX_COUNT=${MAX_COUNT-20}<br>export SLEEP_DURATION=${SLEEP_DURATION-30}<br>ANZIBLE_EXTRA_ARGS=&quot;--timeout 300 --fork 50&quot;<br><br><br>echo &quot;[+] Current dir: $(pwd)&quot;<br>echo &quot;[+] MODULE_PATH: $MODULE_PATH&quot;<br>echo &quot;[+] SSH_AUTH_SOCK: $SSH_AUTH_SOCK&quot;<br>echo &quot;[+] INVENTORY_FILENAME: ${INVENTORY_FILENAME}&quot;<br>echo &quot;[+] REDIS_NODE_COUNT: ${REDIS_NODE_COUNT}&quot;<br>echo &quot;[+] REDIS_VERSION: ${REDIS_VERSION}&quot;<br><br>echo &quot;[+] List all ssh identities&quot;<br>ssh-add -l || echo &#39;List ssh private keys&#39;<br><br>if [[ ${REDIS_NODE_COUNT} -gt 0 ]]; then<br> count=0<br> while true; do<br>  echo &quot;[+] Wait File: ${INVENTORY_FILENAME}&quot;<br>  [[ -f &quot;${INVENTORY_FILENAME}&quot; ]] &amp;&amp; echo &quot;Found it!&quot; &amp;&amp; break<br><br>  count=$((count + 1))<br>  echo &quot;[+] Retrying ...: &#39;$count&#39;&quot;<br>  [[ $count -eq &quot;$MAX_COUNT&quot; ]] &amp;&amp; echo &quot;Exceeded the retry count&quot; &amp;&amp; exit 1<br><br>  echo &quot;[+] Delaying ... : &#39;$SLEEP_DURATION&#39;&quot;<br>  sleep $SLEEP_DURATION<br> done<br><br> echo &quot;[+] Show content of &#39;inventory.ini&#39; file&quot;<br> cat &quot;${INVENTORY_FILENAME}&quot;<br><br> echo &quot;[+] Wait another ansible-galaxy processes&quot;<br> count=0<br> while true; do<br>  if ! pgrep ansible-galaxy &gt;/dev/null; then<br>   break<br>  fi<br><br>  count=$((count + 1))<br>  echo &quot;[+] Retrying ...: &#39;$count&#39;&quot;<br>  [[ $count -eq &quot;$MAX_COUNT&quot; ]] &amp;&amp; echo &quot;Exceeded the retry count&quot; &amp;&amp; exit 1<br><br>  echo &quot;[+] Delaying ... : &#39;5&#39;&quot;<br>  sleep 5<br> done<br><br> echo &quot;[+] Ansible Galaxy: Install requirements&quot;<br> ansible-galaxy install -r &quot;$MODULE_PATH/../../files/playbooks/requirements.yml&quot; -p ~/.ansible/roles/<br><br> if [[ -f &quot;$MODULE_PATH/../../files/playbooks/requirements.txt&quot; ]]; then<br>  echo &quot;[+] Pip: Install requirement.txt&quot;<br>  pip3 install -r &quot;$MODULE_PATH/../../files/playbooks/requirements.txt&quot;<br> fi<br><br> echo &quot;[+] Run ansible-playbook&quot;<br> ansible-playbook \<br>  -i &quot;${INVENTORY_FILENAME}&quot; \<br>  &quot;${MODULE_PATH}/../../files/playbooks/playbook.yml&quot; \<br>  ${ANZIBLE_EXTRA_ARGS} \<br>  --extra-vars &quot;redis_version=$REDIS_VERSION&quot;<br><br> echo &quot;[+] Done ansible playbook&quot;<br> echo &quot;[+] [Cluster] All done&quot;<br>else<br> echo &quot;No need to run any command because redis node count=0&quot;<br>fi</pre><p><strong><em>Very shortly it checks</em></strong>:</p><ul><li>vms are up and connectable via ssh</li><li>inventory is ready to run via ansible</li><li>redis-version and other environment variables are fetched etc.</li></ul><p>Last bu not least to share our example terraform structure to how we provisioning internally.</p><pre>module &quot;multiaz-test-01&quot; {<br>  source = &quot;git::ssh://git@gitlab.com:/xyz/xyz/iac/provisioning/terraform/redis.git//modules/openstack?ref=x.y.z&quot;<br>  providers = {<br>    openstack.zone_1 = openstack.zone1<br>    openstack.zone_2 = openstack.zone2<br>    openstack.zone_3 = openstack.zone3<br>  }<br><br>  instance_tags = {<br>    environment = &quot;production&quot;<br>    team        = &quot;your_tribe_name&quot;<br>    cluster     = &quot;required_master_auth&quot;<br>  }<br><br>  multi_az_enabled = true<br>  <br>  zone_1_network_name = &quot;network_name1&quot;<br>  zone_2_network_name = &quot;network_name2&quot;<br>  security_groups      = [&quot;sg_name&quot;]<br><br>  redis_nodes_count      = 3<br><br>#  redis_version = &quot;7.2.4&quot;<br>}<br><br>output &quot;multiaz-test-01_servers&quot; {<br>  description = &quot;multiaz-test-01 Servers&quot;<br>  value       = module.multiaz-test-01.servers<br>}</pre><p>we didn’t but better to talk about something that we manage our redis cluster name and password uniquely in ansible side.<br>We generate passwords randomly on master and copy to slaves and setting up config files.<br>Also define sentinel auth name by {{ tribe }} — {{terraform_cluster_name}} -{{environment}} to make it unique as we said.</p><p>I would like to thank my colleague <a href="https://medium.com/u/53076196975e">Fatih Sarhan</a> who couraged me in this learning journey.</p><p>If you are interested in this world, you can join Trendyol’s rich technology world.</p><p><a href="https://jobs.lever.co/trendyol?source=post_page-----b531d83fb37e--------------------------------">Trendyol jobs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f2a0eafe45a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/way-to-multiaz-stretch-redis-2f2a0eafe45a">Way to MultiAZ-Stretch Redis</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trendyol Fulfillment Android Instrumented Test CI]]></title>
            <link>https://medium.com/trendyol-tech/trendyol-fulfillment-android-instrumented-test-ci-b531d83fb37e?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b531d83fb37e</guid>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Tue, 20 Jun 2023 06:02:20 GMT</pubDate>
            <atom:updated>2023-06-20T06:02:20.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*k4PXE1ptDdYPQx9oLIKtxA.gif" /></figure><p>Hello! We’re <a href="https://medium.com/@tugceaktepe">Tuğçe</a> and Yunus Emre from<a href="https://medium.com/trendyol-tech"> Trendyol Tech</a>. We have prepared an article about our a long-term adventure of Instrumented Tests in GitLab pipeline.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F1oGXIfkeF4P4w3zPxk%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Flooneytunes-looney-tunes-wile-e-coyote-1oGXIfkeF4P4w3zPxk&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcTYxYW9zangxMm55Z2h0Y3RxdWdyYmJxOXBhOGo1NWVncTRsMm12cSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2F1oGXIfkeF4P4w3zPxk%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fe006de2a448f2736d99b5b8cfc4732/href">https://medium.com/media/6fe006de2a448f2736d99b5b8cfc4732/href</a></iframe><p>When a new feature will be released, we definitely check its effect on other screens before distribution. In this context, UI Test automations have a critical role. However, it takes huge effort to run all tests manually on a local machine every release time. So, it would be great to run these tests on our CI/CD pipeline before the changes merged to release branch.</p><p>For this reason, we decided to move automations to our GitLab pipeline.</p><h3><strong>How did this adventure start ?</strong></h3><p>We had 2 options to realize this desire:</p><ul><li>Creating a Docker image with SDK and emulator installed</li><li>Using a custom Linux runner</li></ul><h3>Option 1: Creating a Docker image with SDK and emulator installed</h3><p>First of all, since our current pipeline is running on a Docker image, at the beginning of each stage in the pipeline, all the SDKs and libraries that we need to build an Android app are downloaded from scratch. When we added the download of the necessary libraries for the installation and operation of the emulator, our development speed would be slowed as the uptime of our pipeline would increase. This was something we never wanted as it would increase our lead time. The scripts we’re talking about are like <a href="https://gist.github.com/illuzor/988385c493d3f7ed7193a6e3ce001a68">here</a>.</p><p>We built an image on Docker with the required tools about Android SDK and emulator installed and tried to run the gitlab pipeline through this docker.</p><p>You can see an example of one of our Docker file to get a view how we were keeping it. As you are about to see, we are embedding all the thing inside our image to shorten our pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c260c3f22d387be8d6869a279b194d64/href">https://medium.com/media/c260c3f22d387be8d6869a279b194d64/href</a></iframe><p>Then we tried to run emulator from gitlab following commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3dcea6d1f55a570827df57c3f908de9/href">https://medium.com/media/f3dcea6d1f55a570827df57c3f908de9/href</a></iframe><p>Then, everything started with the below CI pipeline error message as follows:</p><pre>KVM: entry failed, hardware error 0x8<br>ERROR | crashhandler_die: fatal: kvm_arch_handle_exit: hardware error<br>happened in KVM, aborting nowSo, what was the reason for this error ? Reason was simple but a bit tricky, android emulator was behaving just like running a virtual machine inside our runner. See gitlab runner linked url if you are not familiar with it.</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxThtaaVNrPdQ64U9vq%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fscare-looney-tunes-wile-e-coyote-xThtaaVNrPdQ64U9vq&amp;image=https%3A%2F%2Fmedia3.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcmhpMXJjcmthcXMzdWx6ZnR0Z3ZxdTg4eHcyczh2bHBoYnRiZmEyZiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FxThtaaVNrPdQ64U9vq%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/9736ef63c1819479c2bba59f38a66c02/href">https://medium.com/media/9736ef63c1819479c2bba59f38a66c02/href</a></iframe><p>So, what was the reason for this error ? Reason was simple but a bit tricky, Android emulator was behaving just like running a virtual machine inside our runner. See gitlab runner linked <a href="https://docs.gitlab.com/runner/">url</a> if you are not familiar with it.</p><p>After some research, it seems like we need to enable <a href="https://www.makeuseof.com/best-virtual-machine-linux-kvm-virtualbox-qemu-hyper-v/#:~:text=QEMU%2C%20short%20for%20Quick%20Emulator,on%20a%20single%20physical%20host."><strong>qemu-kvm</strong></a> intel based cores in our runner. So just a quick preview, our runners are running on another <a href="https://kubernetes.io/">Kubernetes</a> cluster which is more scalable and efficient because after your pipeline ends, you will not need to clean disk space or additional resources because runner pod will be gone.</p><p>Then, we tried to enable virtualization in our docker images with openjdk-11, java and all the tools the emulator needs but failed. See below to QEMU enabled Dockerfile and start.sh:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c117f636d4c4ab4ce45ec0d6809cc691/href">https://medium.com/media/c117f636d4c4ab4ce45ec0d6809cc691/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd4786e9df864886858bebb9289c80d1/href">https://medium.com/media/cd4786e9df864886858bebb9289c80d1/href</a></iframe><p>But nope! It did not work for us. So, it seems like we are making a mistake somewhere or as we researched with our valuable colleagues we need to run up a virtual machine itself and enable <a href="https://developer.android.com/studio/run/emulator-acceleration#accel-vm">KVM acceleration</a> there. Meanwhile, we discovered important information about VM acceleration restrictions on Android’s official developer <a href="https://developer.android.com/studio/run/emulator-acceleration#vm-accel-restrictions">site</a>. Facing this truth caused a serious loss of motivation for us 😭 The first option had failed…</p><p>Despite all the difficulties, we continued on the way we started. Because our motto is <a href="https://en.wikipedia.org/wiki/No_pain,_no_gain">“<strong>No pain No Gain”</strong></a>! 🏋‍♀</p><h3>Option 2: Using a custom Linux runner</h3><p>OK, we started research about the VM acceleration problem except for the Docker solution. Then, we tried to build an Ubuntu-22.04 VM (by Openstack (aka <a href="https://www.linux-kvm.org/page/Main_Page">KVM</a>) hypervisor infrastructure design) at first. At Trendyol, we are using different virtualization infrastructures such as VMWare or Openstack or even Hyper-V).</p><p>We’ve set-up a Gitlab-Runner and registered it to our project to run tests on Ubuntu VM.Yeah, it was working with the below config in our gitlab-ci.yml on self VM. (<a href="https://docs.gitlab.com/runner/register/#linux">Register a Runner on Linux</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*emU2jpN0I6E9_Veo" /></figure><p>Anyway, we still get the error even though the QEMU tests and <a href="http://how%20to%20test%20kvm%20acceleration%20enabled/">KVM acceleration</a> tests results were OK. There must be something else on the infra side, so we switched to the <strong>VMware</strong> solution to see its behavior.</p><p>Not to forget, here our developer friends from the <strong>Trendyol Android Platform</strong> team let us be aware of this issue, thanks to them. So, as you can see below screenshot, VMWare gives us the opportunity to enable KVM acceleration on its hypervisor interface. Yeah, benefits of working and getting approaches to different technologies.🤩</p><p>Meanwhile<em>, w</em>e observed the behavior of the emulator and seemed like it was running on a single core so we set up 2 core per each 16 socket (Default one per socket). Then we activated CPU acceleration by enabling <strong>Expose hardware-assisted CPU virtualization to guest OS</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e88fnr9gXLyvgo4Y" /></figure><p><strong>Not long now! Fasten your belts!</strong></p><p>You may estimate the rest. We expanded the disk on the VM because emulator images take up a lot of space.</p><p>Then, we got permissions to add global variables (such as ANDROID_COMPILE_SDK, EMULATOR_VERSION etc.) to <strong>.bashrc </strong>to abondonnate them from our gitlab-ci.yml in <strong>before-script</strong> sections. So we cut the long and one-shot parts from out gitlab-ci.yml file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6d7d293d9083fe3fddd81a5b6d4bb10/href">https://medium.com/media/a6d7d293d9083fe3fddd81a5b6d4bb10/href</a></iframe><p>But we still get an error differently about root user permissions. Because the emulator desires to be working with some super-user privileges. Ok, we may edit the config.toml as the screenshot below to give root permissions to gitlab-runner user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-ThriOEJbMGhm10Y" /></figure><p>But this may cause some vulnerabilites so we decided another simple way, making a script file (such as <strong>android_sdk.sh</strong>), then copy it to the /root/ directory to run some #apt-get and similar commands to update and upgrade packages in each step we run the emulator and let it flow. 🔥</p><p><strong>Let’s go!</strong></p><p>If your Linux runner is ready for VM acceleration, the following operations must be performed on your VM.</p><ul><li>Download the required libraries for Android SDK. (one-time, if an update needs you can run again)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7dd2c7dd5feb4e34875ac76fae7661b3/href">https://medium.com/media/7dd2c7dd5feb4e34875ac76fae7661b3/href</a></iframe><ul><li>Download the required image to create the emulator you want to test (one-time for related emulator)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6a9296857b0cc0a986781f4c3a372a1/href">https://medium.com/media/e6a9296857b0cc0a986781f4c3a372a1/href</a></iframe><ul><li>Turning off animations for the emulator to run properly.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a584e44e19aaad4d03e4ac55bc7435fd/href">https://medium.com/media/a584e44e19aaad4d03e4ac55bc7435fd/href</a></iframe><ul><li>Create and launch the emulator via command line.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3668615c31256c611e2356beb60a64c/href">https://medium.com/media/e3668615c31256c611e2356beb60a64c/href</a></iframe><ul><li>You can use <a href="https://developer.android.com/studio/run/emulator-commandline#startup-options">these</a> parameters to speed up your emulator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N9q2XIsDRNdqHpWy" /></figure><ul><li>Run the ui test automation with gradle command on GitLab pipeline.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e6e262cc145efeca9a0956c59e82794/href">https://medium.com/media/3e6e262cc145efeca9a0956c59e82794/href</a></iframe><ul><li>TA DAA!!! Here <strong>connectedMockDebugAndroidTest</strong> task is our instrumentation test task in gradle.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U_G8-R_oXBIWP82r" /></figure><p><strong>Learnings..</strong></p><p>While unit tests can be run on a shared runner on Gitlab, the instrumentation tests require hardware acceleration which is not available on shared runners. So, the second option above must be preferred. You need to configure your own machine as a runner that Gitlab can use to run your UI tests!</p><p><strong>REFERENCES:</strong></p><ul><li><a href="https://developer.android.com/studio/run/emulator-acceleration#vm-accel-restrictions"><strong>https://developer.android.com/studio/run/emulator-acceleration#vm-accel-restrictions</strong></a></li><li><a href="https://developer.android.com/studio/run/emulator-commandline#common"><strong>https://developer.android.com/studio/run/emulator-commandline#common</strong></a></li><li><a href="https://developer.android.com/tools/adb"><strong>https://developer.android.com/tools/adb</strong></a></li><li><a href="https://www.redhat.com/en/topics/virtualization/what-is-a-hypervisor"><strong>https://www.redhat.com/en/topics/virtualization/what-is-a-hypervisor</strong></a></li><li><a href="https://medium.com/@balachandarkm/disable-animations-while-running-android-ui-tests-2d1748280156"><strong>https://medium.com/@balachandarkm/disable-animations-while-running-android-ui-tests-2d1748280156</strong></a></li><li><a href="https://medium.com/@shepeliev/setting-up-gitlab-ci-for-android-project-2e1082f431a2"><strong>https://medium.com/@shepeliev/setting-up-gitlab-ci-for-android-project-2e1082f431a2</strong></a></li><li><a href="https://medium.com/fotonicia/gitlab-ci-with-instrumentation-tests-bbd702abb53"><strong>https://medium.com/fotonicia/gitlab-ci-with-instrumentation-tests-bbd702abb53</strong></a></li></ul><p><strong>That’s all folks! Thank you for your patience. We hope this article inspired you create your own UI test automation structure on GitLab. We’re waiting your feedbacks. 🙏</strong></p><p><strong>Tuğçe AKTEPE &amp; Yunus Emre GÖRMÜŞ</strong></p><p><strong><em>We would like to thank our Android Platform, Fulfillment SRE and Fulfillment Mobile teams who contributed to the completion of this work.</em></strong></p><p>If you are interested in this world, you can join Trendyol’s rich technology world.</p><p><a href="https://jobs.lever.co/trendyol">Trendyol jobs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b531d83fb37e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/trendyol-fulfillment-android-instrumented-test-ci-b531d83fb37e">Trendyol Fulfillment Android Instrumented Test CI</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ansible? Kurulum ve Ortamlarınızı Otomatikleştirme]]></title>
            <link>https://medium.com/sabancidx/ansible-kurulum-ve-ortamlar%C4%B1n%C4%B1z%C4%B1-otomatikle%C5%9Ftirme-2249b472344d?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2249b472344d</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ansible-playbook]]></category>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Thu, 01 Apr 2021 00:03:46 GMT</pubDate>
            <atom:updated>2021-04-01T07:22:26.695Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL7uecHQ-n_1YWVpccP0cg.png" /></figure><p>Merhaba, bu yazımda Ansible ile ilgili kurulum aşamasından yönetimimizde olan sunucularda herhangi bir güncelleme yaptığımızda (ki bu sayının fazla olduğunu hesap edelim) tek elden nasıl işlerimizi kolaylaştırabildiğini anlatmaya çalıştım.</p><p>Yazıma başlamadan önce de epey kaynak ve video izledim. Olabildiğince ziyaret ettiğim her siteyi yazının sonunda paylaşacağım fakat adı üstünde sunucu ortamlarımızı otomatikleştirirken bunu sağlayan diğer teknoloji araçları olduğunu, aslında hepsinin birbirinin muadili olsa da ansible’ın sunduğu ilk kolaylıkla başlayalım. Öncelikle Ansible puppet gibi bağlantı kurduğu sunucya gidip bir ajan kurma gereği duymaz, bunu serialized data language biçimlendirici olan /etc/ansible/ altında oluşturduğunuz bir yaml dosyasından yola çıkarak küçük programları (tasks) çalıştırarak işini halleder.</p><p>Ansible Server: 35.237.42.208 (RHEL7)</p><p>Ansible Client: 34.73.63.137 (RHEL8)</p><p>Daha sonradan havuza ekleyeceğimiz diğer sunucu;</p><p>Ubuntu: 104.154.21.22 (Ubuntu 18.04)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FG8jXZfpSF2qlG4sIJawlA.png" /></figure><p>Kurulumu için Redhat 7.x sürümü tercih ettim ben özellikle diğer kaynaklardan farklı olması için fakat Centos/Fedora için</p><p><a href="https://urclouds.com/2019/01/19/ansible-installation-and-configuration-on-centos-7/">https://urclouds.com/2019/01/19/ansible-installation-and-configuration-on-centos-7/</a> yararlanabilirsiniz.</p><p><em>Belirtmekte fayda olacaktır, 0.0.0.0/0 allow ile bulut güvenlik duvarından dışarı erişim vermeniz gerekebilir, public ip ile testlerde erişim sıkıntısı yaşamamak adına ve python ön gerekliliklerden biridir.</em></p><p>Sunucuda süper kullanıcı yetkilerini almak için ben işlemlerini her defasında sudo koşmamak için root üzerinden gerçekleştirdim.</p><pre># yum -y install epel-release.</pre><pre># yum repolist</pre><pre># yum update<br># yum install -y <a href="https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm">https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm</a> // ansible kurulumu için gerekli paket yüklemeleri</pre><pre># yum list available — disablerepo=* — enablerepo=epel</pre><pre># yum search ansible //ansible listelenebildikten sonra install edebilirsiniz</pre><pre># yum install ansible</pre><pre># ansible — version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_NkRlFrMxyOh3aqVJLAh6Q.png" /></figure><p>İlk önce client sunucumuza erişebilmemiz ve üzerinde seri şekilde tasklarımızı tamamlayabilmemiz için bu sunucular arasında key oluşturmamız gerekiyor ve birbirlerine kopyaladığımız veya tanıttığımız bu key sayesinde root ssh gidebileceğimize bakalım.</p><p># ssh-keygen</p><p>Enter passphrase adımlarını geçtikten sonra</p><p># ssh-copy-id root@34.73.63.137</p><p>Bu adımda hata almanız gerekiyor. Permission denied ve benzerleri.</p><p>Bu adımları geçmek için öncelikle sunucularınızda</p><p># vi /etc/ssh/sshd_config</p><p>PasswordAuthentication yes ve PermitRootLogin yes. satırlarını güncelleyip sshd restart edin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKQTJhQhvUMEhNimrP9Nzw.png" /></figure><p># systemctl restart sshd</p><p>Ardından /etc/ansible default kurulumu gerçekleştirdiğimiz dizinde hosts dosyası olduğunu göreceksiniz. Aslında bu host dosyası bizim ansible ile yönettiğimiz sunucuları eklediğimiz dns dosyası.</p><p>vi editör ile içerisine girip aşağıdaki gibi client sunucularımızı ekleyebiliriz.</p><pre>[linux]</pre><pre>34.73.63.137</pre><pre>[linux:vars]</pre><pre>ansible_user=root<br>ansible_password=your_root_passwd</pre><p>Bu adımda hata almamak için root olduğunuz sunucularda #passwd ile belirleyebileceğiniz root şifrenizi burada variables’a eklemelisiniz.</p><p>Burada linux o host’daki sunucuları temsil eder (daha sonra ubuntu için de aynı benzer satırları ekleyeceğiz) , yani başlangıç ansible komutlarımızı koşalım:</p><pre># ansible -m ping linux<br># ansible -m command -a “uptime” ‘linux’<br>#ansible linux -a “cat /etc/os-release”</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2BQ01az96_rySCpvxDu0aw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0HDtW9OwEEM7hUfvRX9EKg.png" /></figure><p>Buraya kadar tamamsak eğer asıl ansible nerede söz söyler oraya geçelim.</p><p># vi &lt;anyname&gt;.yml dosyası oluşturun /etc/ansible/ dizininde</p><p>Ben burada yml formatında hata almamak için VsCode ile sunucuma bağlanıp aynı yerde yaml dosyasını oluşturup kopyaladım.</p><p>Google cloud’daki Compute Engine VMlerinize bağlanmak için şu adımları takip edebilirsiniz:</p><pre># gcloud init</pre><pre># gcloud compute config-ssh // burada örnek ssh komutu çıktıda yazdırılır, siz sadece server makinenize bağlanmalısınız.</pre><pre># ssh ansible-server.us-east1-b.phrasal-descent-309313</pre><p>vi editörle oluşturduğumuz yaml içerisine girelim.</p><pre># vi iluvlinux.yml</pre><p>Buraya kadar konfigürasyonda değişiklik yapmadıysanız aşağıdaki şekilde devam edebilirsiniz.</p><pre>---</pre><pre>-  name: iluvlinux</pre><pre>hosts: linux</pre><pre>tasks:</pre><pre>-  name: Apache yukleme</pre><pre>yum:</pre><pre>name: httpd</pre><pre>state: latest</pre><p>Bundan sonra devreye playbook giriyor, ansible-playbook ile yaml dosyamızın içindeki verileri, sorasıyla okuyarak gerekli işlemleri tamamlar, hata alınırsa satırda hangi exception olduğunu belirtir.</p><pre># ansible-playbook iluvlinux.yml</pre><p>Ardından Redhat client sunucumuzda yapmamız gereken tek şey;</p><p># systemctl start&amp;status httpd demek olacaktır</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgnVyjjMjQmISQuu2ik7IA.png" /></figure><p>Peki bizim havuzumuzda farklı yayılım, versiyon ve amaçlar içeren sunucularımız olsa ve ansible ile yine aynı şekilde tek elden yönetmek istesek; o zaman da çok ufak dokunuşlarla aynı işlemi farklı host çatısı altındaki sunucularımıza hatta ilk başta yaptığımız gibi httpd start demeden bile onu da farklı görevde tanımlayarak gerçekleştirebiliriz. Yaml son hali;</p><pre>---</pre><pre>-  name: iluvlinux</pre><pre>hosts: linux</pre><pre>tasks:</pre><pre>-  name: Apache yukleme</pre><pre>yum:</pre><pre>name: httpd</pre><pre>state: latest</pre><pre>-  name: Apache baslatma</pre><pre>service:</pre><pre>name: httpd</pre><pre>state: started</pre><pre>-  name: iluvlinux</pre><pre>hosts: ubuntu</pre><pre>tasks:</pre><pre>-  name: Apache yukleme Ubuntu</pre><pre>apt:</pre><pre>name: apache2</pre><pre>state: latest</pre><p>Gördüğünüz gibi artık linux hostumuzun bir taskı başlatma ve ubuntu için yumdan farklı olarak apt koşuyoruz.</p><p>Rhel8 için httpd stop edip playbook ile yeniden sunuculardaki işleri başlatalım.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnoLsm9DA8ui-0Nscmh7rA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AUJIPFapAQRpiUCtKyVEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyvhvzAriCZ6jn596Xn8OA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0bc172lYohC3KzPXVrVsw.png" /></figure><p>Tarayıcıdan da apache web sunucularımızı ön yüzde çağırabildik. Aynı işlemleri tekrardan yapsaydık Ansible yeşil yansa da changed 0 olarak gösterecekti yani yaptığı işi tekrardan yapma gereği duymuyor, bu 400 sunucuyu yönettiğiniz ve core os upgrade dahil daha zaman alıcı işlerde bize her trülü vakit kazandıran ansible yeteneklerinden birisi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpCsJkkF1rijAqpqvkrL1A.png" /></figure><p>Yararlandığım kaynaklar:</p><p><a href="https://medium.com/devopsturkiye/ansible-nedir-dosya-yap%C4%B1s%C4%B1-nas%C4%B1ld%C4%B1r-nas%C4%B1l-kullan%C4%B1l%C4%B1r-4d8c90cdb266">https://medium.com/devopsturkiye/ansible-nedir-dosya-yap%C4%B1s%C4%B1-nas%C4%B1ld%C4%B1r-nas%C4%B1l-kullan%C4%B1l%C4%B1r-4d8c90cdb266</a></p><p><a href="https://www.youtube.com/watch?v=5hycyr-8EKs">https://www.youtube.com/watch?v=5hycyr-8EKs</a></p><p><a href="https://www.itzgeek.com/how-tos/linux/centos-how-tos/enable-epel-repository-for-centos-7-rhel-7.html">https://www.itzgeek.com/how-tos/linux/centos-how-tos/enable-epel-repository-for-centos-7-rhel-7.html</a></p><p><a href="https://www.digitalocean.com/community/questions/ssh-copy-id-not-working-permission-denied-publickey">https://www.digitalocean.com/community/questions/ssh-copy-id-not-working-permission-denied-publickey</a></p><p>Faydalı olması dileğiyle, bir sonraki yazımda görüşmek üzere. Sağlıkla kalın.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2249b472344d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sabancidx/ansible-kurulum-ve-ortamlar%C4%B1n%C4%B1z%C4%B1-otomatikle%C5%9Ftirme-2249b472344d">Ansible? Kurulum ve Ortamlarınızı Otomatikleştirme</a> was originally published in <a href="https://medium.com/sabancidx">SabancıDx</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ELK ve Prometheus & Grafana Kurulumları, Temel Dashboard Hazırlama]]></title>
            <link>https://medium.com/sabancidx/elk-ve-prometheus-grafana-kurulumlar%C4%B1-temel-dashboard-haz%C4%B1rlama-3a536dbcbc10?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a536dbcbc10</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 02:13:00 GMT</pubDate>
            <atom:updated>2021-02-08T02:16:51.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*orNyn0n2sN5p2M7gZUemCg.png" /></figure><p>Merhabalar,</p><p>Monitoring sistemler ve araçlar uygulamalarınızın devamlılığı, destek verilebilirliği açısından en çok öneme sahip araçlar arasında gittikçe önemini artırmaktalar. Bu senaryoda Ubuntu yayılımlı sunucunuzda apache web server kurmayı ve uygulama loglarını (/var/log(access.log, error.log) ELK üzerinde logstash aracılığıyla görüntülemek; ayrıca bu sunucunun sistem loglarını docker olarak ayağa kaldırdığımız Prometheus’a aktarmak istiyoruz. En sonunda da kibana üzerinde oluşturulan dashboardlardan anlık takip sağlanabilecek.</p><p>Öncelikle biz Google Cloud’dan kiraladığımız 4 vCPU 18 GB kaynaklı Ubuntu 20.04 versiyonlu makina üzerinde çalıştık. Aynı adımları denerseniz RHEL/Centos ve Debian distrolarda bazı komutların muadillerini kullanmanız gerekebilir.</p><p>İlk önce ELK Stack kurulumuyla başlayalım.</p><p>İlk gereklilik <strong>Java</strong>:</p><p>#sudo su -</p><p>#apt update</p><p>#apt install default-jre</p><p>#apt install default-jdk</p><p>#javac -version &amp;&amp; java -version</p><p>Beklenen Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*1esisoQR9DyeHDP4pBv2iA.png" /></figure><ol><li><strong>Elasticsearch</strong></li></ol><p>Elasticsearch GPG anahtarını apt’e bağlamak için:</p><p>#wget -qO — <a href="https://artifacts.elastic.co/GPG-KEY-elasticsearch">https://artifacts.elastic.co/GPG-KEY-elasticsearch</a> | sudo apt-key add -</p><p>Ardından Elastic source listi sources.list.d’e eklemek için:</p><p>#echo “deb <a href="https://artifacts.elastic.co/packages/7.x/apt">https://artifacts.elastic.co/packages/7.x/apt</a> stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list</p><p>#apt update</p><p>#apt install elasticsearch</p><p># vi/etc/elasticsearch/elasticsearch.yml</p><p>Aşağıdaki satırları yorumdan çıkarın. Herhangi bir değişiklik yapmanıza gerek kalmayacaktır.</p><p>cluster.name: my-application<br>node.name: node-1<br>network.host: localhost<br>http.port: 9200</p><p>#systemctl start elasticsearch</p><p>#systemctl status elasticsearch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Ro4eD-4LYFM7WLWA3PsHg.png" /></figure><p>veya #curl -X GET <a href="http://localhost:9200/">http://localhost:9200</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*CuKpXikj1s4V5jQNPd1XFQ.png" /></figure><p><strong>2.Logstash</strong></p><p>#apt install logstash</p><p>#systemctl start logstash</p><p>#systemctl status logstash</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vAhlRp0xTm5_DGC4d4X5Pg.png" /></figure><p><strong>3.Kibana</strong></p><p>Kibana, bu üçlü kombinde topladığımız verileri gerçek zamanlı analizle görsel arayüz üzerinde açıklamamıza yarayan araç.</p><p>#apt install kibana</p><p>#systemctl start kibana</p><p>#systemctl status kibana</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OC_cur2hqs7zG-DI0ehYeg.png" /></figure><p>Bu aşamada <a href="http://localhost:5601/">http://localhost:5601</a> üzerinden arayüze ulaşmaya çalıştığınızda muhtemelen hata alıp ulaşamayacaksınız. Burayı aşmak için stack toollarının konfigürasyon dosyalarında bazı yerlere ufaktan dokunmak gerekecektir.</p><p>#vi /etc/kibana/kibana.yml</p><p>server.host ve elasticsearch.host satırları ekran görüntüsündeki gibi ideal kabul edilir.</p><p>(default’da gelen “localhost” yerine “0.0.0.0” verdik çünkü bizim sunucumuz external ve internal (nic) IP’e sahib)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D7mZsaZ8xgc6vsCeFBLLKg.png" /></figure><p>IP’nizi öğrenmek için</p><p>#hostname -I</p><p>(Benim dışarı çıktığım IP’im: 104.196.147.125)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1afeezrfsqxUZZl-x5AWOQ.png" /></figure><p>#vi /etc/elasticsearch/elasticsearch.yml</p><p>network.host ve http.port satırlarını default değerlerinde comment out yapmanız yeterli olacaktır.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*MI9gDh2DNsc72wmi-wOVLg.png" /></figure><p><strong>Apache Kurulumu:</strong></p><p>Bazı paket güncellemeleri:</p><p>#apt update</p><p>#apt upgrade</p><p>#apt install apache2</p><p>#systemctl is-enabled apache2.service</p><p>#systemctl enable apache2.service</p><p>#systemctl start apache2.service</p><p><strong><em>Optional olarak 80&amp;443 TCP portlarını herkese açmak için:</em></strong></p><p>#ufw allow 80/tcp comment ‘accept Apache’<br>#ufw allow 443/tcp comment ‘accept HTTPS connections’</p><p>Bunun yerine GCP konsolunda kendinize Firewall kuralları set edebilirsiniz.</p><p>Ayrıca sunucunuzu kurma aşamasında default bu değerleri enable’a çekebilme şansınız da var.</p><blockquote><a href="https://console.cloud.google.com/networking/firewalls/list">https://console.cloud.google.com/networking/firewalls/list</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nVQcSmhDpR2dGBXJuIDkgw.png" /></figure><p>Apache access ve error loglarını logstash’a aktarmak için pipeline kurmak gerekiyor. Bunun için metricbeat veya filebeat benzeri bir uygulamaya daha ihtiyacınız var. Burada filebeat kullanıldı.</p><p><strong>Filebeat Kurulumu</strong></p><p>#wget -qO — <a href="https://artifacts.elastic.co/GPG-KEY-elasticsearch">https://artifacts.elastic.co/GPG-KEY-elasticsearch</a> | sudo apt-key add -</p><p>Repository indirilir:</p><p>#echo “deb <a href="https://artifacts.elastic.co/packages/7.x/apt">https://artifacts.elastic.co/packages/7.x/apt</a> stable main” | sudo <br>tee -a /etc/apt/sources.list.d/elastic-7.x.list</p><p>Paket güncellemesi yapılır ve kurulum:</p><p>#apt-get update &amp;&amp; sudo apt-get install filebeat</p><p>Apache modulü aktif edilir.</p><p>#filebeat modules enable apache</p><p>#filebeat setup -e</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtiwEMuX3tG4TFHM3YsS2w.png" /></figure><p>#service filebeat start</p><p>Healthcheck koşmak için:</p><p>#curl -X GET “localhost:9200/_cat/indices?v”</p><p>Beklenen output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*cSRxkKmNJWnTN5ooFbyYrg.png" /></figure><p>son olarak elastic yerine logstash’i aktif etmek için:</p><p>#vi /etc/filebeat/filebeat.yml</p><p>output.elasticsearch yoruma alınır ve logstash uncomment edilir.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*WtrzE3-7AbNDvZkF4gfgAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*eQlE_FL2ncIARRyI8UfMXg.png" /></figure><p>#systemctl restart logstash</p><p>#systemctl restart kibana</p><p><a href="http://localhsot:5601/">http://localhsot:5601</a> veya bende <a href="http://my_external_ip:5601/">http://my_external_IP:5601</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41zvOZyaAuRL16v3MogiRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJaQa7_x_DA4kSZAoMyMkA.png" /></figure><p><strong>2.Aşamada Prometheus ve Grafanayı Docker üzerinde ayağa kaldıracağız.</strong></p><p>İlk önce docker kurulumunu gerçekleştirelim:</p><p>varsa eski sürümleri kaldırmak için:</p><p>#apt-get remove docker docker-engine docker.io containerd runc</p><p>#apt-get update</p><p>#apt-get install \<br> apt-transport-https \<br> ca-certificates \<br> curl \<br> gnupg-agent \<br> software-properties-common</p><p>Docker resmi GPG keyi eklenir.</p><p>#curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</p><p>Doğrulanır.</p><pre>apt-key fingerprint 0EBFCD88</pre><p>x86_64 / amd64 için</p><p>#add-apt-repository \<br> &quot;deb [arch=amd64] https://download.docker.com/linux/ubuntu \<br> $(lsb_release -cs) \<br> stable&quot;</p><p><strong>Docker engine kurulumu:</strong></p><p>#apt-get update</p><p>#apt-get install docker-ce docker-ce-cli containerd.io</p><p>#apt-get install docker-ce</p><p>Kaynak: <a href="https://docs.docker.com/engine/install/ubuntu/">https://docs.docker.com/engine/install/ubuntu/</a></p><p>(Bu adımdan sonra docker kurulumunuzu #docker run hello world koşarak çıktıda “Hello from Docker” gördükten sonra doğrulayabilirsiniz)</p><p>Prometheus kurulumu:</p><p>#docker pull /prom/prometheus</p><p>#mkdir /etc/prometheus</p><p>#vi /etc/prometheus/prometheus.yml</p><pre># A scrape configuration scraping a Node Exporter and the Prometheus server<br># itself.<br>scrape_configs:<br>  # Scrape Prometheus itself every 10 seconds.<br>  - job_name: &#39;prometheus&#39;<br>    scrape_interval: 10s<br>    target_groups:<br>      - targets: [&#39;localhost:9090&#39;]</pre><p>Şimdilik sadece kendini monitor edecek.</p><pre>#docker run -d -p 9090:9090 \--net=host \<br>-v /etc/prometheus/prometheus.yml \ <br>-v /data/prometheus \ <br>prom/prometheus \ <br>--config.file=&quot;/etc/prometheus/prometheus.yml&quot; \ <br>--storage.tsdb.path=&quot;/data/prometheus&quot;</pre><p>ilk etapta container ayağa kalkmamış olabilir.</p><p>#docker images</p><p>#docker run &lt;Image_ID&gt;</p><p>#docker container ls veya docker ps -a</p><p>#docker container start &lt;Container_ID&gt;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*AWDaQsHlLF-3Z1ScEhp-Rg.png" /></figure><p><a href="http://my_external_ip:9090/">http://my_external_ip:9090</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVOrjeJjRQDqkIdUnA5b6g.png" /></figure><p><strong>Grafana Kurulumu ve basic dashboard oluşturma</strong></p><p>#docker run -d -p 3000:3000 grafana/grafana</p><p><a href="http://my_external_ip:3000/">http://my_external_ip:3000</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8Mp9N_6m_aclRttDAbNew.png" /></figure><p>Add your first data source -&gt;</p><p>Prometheus seçilir:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efToWu7OTPzmgK4zb5Hvlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BwqrvPBGPjEf0PyMAJzFLg.png" /></figure><p>Save &amp; Test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJGmKSuH_c82qj4aRXJGug.png" /></figure><p>Ekran görüntüsündeki gibi soldaki panelden -&gt; Import</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w9arbnAUdkE8k2sh7FVSSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2m_oXMN96Zoo9CpefzzzNg.png" /></figure><p>Karşınızda temel bir performans dashboard’ı göreceksiniz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lbGoX_uXjCZHy6ccfZxbQ.png" /></figure><p>Dilerseniz Google Cloud veya Azure’da çalışan Kubernetesleriniz, VMleriniz ve network ortamlarınızı da Data Source kısmında bizim örneğimizde Google Cloud Monitoring seçerek görselleştirebilirsiniz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnjiavOxbbAyLKozPA6CNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2JNICO6U5W7_4jvzABa1g.png" /></figure><p>Servis account keyinizi json formatında indirmek için Cloud konsolda</p><p>IAM &amp; Admin-&gt; Servis Account -&gt; Create key-&gt;</p><p><a href="https://console.cloud.google.com/iam-admin/serviceaccounts">https://console.cloud.google.com/iam-admin/serviceaccounts</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*0qc1oTGI1SB7TKztF_DEAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*camu51k858YuQj1Tu5lY_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ap4qwFt4j52MRR84Y8K38A.png" /></figure><blockquote><strong>Cloud monitoringde ikinci seçenek olarak provider’ın kendi sunduğu platformdaki elk ve diğer toolları tercih edebilirsiniz:</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QMHEP-JxOq5HJkfpO3t69A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCyR20ehK867xFruc4TkIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwIm532uv_Vd40BsSmspMQ.png" /></figure><p>Son olarak prometheus’u sunucunuza bağlamak ve performansını izlemek için Node Exporter kurulumu yapmalısınız.</p><pre>#docker run -d -p 9100:9100 \<br>-v &quot;/:/hostfs&quot; \<br>--net=&quot;host&quot; \<br>prom/node-exporter \<br>--path.rootfs=/hostfs</pre><p>Node exporter’ın çalıştığından emin olmak için:</p><p>#curl <a href="http://localhost:9100/metrics">http://localhost:9100/metrics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*VYER1ZZnJMtY7qIXJ5z4Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*DnyoiSaNp3zhkL9ziTFh3A.png" /></figure><p>Prometheus ve Node exporter’ı bind etmek istereniz prometheus.yml’a</p><pre>- job_name: &#39;node_exporter&#39;<br>    scrape_interval: 10s<br>    target_groups:<br>      - targets: [&#39;localhost:9100&#39;]</pre><p>satırlarını eklemelisiniz; restart gerektirmeyecektir.</p><p>Bu defa biraz uzunca oldu, yardımı olması dileklerimle. Bir sonrakinde görüşmek üzere. Sağlıkla hoşça kalın.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a536dbcbc10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sabancidx/elk-ve-prometheus-grafana-kurulumlar%C4%B1-temel-dashboard-haz%C4%B1rlama-3a536dbcbc10">ELK ve Prometheus &amp; Grafana Kurulumları, Temel Dashboard Hazırlama</a> was originally published in <a href="https://medium.com/sabancidx">SabancıDx</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Cloud App Engine ile CI/CD Pipeline Süreçler İnşa etmek]]></title>
            <link>https://medium.com/sabancidx/google-cloud-app-engine-ile-ci-cd-pipeline-s%C3%BCre%C3%A7ler-i%CC%87n%C5%9Fa-etmek-bb67aa7ad49d?source=rss-46bc7b096d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb67aa7ad49d</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[app-engine]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[web-apps]]></category>
            <dc:creator><![CDATA[Yunus emre]]></dc:creator>
            <pubDate>Tue, 02 Feb 2021 21:54:01 GMT</pubDate>
            <atom:updated>2021-02-02T21:54:01.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*L3LJP5zVW83M6HLpDWfvbA.png" /></figure><p>App Engine, web uygulamalarınızın auto-deploy buildlerini gerçekleştirmenize yarayan Google tarafından sunulan bir geliştirme platformudur.</p><p>DevOps alanında çalışanlar olarak projelerimizde ve müşterilerimizin taleplerine daha hızlı cevap verebilmek adına çeşitli teknolojiler, bunların yan ürünleri ve bu araçları daha kolay ve güvenilir hatta ekonomik kullanmamızı sağlayan Bulut Teknolojilerine yöneliyoruz. Sürekli Entegrasyon ve Sürekli Dağıtma (CI/CD) terimlerine yakından bakabilmek, ekibimize yeni katılan arkadaşlarımızın bu kültüre daha kolay aşina olmasını sağlayabilmek adına bizler de internal eğitimlerle bildiklerimizi paylaşarak pekiştirmekteyiz. Bu çalışmamızda GCP (Google Cloud Platform)’nin bize sağladığı kolaylıklarla uçtan uca CI/CD pipeline’mızı gelin birlikte kuralım.</p><p>Sıfırdan başladığımızı düşünecek olursak işe Projenizi seçmek&amp;yaratmakla başlayabilirsiniz.</p><figure><img alt="Create your project" src="https://cdn-images-1.medium.com/max/527/1*wVlDbOoJz1lUCkc8X6cLEA.png" /></figure><p>İşlemlerinize Cloudshell’de, Konsol editöründe veya lokalinizdeki editörde devam edebilirsiniz. Bu özgürlüğe sahip olmak için yapmanız gereken tek şey Cloud SDK’yı indirmek ve Google Cloud için ‘Authorize’ etmek olacaktır. Ayrıca VSCode içerisindeki Cloud extensionları sayesinde biçimlendirmelerinizi daha kolay yapıp daha kolay kod review, bugfix ve paket import edebileceksiniz.</p><p><strong><em>Install SDK link</em></strong>: <a href="https://cloud.google.com/sdk/docs/install">https://cloud.google.com/sdk/docs/install</a></p><p><strong>İkinci adım </strong>cloud repository’nizi oluşturmak olacaktır, bundan önce Cloud Source Repositories API’yi etkinleştirmeniz gerektiğini unutmayın.</p><p><a href="https://console.cloud.google.com/flows/enableapi">https://console.cloud.google.com/flows/enableapi</a></p><p>#gcloud init</p><p><strong><em>hello-world adında bir repo oluşturun:</em></strong></p><p># gcloud source repos create hello-world</p><p><strong><em>Yarattığınız repoyu lokal gitinize klonlayın:</em></strong></p><p>#gcloud source repos clone hello-world</p><p>Bu adımdan sonra reponuzu editörünüzde görebiliyor olmanız gerekiyor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*_aN9OEbxapHplfAA1R5qnw.png" /></figure><p>Sırasıyla <strong>main.py</strong> (Source kodunuzun olduğu uygulamanızın neyi çalıştırdığı base kod)</p><pre>#!/usr/bin/env python<br><br>import webapp2<br><br>class MainHandler(webapp2.RequestHandler):<br>    def get(self):<br>        self.response.write(&#39;Open your ingress to everyone&#39;)<br><br>app = webapp2.WSGIApplication([<br>    (&#39;/&#39;, MainHandler)<br>], debug=True)</pre><p><strong><em>app.yaml</em></strong> dosyalarını oluşturun: (Uygulamanızı dağıtmanızı sağlayacak olan konfigürasyon bilgilerini içeren dosya)</p><pre>script: main.app<br><br>libraries:<br>- name: webapp2<br>  version: &quot;2.5.2&quot;</pre><p>ve daha sonrasında CI/CD Pipeline&#39;da otomasyonu sağlayacak olan <strong><em>Cloudbuild.yaml</em></strong> dosyamızı oluşturmalıyız.</p><pre>steps:<br>- name: &quot;<a href="http://gcr.io/cloud-builders/gcloud">gcr.io/cloud-builders/gcloud</a>&quot;<br>  args: [&quot;app&quot;, &quot;deploy&quot;]<br>timeout: &quot;1600s&quot;</pre><p>Dizinimize gidip bütün dosyalarımız cloud repomuzdaki master branch&#39;imize push edelim.</p><p>#cd hello-world</p><p># git add . // <em>bu dizindeki hepsini kapsa</em></p><p>#git commit -m &quot;commit message&quot; //<em>yaptığımız değişiklikleri onaylıyoruz gibi </em>düşünebilirsiniz.</p><p>#git push origin master</p><p>Reponuzu Cloud&#39;da görüntülemek için</p><p><a href="https://source.cloud.google.com/repos">https://source.cloud.google.com/repos</a></p><p>Bizdeki görüntü;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*uAXI6n5Rbsmo4SVdqDHhaQ.png" /></figure><p>Sıra geldi uygulamamızı App Engine’e dağıtmaya diğer tabirle biz kodumuzu (source repomuzu) içine atacağız gerisini o halledecek :)</p><p>Örnek uygulamamızı deploy edelim:</p><p>#gcloud app deploy app.yaml</p><p>VSCode Authorize ettiyseniz birazdan yazacağınız komut otomatik olarak default tarayıcınızda uygulamanızın ön yüzünü getirir; konsol arayüzünde ise web görünümüne tıklayıp veya CLI’da verdiği linkten ulaşabilirsiniz.</p><p>#gcloud app browse</p><p>alacağınız url formatı genellikle https://$<a href="http://project_id.location.r.appspot.com/">PROJECT_ID.location.r.appspot.com</a> olacaktır.</p><p><a href="https://deployment-emre1.ue.r.appspot.com/">https://deployment-emre1.ue.r.appspot.com</a> // Daha sonra bunun ingress ile nasıl external kullanıcılara da açılacağını&amp;kapanacağına göz atacağız.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*zEZEK5CjLNJ8v0b7g0CkBg.png" /></figure><p>CI/CD entegrasyonumuza geçmeden kod değişikliği ve manual deployment gerçekleştirelim.</p><p>Editörünüzde main.py dosyanızda bir değişiklik yapın ardından yeniden lokal git’e ekleyip commit ve push’layın ve</p><pre>#!/usr/bin/env python<br><br>import webapp2<br><br>class MainHandler(webapp2.RequestHandler):<br>    def get(self):<br>        self.response.write(&#39;First change inside big code&#39;)<br><br>app = webapp2.WSGIApplication([<br>    (&#39;/&#39;, MainHandler)<br>], debug=True)</pre><p>#gcloud app deploy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*MVqu6vcIHEwgGqLHrlfAEg.png" /></figure><p>Tek komut koşmak küçük bir projede ilk bakışta bize zor gelmeyebilir fakat çoklu organizasyonlarda, geniş süreçlerde ve karmaşık takımların ve departmanların ortak noktada birleştiğini düşündüğünüzde özellikle operasyon ekiplerine ekstra mesai demektir, bunun yanında yapılacak hataları ve örnek uygulamamızda master branch üzerinde çalıştığımızı hesaba katmıyoruz bile.</p><p>Şimdi otomatik buildimizi oluşturalım, ilk adım olarak <a href="https://console.cloud.google.com/cloud-build/settings/service-account">https://console.cloud.google.com/cloud-build/settings/service-account</a>. linkinden servis hesabı izinlerinizin etkin olduğunu (değilse etkinleştirin) emin olun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*j08NGNQoILZoaIrCmH4S3w.png" /></figure><p><a href="https://console.cloud.google.com/cloud-build/triggers">https://console.cloud.google.com/cloud-build/triggers</a> linkine gidip yeni bir trigger oluşturalım.</p><p>Cloudbuild dosyamızı da tam burada ‘Build Configuration’ altında kullanıyoruz.</p><p><a href="https://console.cloud.google.com/cloud-build/triggers">https://console.cloud.google.com/cloud-build/triggers</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*yc_wntkK1IDzApP9V-YU7A.png" /></figure><p>Tekrar main.py aracılığıyla kodumuzda ufak bir format değişikliği yapalım.</p><pre>#!/usr/bin/env python<br><br>import webapp2<br><br>class MainHandler(webapp2.RequestHandler):<br>    def get(self):<br>        self.response.write(&#39;Deployed automatically!&#39;)<br><br>app = webapp2.WSGIApplication([<br>    (&#39;/&#39;, MainHandler)<br>], debug=True)</pre><p>#git add .</p><p>#git commit -m “Update auto”</p><p>#git push origin master</p><p>değişikliklerimizin işe yaradığını endpointte ve Build history loglarından takip edelim.</p><p>#gcloud app browse veya sayfanızı yenileyin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*yED2fZVrXD6EU8SF4z5GYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*0_RR6ijpeF4vfNK1a8Y63Q.png" /></figure><p>Son olarak uygulamanızın erişimini dışarıya açtığınızdan emin olmak için GKE (Google Kubernetes Engine) altında cluster ayarlarınızda ingress konfigürasyonunuza HTTP Enabled izni vermeniz gerekmektedir.</p><p>Clusterınıza VSCode ile bağlanmak, Dockerize ettiğimiz (“Go”) uygulamamız için create edeceğimiz deployment ve servis yaml’larınızı yönetebileceğiniz CLI’ı lokalinize bağlamak için;</p><p>#gcloud cloud-shell ssh — authorize-session</p><p><strong><em>Project_ID’nizi öğrenmek için;</em></strong></p><p>#gcloud config get-value project</p><p>#gcloud config set project PROJECT_ID</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*lNh9Q-HzmSImBVpr3bwQYw.png" /></figure><p><a href="https://console.cloud.google.com/kubernetes/list">https://console.cloud.google.com/kubernetes/list</a> linkinden clusterlarınızı listeyin ve ingresinizi deploy ettiğiniz clusterı seçip edit mode’a gelin.</p><p>Add-ons altında HTTP Load balancing -&gt; Enabled</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*wooNCH91B9ocn-1XTtvGVQ.png" /></figure><p>Bizim endpointimizi ziyaret etmek isterseniz :)</p><p><a href="https://deployment-emre1.ue.r.appspot.com/">https://deployment-emre1.ue.r.appspot.com</a></p><p>veya hazırdaki repoyu klonlamak için</p><p>#gcloud source repos clone hello-world — project=deployment-emre1</p><p>Bir sonraki yazımızda görüşmek üzere sağlıkla, hoşça kalın.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb67aa7ad49d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sabancidx/google-cloud-app-engine-ile-ci-cd-pipeline-s%C3%BCre%C3%A7ler-i%CC%87n%C5%9Fa-etmek-bb67aa7ad49d">Google Cloud App Engine ile CI/CD Pipeline Süreçler İnşa etmek</a> was originally published in <a href="https://medium.com/sabancidx">SabancıDx</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>