<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Asilcan Şumnu on Medium]]></title>
        <description><![CDATA[Stories by Asilcan Şumnu on Medium]]></description>
        <link>https://medium.com/@asilcansumnu?source=rss-e2cfbac281b8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9i86p5VYnUhmSmriDUfHOQ.jpeg</url>
            <title>Stories by Asilcan Şumnu on Medium</title>
            <link>https://medium.com/@asilcansumnu?source=rss-e2cfbac281b8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:09:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asilcansumnu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How We Built an Automated, Production-Ready Sharded MongoDB Infrastructure]]></title>
            <link>https://medium.com/trendyol-tech/how-we-built-an-automated-production-ready-sharded-mongodb-infrastructure-00e29b26542f?source=rss-e2cfbac281b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/00e29b26542f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[sharding]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Asilcan Şumnu]]></dc:creator>
            <pubDate>Wed, 21 Jan 2026 05:54:49 GMT</pubDate>
            <atom:updated>2026-01-21T05:54:49.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5thIRYGv3OKn9K3M87tbhg.jpeg" /></figure><p>At Trendyol Tech, we as the Datastore NoSQL Team operate at a massive scale, managing thousands of VMs within our private cloud infrastructure. Our team oversees a diverse technology stack including Couchbase, Elasticsearch, Redis, and Cassandra. Although our existing solutions served us well, relying on them for general-purpose document storage posed a potential risk of vendor lock-in. We recognized the need to diversify our database infrastructure by adopting a suitable, highly scalable open-source solution.</p><p>We needed a general-purpose document database. But operating at our scale, we had strict requirements:</p><ol><li><strong>No Vendor Lock-in:</strong> We prioritized open solutions to ensure that licensing costs and restrictions don’t become a bottleneck as we scale.</li><li><strong>Enterprise Feature Set:</strong> We needed hot backups, LDAP integration, and audit logging features.</li><li><strong>Proven Resilience:</strong> It had to survive a full Data Center (DC) outage without manual intervention.</li></ol><p>This led us to Percona Server for MongoDB (PSMDB).</p><h3>The Decision Matrix: Why Percona MongoDB over MongoDB Community?</h3><p>We confirmed our decision through a strict operational review. Because we were adding a new document store to our ecosystem, we had to guarantee it met our standards for security and reliability. We compared the standard MongoDB Community Edition with Percona Server for MongoDB to determine which one worked best.</p><p><strong>Comparing the Features</strong></p><p>While MongoDB Community Edition offers the basic engine, it does not have the tools necessary for our needs. Percona Server for MongoDB solves this by offering essential “Day 2” features, like hot backups and encryption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l8vIdYLLkW00w1aC2UDang.png" /></figure><p><strong>The Decision Driver:</strong></p><p>Using the standard MongoDB Community edition often requires managing separate tools for different tasks. Although Percona Monitoring and Management (PMM) works with the Community version, choosing the complete Percona platform offers advantages like UI-based backup management. Also Percona Backup for MongoDB (PBM) allowed us to seamlessly integrate physical backups and Point-in-Time Recovery. This unified approach solved both our monitoring and disaster recovery needs before we even launched.</p><h3>Architecture: Multi-Datacenter Topology</h3><p>We implemented a <strong>3-Data-Center (3DC)</strong> topology to maximize availability within our private cloud. Our deployment distributes every component of the sharded cluster across three distinct data centers (DC1, DC2, DC3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPjCZTVOrDH480TmCZuRQw.png" /><figcaption><em>MongoDB multi-data center sharding strategy designed for high availability. </em>(Key: P = Primary, S = Secondary, DC = Data Center)</figcaption></figure><h4>Topology: The “PSS” Model</h4><p>As illustrated in the diagram above, our architecture relies on a <strong>Primary-Secondary-Secondary (PSS)</strong> distribution for high availability at every layer:</p><ul><li><strong>Shards:</strong> Each shard is a replica set with its members distributed across DC1, DC2, and DC3. For example, Shard1 has its Primary (P) in DC1, while its Secondaries (S) are in DC2 and DC3. This pattern is rotated across shards to balance the load.</li><li><strong>Config Servers:</strong> The metadata backbone is also a 3-node replica set, similarly spanning all three DCs to ensure resilience.</li><li><strong>Mongos Routers:</strong> Stateless query routers are deployed in each data center, allowing applications to connect to the nearest entry point.</li></ul><p>This topology ensures that if any single DC goes offline, the remaining two DCs still form a <strong>majority (2/3)</strong> for every shard and the config server replica set. This allows the cluster to elect new primaries and continue accepting writes without manual intervention.</p><pre>writeConcern:<br>  w: &quot;majority&quot;  # Requires acknowledgement from 2 out of 3 nodes<br>  j: true        # Requires journaling to disk</pre><p>We prioritize data consistency by enforcing w:majority. Given the low network latency between our availability zones, this configuration guarantees durability across data centers with minimal impact on write performance.</p><h3>Validating Resilience: Chaos Testing</h3><p>To validate the robustness of our architecture, we executed specific failure scenarios.</p><p>Below are the technical observations from the most critical tests.</p><p><strong>Scenario 1: Primary Node Hard Kill <br></strong>We simulated an immediate failure of the Primary node without a graceful step down.</p><ul><li><strong>Observation:</strong> The cluster detected the heartbeat failure and initiated a new election.</li><li><strong>Impact:</strong> The application experienced connection timeouts for approximately 10–12 seconds.</li><li><strong>Recovery:</strong> The MongoDB driver automatically refreshed its topology view, discovering the new Primary. Write operations resumed immediately after the election.</li></ul><p><strong>Scenario 2: Full Data Center Isolation (“DC1” Down)</strong></p><p>We isolated the “DC1”, which hosted the active Primary node for Shard 1 and a Mongos router.</p><ul><li><strong>Mechanism:</strong> Network traffic was blocked between DC1 and the other two DCs (DC2, DC3).</li><li><strong>Observation:</strong> Since DC2 and DC3 retained a majority (2/3 of voting members), they successfully elected a new Primary.</li><li><strong>Resilience:</strong> Due to writeConcern: “majority”, no acknowledged writes were lost. The application experienced a latency spike during the topology update but successfully failed over to the remaining Mongos instances.</li></ul><p><strong>Scenario 3: Config Server Majority Loss</strong></p><p>We shut down 2 out of 3 Config Server nodes, leaving the cluster without a writable metadata plane.</p><ul><li><strong>Observation:</strong> The cluster entered a “metadata read-only” state.</li><li><strong>Impact:</strong> Operations requiring metadata changes (Chunk splits, Balancer migrations, DDL operations) failed as expected.</li><li><strong>Continuity:</strong> Critical CRUD (Create, Read, Update, Delete) operations on existing chunks continued successfully. This confirmed that the Data Plane is resilient to temporary Control Plane outages.</li></ul><h3>Automation: Orchestrating Sharded Clusters at Scale</h3><p>Deploying a large sharded cluster from scratch is often a difficult operational challenge, so doing it manually was never an option. Instead, we treated MongoDB just like any other component in our existing ‘Infrastructure as Code’ pipeline. By using our established automation tools, we were able to build a standard, reproducible MongoDB environment right from Day 0.</p><p>Our installation pipeline automatically sets up the entire cluster, configuring Config Servers, Replica Sets for each shard, and Mongos Routers and wires them all together without human intervention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g178pFtUQdJQ3JFblrH3ug.png" /></figure><h3>Project Structure</h3><p>We organized our Ansible roles to be modular and scalable. A single repository manages the entire lifecycle of the cluster:</p><pre>mongodb-installation/<br>├── inventory/hosts.ini          # Dynamic Cluster inventory<br>├── group_vars/all.yml           # Global variables (Versions, Auth, etc.)<br>├── roles/<br>│   ├── mongodb_configsvr/       # Config server role<br>│   ├── mongodb_shard/           # Shard server role<br>│   ├── mongodb_mongos/          # Mongos router role<br>│   └── mongodb_users/           # User &amp; Role management<br>└── misc/<br>    ├── install-pbm-agent.yml    # Backup agent installation<br>    ├── install-pmm-agent.yml    # Monitoring agent installation<br>    └── backup-schedule.yml      # Automated backup scheduling</pre><h3>Code Example: Dynamic Configuration</h3><p>Our Ansible playbook calculates topology dynamically. We don’t hardcode shard names. Instead, we use Ansible Facts and Jinja2 Templating to scale the cluster by simply changing a single variable (mongodb_shard_count).</p><h4>1. Dynamic Shard Pattern Generation</h4><p>This task generates host patterns (e.g., shard1:shard2:shard3) on the fly, allowing us to loop through shard groups without manual inventory updates.</p><pre># roles/common/tasks/dynamic_sharding.yml<br>- name: Generate shard pattern based on shard count<br>  set_fact:<br>    shard_pattern: &quot;{{ range(1, mongodb_shard_count | int + 1) | map(&#39;string&#39;) | map(&#39;regex_replace&#39;, &#39;^(.*)$&#39;, &#39;shard\\1&#39;) | join(&#39;:&#39;) }}&quot;<br><br>- name: Add shards to cluster dynamically<br>  community.mongodb.mongodb_shard:<br>    login_host: &quot;{{ groups[&#39;mongos_routers&#39;][0] }}&quot;<br>    shard: &quot;shard{{ item }}-replica/{{ groups[&#39;shard&#39; ~ item][0] }}:27018&quot;<br>    state: present<br>  loop: &quot;{{ range(1, mongodb_shard_count | int + 1) | list }}&quot;</pre><h4>2. Context-Aware Configuration (Jinja2)</h4><p>Instead of maintaining separate config files for each shard, a single intelligent Jinja2 template determines the replSetName based on which inventory group the host belongs to.</p><pre># templates/mongod_shard.conf.j2<br>sharding:<br>  clusterRole: shardsvr<br>replication:<br>  # Dynamically assign Replica Set Name (e.g., shard1-replica)<br>  {% for i in range(1, mongodb_shard_count | int + 1) %}<br>  {% if inventory_hostname in groups[&#39;shard&#39; ~ i | string] %}<br>  replSetName: shard{{ i }}-replica<br>  {% endif %}<br>  {% endfor %}<br>net:<br>  port: 27018<br>  bindIp: 0.0.0.0<br>security:<br>  keyFile: /etc/mongod.keyfile<br>  clusterAuthMode: keyFile<br>  authorization: enabled</pre><p><strong>3. Robust Initialization with Primary Detection</strong></p><p>Roles include retry logic to ensure the cluster is healthy before proceeding with user creation or backup configuration.</p><pre># roles/mongodb_shard/tasks/init_replica.yml<br>- name: &quot;Initialize shard{{ shard_number }} replica set&quot;<br>  community.mongodb.mongodb_replicaset:<br>    login_host: &quot;127.0.0.1&quot;<br>    replica_set: &quot;shard{{ shard_number }}-replica&quot;<br>    members:<br>      - host: &quot;{{ groups[&#39;shard&#39; ~ shard_number][0] }}:27018&quot;<br>      - host: &quot;{{ groups[&#39;shard&#39; ~ shard_number][1] }}:27018&quot;<br>      - host: &quot;{{ groups[&#39;shard&#39; ~ shard_number][2] }}:27018&quot;<br>  register: replicaset_init<br><br>- name: Wait for Primary node election<br>  ansible.builtin.command: mongosh --quiet --eval &quot;db.hello().primary&quot;<br>  register: primary_node<br>  until: primary_node.rc == 0 and primary_node.stdout != &#39;null&#39;<br>  retries: 10<br>  delay: 15</pre><h4>4. Automated Backup Integration (PBM)</h4><p>Ansible automatically configures Percona Backup for MongoDB (PBM) by creating the necessary roles and linking the agent to our S3 or NFS storage, ensuring every new node is “Backup-Ready” from day one.</p><pre>- name: Configure PBM_MONGODB_URI<br>  lineinfile:<br>    path: &quot;/etc/default/pbm-agent&quot;<br>    regexp: &#39;^PBM_MONGODB_URI=&#39;<br>    line: &quot;PBM_MONGODB_URI=mongodb://{{ pbm_user }}:{{ pbm_pass }}@localhost:{{ mongo_port }}/admin&quot;<br>    create: yes<br>    mode: &#39;0600&#39;</pre><p><strong>5. Automated Monitoring Integration (PMM)</strong></p><p>Finally, we automated the monitoring setup. When the Terraform variable pmm_register is set to true, Ansible configures the PMM agent and registers the cluster to the PMM server. This ensures that every new node is automatically monitored as soon as the cluster is provisioned.</p><pre># variables.tf<br>variable &quot;pmm_register&quot; {<br>  type        = bool<br>  default     = false<br>  description = &quot;Enable PMM client registration to PMM server&quot;<br>}</pre><pre># misc/install-pmm-agent.yml<br>- name: Add MongoDB shard to PMM2<br>  command: &gt;<br>    pmm-admin add mongodb<br>    --username={{ mongodb_username }} --password=&#39;{{ mongodb_password }}&#39;<br>    --service-name={{ service_name }}<br>    --host={{ ansible_hostname }} --port={{ shard_port }}<br>    --cluster={{ pmm_cluster_name }}<br>    --replication-set=shard{{ replication_set_number }}-replica<br>  when: service_name not in existing_pmm_services</pre><h3>Observability: PMM &amp; Alerting with Prometheus</h3><p>We employ a dual-layer observability strategy. Percona Monitoring and Management (PMM) for deep performance analysis and our standard Prometheus + Alertmanager stack for real-time health protection.</p><h3>Layer 1: Performance Analysis (PMM)</h3><p>For us, the best feature of PMM is Query Analytics (QAN). It allows us to instantly visualize slow queries, examine execution plans, and identify unindexed operations without grepping through log files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbBpJVBtHA5P-i3ZJCly-A.png" /></figure><h3>Layer 2: The “On-Call” Alert Rules (Prometheus)</h3><p>While PMM handles the “Why”, our Prometheus exporters handle the “When.” We utilize mongodb_exporter metrics to trigger alerts based on severity.</p><p>Here are the rule sets that protect our production data:</p><h4>Cluster Availability</h4><ul><li><strong>MongoDB Service Down (Error):</strong> Triggered if mongodb_up == 0. This means an instance has stopped serving traffic entirely.</li><li><strong>Replica Set Member Unhealthy (Critical):</strong> Triggered if mongodb_members_health == 0. This indicates redundancy is compromised (e.g., a node fell out of the replica set), requiring immediate intervention.</li><li><strong>Oplog Window &lt; 24h (Critical):</strong> Triggered if the replication oplog window drops below 24 hours. This safeguards our ability to recover a node after a day-long outage without needing a full (and expensive) initial sync.</li></ul><h4>Data Safety &amp; Backup (PBM)</h4><p>Since backups are our last line of defense, we monitor them aggressively:</p><ul><li><strong>Backup Agent Down (Critical):</strong> Triggered if the PBM agent stops. We are flying blind without a running agent.</li><li><strong>No Backup in 24h (Critical):</strong> An RPO violation. Triggered if no successful backup is recorded in the last 24 hours.</li><li><strong>Backup Failed (Critical):</strong> Immediate trigger if a backup job reports an “error” status.</li><li><strong>Backup Size Dropped (Warning):</strong> A sanity check that triggers if the backup size is <strong>&lt; 50%</strong> of the 7-day average. This catches “successful” empty backups or configuration errors.</li><li><strong>PITR Not Enabled (Warning):</strong> Checks if Point-In-Time Recovery is inadvertently disabled on a cluster that requires it.</li></ul><h3>Validation: The YCSB Benchmarks</h3><p>We used the Yahoo! Cloud Serving Benchmark (YCSB) to validate our architectural choices. Our objective was not only to achieve a high “Operations Per Second” (OPS) but to prove that our hashed sharding strategy effectively distributes load and maintains low latency under varying concurrency levels.</p><p>​​<strong>The Test Environment</strong></p><p>We executed these tests on a <strong>3-Shard Cluster</strong> running on OpenStack.</p><ul><li><strong>Hardware per MongoDB Node:</strong> 8 vCPUs, 16GB RAM, Ubuntu 22.04 LTS.</li><li><strong>Hardware per Benchmark Client Node: </strong>4 vCPUs, 16GB RAM,<strong> </strong>Ubuntu 22.04 LTS.</li><li><strong>Topology:</strong> 3 Shards, 3 Config Servers, 3 Mongos Routers.</li><li><strong>MongoDB Version:</strong> Percona Server for MongoDB 7.0.14–8</li><li><strong>Dataset:</strong> 3 Million documents (~1.1KB size).</li><li><strong>Concurrency:</strong> Tests pushed from 16 up to 128 concurrent threads.</li></ul><h3>1. Cluster Performance Under Load (Concurrency)</h3><p>First, we stressed our target topology (3 Shards) to observe how it handles increasing concurrency. We wanted to verify that the cluster could maintain stability and a 100% success rate even as we saturated the client threads.</p><p><strong>Write-Heavy (100% Insert)</strong></p><p>As we increased threads from 16 to 128, the cluster scaled its throughput effectively, peaking at ~21,800 ops/sec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wj31rgvRSAjcONKjk-ObJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*k80hGDMv-fmAkuPd1IcVXg.png" /></figure><p><strong>Read-Heavy (90% Read, 10% Update)</strong></p><p>The results were even more impressive for read-heavy workloads, where the cluster sustained over 35,000 ops/sec with single-digit average latency (~9ms) at max load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_g1V8sOMr75MrrOrQt9vDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UJNREfO0OpQi-kyrzkRU7A.png" /></figure><p><strong>Balanced (50% Read, 50% Update)<br></strong>The<strong> </strong>workload demonstrated the cluster’s versatility in handling mixed operations, peaking at over 26,500 ops/sec. Even with the high contention of a 50/50 read-update split, the system scaled efficiently and maintained stable latencies as concurrency increased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xaK1MGtp58jN58x9jZGtJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*G0wPteUcMvEORPqbKjoAGg.png" /><figcaption><em>Observation:</em> The cluster maintained a 100% success rate across all tests. Even at 128 threads, P99 latency remained well within acceptable limits (56ms for writes, 22ms for reads), proving the stability of the PSS topology.</figcaption></figure><h3>2. The Scaling Factor: Why Sharding Matters?</h3><p>Having established that the 3-Shard cluster performs well, the critical question remained: Does adding shards actually provide proportional performance gains?</p><p>To validate this, we compared the performance of 1, 2, and 3 Shards under the maximum load (128 Threads). The results confirmed that sharding is a direct performance multiplier.</p><p><strong>Performance Matrix (128 Threads)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdWN-idSRTc3XHUVQ-XmIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*soNX5FQDSrJLfxFIfietWg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0SheR8O6jpkU2o31pZNMA.png" /></figure><h3>Key Takeaways</h3><p>1. <strong>Throughput Gains</strong></p><p>We observed substantial gains when moving from a single shard to two. While adding the third shard provided further growth, the scaling curve notably flattened. This sub-linear growth is expected due to the increased coordination overhead on the mongos routers and network latency from strict w:majority write concerns.</p><p>2. <strong>Latency as a Function of Sharding</strong></p><p>Scaling out had a direct positive impact on user experience. In the Write-Heavy test, as we expanded from 1 to 3 shards, the Average Latency dropped from 38.72ms to 18.07ms. This confirmed that our sharding key strategy successfully prevented “hot spots,” allowing parallel writes to occur without contention.</p><p>3. <strong>Stability at Limit</strong></p><p>Even under maximum load (128 threads) and varying workloads (including Read-Modify-Write), the cluster maintained a 100% Success Rate. While we noticed diminishing returns in raw throughput when adding the 3rd shard (due to Mongos coordination overhead), the latency remained stable.</p><h3>Conclusion</h3><p>Building this MongoDB stack wasn’t about replacing our existing databases. Instead, our goal was to offer a solid alternative for projects that specifically need a general-purpose document store.</p><p>By adding Percona Server for MongoDB to our tech stack, we provided a new option that helps us avoid vendor lock-in without compromising on quality:</p><ul><li><strong>More Options, Less Lock-in:</strong> We created a cost-effective choice for specific use cases. This allows us to use the right tool for the job while reducing our dependence on a single vendor.</li><li><strong>Proven Resilience:</strong> Our 3-Data-Center setup works. Chaos testing proved that this new option is enterprise-ready and keeps data safe even during major outages.</li><li><strong>Predictable Scaling:</strong> As our benchmarks showed, the system scales efficiently. While coordination overhead introduces a natural ceiling, the architecture effectively distributes load to handle high-growth projects.</li></ul><p><strong>What’s Next?</strong> This architecture is now a ready-to-use option for teams where MongoDB is the right fit. Moving forward, we are actively testing mongosync for Cluster-to-Cluster (C2C) synchronization. By validating mongosync, we aim to streamline data migrations between environments, ensuring that moving workloads into new clusters is as seamless as possible.</p><h3>About Us</h3><p>Want to be a part of our growing company? We’re hiring! Check out our open positions and other media pages from the links below.</p><p><a href="https://careers.trendyol.com/">Home - Trendyol Careers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00e29b26542f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/how-we-built-an-automated-production-ready-sharded-mongodb-infrastructure-00e29b26542f">How We Built an Automated, Production-Ready Sharded MongoDB Infrastructure</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>