<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neelanjan Manna on Medium]]></title>
        <description><![CDATA[Stories by Neelanjan Manna on Medium]]></description>
        <link>https://medium.com/@neelanjanmanna?source=rss-dcf285ed58a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Lm-2kCnZryT-svVsJ-jWqw.png</url>
            <title>Stories by Neelanjan Manna on Medium</title>
            <link>https://medium.com/@neelanjanmanna?source=rss-dcf285ed58a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:00:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neelanjanmanna/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Distributed Database from Scratch: The ConureDB JourneyThis blog chronicles building…]]></title>
            <link>https://neelanjanmanna.medium.com/building-a-distributed-database-from-scratch-the-conuredb-journeythis-blog-chronicles-building-842e70ad7d95?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/842e70ad7d95</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sat, 23 Aug 2025 18:51:06 GMT</pubDate>
            <atom:updated>2025-08-24T04:20:10.922Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Distributed Database from Scratch: The ConureDB Journey</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UMX4ifBhTmWBEG9_KRTfrw.jpeg" /></figure><p>When I set out to build ConureDB, I wanted to create something meaningful — a persistent, distributed key-value database that could serve as both a learning exercise and a production-ready system. The goal was ambitious: build something similar to etcd, with quorum-based consensus that could survive up to 50% of replicas failing. I essentially knew nothing about the database internals. This blog chronicles the entire journey, from initial planning to the final Kubernetes deployment.</p><h3>1. The Vision: A Resilient Distributed Database</h3><p>The database landscape is dominated by a few key players, but understanding how they work internally requires building one yourself. I wanted ConureDB to have these characteristics:</p><ul><li><strong>Persistent storage</strong>: Data should survive process restarts and system failures</li><li><strong>Distributed architecture</strong>: Multiple replicas working together for high availability</li><li><strong>Consensus-based</strong>: Using Raft algorithm for strong consistency guarantees</li><li><strong>Fault-tolerant</strong>: Surviving up to 50% node failures (N/2–1 failures in an N-node cluster)</li><li><strong>Production-ready</strong>: Complete with APIs, monitoring, and Kubernetes deployment</li></ul><p>The inspirationcame from etcd, which powers Kubernetes’ control plane and has proven itself in countless production environments. However, building from scratch would give me complete control over design decisions and deep understanding of every component.</p><h3>2. Choosing the Right Data Structure: The Storage Engine Decision</h3><p>The heart of any database is its storage engine, and the choice of data structure fundamentally affects performance characteristics. I evaluated several options:</p><h3>B-Tree</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bx5xiA6ppRhzTx5f8JxLVw.gif" /></figure><p>A B-tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time. Unlike binary trees, B-trees can have multiple keys per node and more than two children, making them particularly well-suited for systems that read and write large blocks of data, such as databases and file systems. B-trees have been the backbone of databases for decades. They work by:</p><ul><li><strong>Maintaining sorted order</strong>: Keys are always kept in sorted order, enabling efficient range queries</li><li><strong>Balanced structure</strong>: All leaf nodes are at the same depth, guaranteeing O(log n) operations</li><li><strong>Node-based organization</strong>: Data is organized in pages/nodes, typically matching disk block sizes</li><li><strong>Efficient for reads</strong>: Direct path from root to any key with minimal disk I/O</li></ul><p>B-trees excel at:</p><ul><li><strong>Read performance</strong>: Logarithmic time complexity for point queries</li><li><strong>Range queries</strong>: Excellent for scanning ranges of keys</li><li><strong>Update efficiency</strong>: In-place updates with minimal structural changes</li><li><strong>Memory locality</strong>: Good cache performance due to node-based organisation</li></ul><h3>LSM Trees</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*71hlLKX_rhcgAl6rnqW7iw.png" /></figure><p>An LSM-tree (Log-Structured Merge-tree) is a data structure optimized for write-heavy workloads that maintains data across multiple levels of storage, typically starting with an in-memory component and cascading to disk-based structures. New writes are first stored in memory (often called a memtable), and when this fills up, it’s flushed to disk as an immutable sorted file.</p><p>Over time, these files are merged and compacted in a background process that combines smaller files into larger ones at different levels, maintaining sorted order while removing obsolete or deleted entries. This design provides excellent write performance since writes are sequential and buffered, though it trades off some read performance due to the need to potentially search multiple files.</p><ul><li><strong>Write-optimized</strong>: All writes go to memory (memtable) first</li><li><strong>Immutable files</strong>: Data is organized in immutable, sorted files (SSTables)</li><li><strong>Compaction process</strong>: Background merging of files to maintain performance</li><li><strong>Write amplification</strong>: Trade-off where writes may be amplified during compaction</li></ul><p>LSM trees are ideal for:</p><ul><li><strong>High write throughput</strong>: Sequential writes to immutable files</li><li><strong>Append-heavy workloads</strong>: Perfect for time-series data or logging</li><li><strong>Compression</strong>: Immutable files can be heavily compressed</li></ul><p>However, they have drawbacks:</p><ul><li><strong>Read amplification</strong>: May need to check multiple files for a single key</li><li><strong>Compaction overhead</strong>: Background processes that consume I/O and CPU</li><li><strong>Write amplification</strong>: Data may be rewritten multiple times during compaction</li></ul><h3>Modern Alternatives: Bw-Trees and Hybrid Approaches</h3><h4>Bw-Tree</h4><p>A <a href="https://15721.courses.cs.cmu.edu/spring2016/papers/bwtree-icde2013.pdf">Bw-tree</a> (Buzz Word tree) is a lock-free B+-tree variant designed for high-performance, multi-threaded environments, particularly in main-memory database systems.</p><ul><li>Instead of using traditional locks for concurrency control, the Bw-tree employs a mapping table that indirectly references tree nodes, allowing for atomic updates through compare-and-swap operations on pointers.</li><li>When modifications are needed, rather than updating nodes in-place, the structure creates delta records that are prepended to existing nodes, forming chains of changes that can be later consolidated during page reorganization.</li><li>This approach enables lock-free reads and writes while maintaining the sorted order and logarithmic search properties of B+-trees.</li></ul><p><strong>Advantages</strong></p><ul><li>Lock-free, in-memory B-tree variant</li><li>Uses modern CPU features like compare-and-swap operations</li><li>Eliminates traditional locking mechanisms</li><li>Excellent for multi-core systems with high concurrency</li></ul><h4>B-Tree and LSM Tree Hybrid</h4><p>A hybrid key-value database can be designed to deliver high performance while supporting both LSM-tree and B+ tree data structures. It follows a log-structured approach and functions as an embedded DBMS, making it suitable for direct integration into applications without requiring a separate server process. <a href="https://github.com/lotusdblabs/lotusdb">LotusDB</a> uses this approach.</p><ul><li>The core idea is to combine the strengths of different indexing structures. LSM-trees excel in write-heavy workloads due to their sequential write patterns and compaction strategies, while B+ trees offer efficient point lookups and range queries, making them ideal for read-heavy scenarios. By supporting both, the database allows users to choose the most suitable structure based on workload characteristics.</li><li>As an embedded system, this type of database provides high-performance local storage with minimal operational overhead, positioning it as a more optimized and flexible alternative to traditional Go-based storage engines.</li></ul><p><strong>Advantages</strong></p><ul><li>Combine benefits of both B-trees and LSM trees</li><li>Use LSM-like structure for writes with B-tree-like access patterns</li><li>Attempt to minimize both read and write amplification</li></ul><h3>The Decision: Why B-Trees Won</h3><p>After careful consideration, B-trees was chosen to be used for ConureDB because:</p><ol><li><strong>Proven scalability</strong>: Used by PostgreSQL, MySQL, InnoDB, and countless other systems</li><li><strong>Predictable performance</strong>: No surprise compaction storms or write amplification</li><li><strong>Excellent read performance</strong>: Critical for a consensus-based system where reads need to be fast</li><li><strong>Simpler implementation</strong>: More straightforward to implement correctly</li><li><strong>Better suited for consensus</strong>: Raft requires stable, consistent storage characteristics</li></ol><p>The trade-off was potentially lower write throughput compared to LSM trees, but for a consensus-based system where consistency trumps raw write performance, this was acceptable.</p><h3>3. Concurrency and Performance: Copy-on-Write and Node Pooling</h3><h3>Copy-on-Write: Lock-Free Concurrency</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fV0kmL-vF1WPzUv7IPd14A.png" /></figure><p>Traditional databases use complex locking mechanisms to handle concurrent access, but these can become bottlenecks. <strong>Copy-on-Write (CoW)</strong> is a technique used in databases and file systems to update a B-tree (or B+ tree) without modifying nodes in place. Instead of directly changing the existing tree, the modified nodes are copied, and references are updated to point to these new nodes.</p><ol><li><strong>Immutable reads</strong>: Readers always see a consistent snapshot without locks</li><li><strong>Copy on modification</strong>: Writers create new versions of modified nodes</li><li><strong>Atomic updates</strong>: Root pointer updates make changes visible atomically</li><li><strong>Garbage collection</strong>: Old versions are cleaned up when no longer referenced</li></ol><h3>How It Works:</h3><ol><li><strong>Read-only nodes remain unchanged</strong> → Only the path from the root to the modified leaf is copied.</li><li><strong>Modification</strong> → When inserting/updating/deleting a key, a new copy of the affected leaf (and any nodes on the path up to the root) is created.</li><li><strong>Re-linking</strong> → The new nodes are connected back to the existing unchanged parts of the tree.</li><li><strong>Root update</strong> → A new root pointer references the updated tree version, while the old root still points to the old version.</li></ol><pre>// Example: COW node modification<br>func (t *BTree) insert(node *Node, key, value []byte) (*Node, error) {<br>    // Create a copy of the node (copy-on-write)<br>    nodeCopy, err := t.storage.CloneNode(node)<br>    if err != nil {<br>        return nil, err<br>    }<br>    <br>    // Modify the copy<br>    nodeCopy.AddItem(Item{Key: key, Value: value})<br>    <br>    // Return new version<br>    return nodeCopy, nil<br>}</pre><p><strong>Benefits of COW:</strong></p><ul><li><strong>No read locks</strong>: Readers never block or get blocked</li><li><strong>Consistent snapshots</strong>: Each transaction sees a consistent view</li><li><strong>Simplified concurrency</strong>: No complex lock ordering or deadlock detection</li><li><strong>MVCC-ready</strong>: Natural foundation for multi-version concurrency control</li></ul><p><strong>Trade-offs:</strong></p><ul><li><strong>Memory overhead</strong>: Multiple versions exist simultaneously</li><li><strong>Write amplification</strong>: Need to copy nodes even for small changes</li><li><strong>Garbage collection</strong>: Need to clean up old versions</li></ul><h3>Node Pooling: Optimizing Allocations</h3><p>Database performance is often limited by memory allocation patterns. Node pooling is an optimization technique used to improve the performance and memory efficiency of B-tree (and B+-tree) implementations. Instead of constantly allocating and deallocating memory for nodes during inserts, deletes, and splits, the system maintains a <strong>pool of reusable nodes</strong>.</p><p>Here’s the reasoning and mechanics:</p><h3>Why node pooling?</h3><ul><li><strong>Reduce allocation overhead</strong>: Memory allocations in languages like Go can be expensive under high concurrency. Reusing nodes reduces GC pressure.</li><li><strong>Cache efficiency</strong>: Reusing recently freed nodes often keeps them in CPU caches, improving performance.</li><li><strong>Stability under churn</strong>: Workloads with frequent splits and merges (e.g., random inserts/deletes) benefit since they don’t cause large amounts of garbage.</li></ul><h3>How it works</h3><ol><li><strong>Free list or pool</strong>: Maintain a pool (e.g., a lock-free queue or sync.Pool in Go) of previously used nodes.</li><li><strong>Node creation</strong>: When a new node is required (due to a split), the system first checks the pool; if a free node exists, it reuses it instead of allocating new memory.</li><li><strong>Node deletion/merge</strong>: When a node becomes unnecessary (e.g., after merging), instead of freeing it outright, the system returns it to the pool.</li><li><strong>Pool tuning</strong>: The pool may have a max size to prevent unbounded memory use; beyond that, excess nodes are released to the allocator.</li></ol><pre>type NodePool struct {<br>    nextNodeID   NodeID<br>    freeNodeIDs  []NodeID<br>    mu           sync.Mutex<br>}<br><br>func (np *NodePool) Allocate() NodeID {<br>    np.mu.Lock()<br>    defer np.mu.Unlock()<br>    if len(np.freeNodeIDs) &gt; 0 {<br>        // Reuse freed node ID<br>        nodeID := np.freeNodeIDs[len(np.freeNodeIDs)-1]<br>        np.freeNodeIDs = np.freeNodeIDs[:len(np.freeNodeIDs)-1]<br>        return nodeID<br>    }<br>    <br>    // Allocate new node ID<br>    nodeID := np.nextNodeID<br>    np.nextNodeID++<br>    return nodeID<br>}</pre><p><strong>Node Pool Benefits:</strong></p><ul><li><strong>Reduced allocation overhead</strong>: Reuse existing node IDs</li><li><strong>Better memory locality</strong>: Nodes are allocated in predictable patterns</li><li><strong>Fragmentation reduction</strong>: Minimize memory fragmentation over time</li><li><strong>Faster operations</strong>: Avoid expensive allocation/deallocation syscalls</li></ul><h3>Storage Layer Optimizations</h3><p>The storage layer includes several other optimizations:</p><ul><li><strong>Fixed-size pages</strong>: All nodes are stored in 4KB pages, matching typical filesystem block sizes for optimal I/O performance.</li><li><strong>Memory-mapped I/O</strong>: Critical paths use memory mapping for better performance and simpler code.</li><li><strong>Write-ahead logging</strong>: Although not fully implemented, the foundation exists for WAL-based durability.</li><li><strong>Efficient serialization</strong>: Custom binary format optimized for the B-tree structure, avoiding JSON/protobuf overhead.</li></ul><h3>4. The Complete System: APIs, CLI, and Tooling</h3><p>Beyond the core storage engine, ConureDB includes a complete ecosystem:</p><h3>HTTP API Layer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSEmUbRRfM38-ZUx73j2nQ.gif" /></figure><p>The API layer provides RESTful endpoints for database operations:</p><pre>GET /kv?key=mykey - Retrieve a value<br>PUT /kv?key=mykey - Store a value (body contains the value)<br>DELETE /kv?key=mykey - Delete a key<br>GET /status - Health check and status<br>GET /raft/config - Raft cluster configuration<br>POST /join - Join a node to the cluster</pre><p>The API layer handles:</p><ul><li><strong>Leader routing</strong>: Automatically routes writes to the current Raft leader</li><li><strong>Consistency levels</strong>: Ensures reads reflect the latest committed state</li><li><strong>Error handling</strong>: Proper HTTP status codes and error messages</li><li><strong>Content negotiation</strong>: JSON responses with appropriate headers</li></ul><h3>Command Line Interface (CLI)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpxQABridOl_-G6OYDdEuA.gif" /></figure><ul><li>Interactive shell for database operations</li><li>Tab completion for commands</li><li>Command history and editing</li><li>Direct connection to any cluster node</li></ul><h3>Configuration and Deployment</h3><p>The system supports flexible configuration:</p><pre># config.yaml example<br>node_id: &quot;node-1&quot;<br>data_dir: &quot;/var/lib/conuredb&quot;<br>raft_addr: &quot;node-1:7000&quot;<br>http_addr: &quot;0.0.0.0:8081&quot;<br>bootstrap: false<br>barrier_timeout: &quot;30s&quot;</pre><p>Environment variable overrides allow easy containerization:</p><pre>export CONURE_SEEDS=&quot;node-1:8081,node-2:8081,node-3:8081&quot;<br>export CONURE_NODE_ID=&quot;node-1&quot;</pre><h3>5. Consensus with Raft: The Distributed Systems Heart</h3><h3>Understanding the Raft Algorithm</h3><p>Raft solves the fundamental problem of distributed consensus; how do multiple machines agree on a shared state despite failures and network partitions?</p><p>Raft is a <strong>distributed consensus algorithm</strong> that helps a group of servers (or nodes) agree on a consistent state, even if some of them fail. It’s most often used to manage a replicated log, which is the foundation for building fault-tolerant systems like databases, key-value stores, and configuration services.</p><h3>Key ideas in Raft</h3><h4><strong>Consensus problem</strong></h4><ul><li>In a distributed system, multiple nodes must agree on a single sequence of operations (e.g., writes to a database).</li><li>The challenge is to do this correctly even if nodes crash, restart, or messages get delayed.</li></ul><h4><strong>Leader-based approach</strong></h4><ul><li>Raft elects one node as the <strong>leader</strong>.</li><li>The leader handles all client requests that modify the system state and replicates them to follower nodes.</li><li>Followers are passive: they just replicate and respond to leader heartbeats.</li></ul><h4><strong>Phases of Raft</strong></h4><ul><li><strong>Leader election</strong>: If no leader exists, nodes vote to elect one. Majority wins.</li><li><strong>Log replication</strong>: The leader appends client requests to its log and replicates them to followers. Once a majority acknowledges, the entry is committed.</li><li><strong>Safety</strong>: Raft ensures logs are consistent across nodes. Even if leaders change, committed entries remain safe.</li></ul><h4><strong>Fault tolerance</strong></h4><ul><li>Raft works as long as a <strong>majority</strong> of nodes (N/2 + 1) are available.</li><li>Example: In a 5-node cluster, Raft tolerates 2 node failures.</li></ul><h3>Raft State Machine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*9_FPxVYVAHFaKpkhVRh58A.png" /></figure><p><strong>Leader Election Process:</strong></p><ol><li>Followers timeout waiting for heartbeats</li><li>Follower becomes candidate, increments term, votes for self</li><li>Candidate requests votes from other nodes</li><li>If majority votes received, becomes leader</li><li>Leader sends heartbeats to maintain leadership</li></ol><p><strong>Log Replication:</strong></p><ol><li>Client sends command to leader</li><li>Leader appends entry to local log</li><li>Leader sends AppendEntries RPCs to followers</li><li>Followers append entry and send acknowledgment</li><li>When majority acknowledges, leader commits entry</li><li>Leader notifies followers of commitment in next heartbeat</li></ol><h3>Raft in ConureDB</h3><p>ConureDB uses the HashiCorp Raft library, which provides a production-ready implementation:</p><pre>// Raft configuration<br>config := raft.DefaultConfig()<br>config.LocalID = raft.ServerID(nodeID)<br>config.HeartbeatTimeout = 1000 * time.Millisecond<br>config.ElectionTimeout = 1000 * time.Millisecond<br>config.CommitTimeout = 50 * time.Millisecond<br><br>// Create Raft node<br>r, err := raft.NewRaft(config, fsm, logStore, stableStore, snapshotStore, transport)</pre><p><strong>Finite State Machine (FSM):</strong> The FSM applies committed log entries to the database:</p><pre>func (f *FSM) Apply(l *raft.Log) interface{} {<br>    cmd, err := DecodeCommand(l.Data)<br>    if err != nil {<br>        return err<br>    }<br><br>    switch cmd.Type {<br>    case CmdPut:<br>        return f.DB.Put(cmd.Key, cmd.Value)<br>    case CmdDelete:<br>        return f.DB.Delete(cmd.Key)<br>    default:<br>        return nil<br>    }<br>}</pre><p><strong>Snapshots for Efficiency:</strong> To prevent log growth, Raft takes periodic snapshots:</p><pre>func (f *FSM) Snapshot() (raft.FSMSnapshot, error) {<br>    return &amp;dbSnapshot{db: f.DB}, nil<br>}<br><br>func (s *dbSnapshot) Persist(sink raft.SnapshotSink) error {<br>    return s.db.SnapshotTo(sink)<br>}</pre><p><strong>Quorum Math:</strong> For a cluster of N nodes:</p><ul><li><strong>Quorum size</strong>: (N/2) + 1</li><li><strong>Tolerated failures</strong>: (N-1)/2</li><li><strong>3-node cluster</strong>: Tolerates 1 failure</li><li><strong>5-node cluster</strong>: Tolerates 2 failures</li></ul><h3>Handling Network Partitions</h3><p>Raft’s design ensures safety during network partitions:</p><p><strong>Split-brain prevention</strong>: Only one partition can have a majority, so only one leader can exist.</p><p><strong>Consistency guarantees</strong>: Clients in the minority partition cannot write, ensuring consistency.</p><p><strong>Automatic recovery</strong>: When partitions heal, nodes automatically rejoin and catch up.</p><h3>6. Kubernetes Deployment: Production-Ready Operations</h3><h3>Helm Charts for Easy Deployment</h3><p>ConureDB includes two Helm charts for different deployment scenarios:</p><h4>Single Instance Deployment</h4><p>For development or testing:</p><pre># charts/conuredb-single/values.yaml<br>replicaCount: 1<br>image:<br>  repository: conuredb/conuredb<br>  tag: &quot;latest&quot;<br><br>service:<br>  type: ClusterIP<br>  httpPort: 8081<br>  raftPort: 7000<br><br>persistence:<br>  enabled: true<br>  size: 10Gi<br>  storageClass: &quot;standard&quot;</pre><h4>High Availability Deployment</h4><p>For production with full consensus:</p><pre># charts/conuredb-ha/values.yaml<br>replicaCount: 3  # or 5 for higher availability<br><br># StatefulSet ensures stable network identities<br>statefulset:<br>  enabled: true<br><br># Headless service for peer discovery<br>service:<br>  headless:<br>    enabled: true<br><br># Pod disruption budget prevents majority loss<br>podDisruptionBudget:<br>  enabled: true<br>  minAvailable: 2  # Ensures majority always available</pre><h3>Kubernetes-Native Features</h3><p><strong>StatefulSets</strong>: Provide stable network identities (conure-0, conure-1, conure-2) essential for Raft cluster formation. Unlike Deployments that create pods with random names, StatefulSets guarantee predictable naming that persists across restarts. This is crucial for Raft because each node needs a stable identity that other nodes can reference consistently. When conure-1 restarts, it remains conure-1, allowing the Raft cluster to maintain its membership configuration.</p><p><strong>Headless Services</strong>: Enable direct pod-to-pod communication for Raft traffic while maintaining service discovery. A headless service (with clusterIP: None) doesn&#39;t provide load balancing but instead returns DNS A records pointing directly to individual pod IPs. This is essential for Raft because:</p><ul><li><strong>Direct communication</strong>: Raft nodes need to establish persistent connections to specific peers, not load-balanced connections</li><li><strong>DNS-based discovery</strong>: Pods can discover each other using predictable DNS names like conure-0.conure-headless.default.svc.cluster.local</li><li><strong>Port separation</strong>: The headless service allows Raft traffic (port 7000) to flow directly between pods while client traffic (port 8081) can still use a regular ClusterIP service</li><li><strong>Network topology awareness</strong>: Each Raft node knows exactly which physical pod it’s communicating with, important for leader election and log replication</li></ul><p>For example, when conure-1 needs to send a heartbeat to conure-2, it connects directly to conure-2.conure-headless:7000, bypassing any load balancer that might interfere with the persistent connection Raft requires.</p><p><strong>Pod Disruption Budgets</strong>: Prevent Kubernetes from disrupting too many pods simultaneously, maintaining quorum. A PDB with minAvailable: 2 in a 3-node cluster ensures that during voluntary disruptions (like node maintenance or cluster upgrades), Kubernetes will never terminate more than one ConureDB pod at a time. This guarantees that the Raft cluster always maintains its quorum of 2 nodes, preventing service unavailability during maintenance windows.</p><p><strong>Persistent Volume Claims</strong>: Ensure data persistence across pod restarts and rescheduling. Each StatefulSet pod gets its own PVC (e.g., data-conure-0, data-conure-1) that persists even if the pod is deleted and recreated. This is critical for Raft because each node&#39;s log and state must survive pod restarts. Without persistent storage, a restarted node would lose its Raft log and require a full snapshot transfer to rejoin the cluster.</p><p><strong>Init Containers</strong>: Handle cluster bootstrapping and node discovery automatically. Init containers run before the main ConureDB container and ensure proper startup ordering. For example, voter pods use init containers to wait for the bootstrap pod’s health endpoint (http://conure-bootstrap-0:8081/status) to return successfully before starting their own ConureDB process. This prevents race conditions where nodes try to join a cluster that doesn&#39;t exist yet.</p><h3>Bootstrap Pod Design: Solving the Chicken-and-Egg Problem</h3><p>One of the trickiest challenges in deploying a Raft-based system on Kubernetes is the bootstrap problem: how do you start the first node in a cluster when every node expects to join an existing cluster?</p><p><strong>The Challenge:</strong></p><ul><li>Raft clusters need at least one node to start (the bootstrap node)</li><li>Multiple StatefulSets need coordinated startup to avoid race conditions</li><li>Network timing and ordering must be handled gracefully</li></ul><p><strong>ConureDB’s Two-StatefulSet Strategy:</strong></p><p>ConureDB solves this using separate StatefulSets for different roles:</p><ol><li><strong>Bootstrap StatefulSet</strong> (conure-bootstrap):</li></ol><ul><li>Creates a single pod (conure-bootstrap-0)</li><li>Starts with --bootstrap=true flag</li><li>Establishes the initial cluster state</li></ul><p><strong>2. Voter StatefulSet</strong> (conure):</p><ul><li>Creates the remaining pods (conure-0, conure-1, etc.)</li><li>Uses init containers to wait for bootstrap pod readiness</li><li>Joins the cluster using seeds configuration</li></ul><p><strong>How It Works:</strong></p><p>The bootstrap process follows these steps:</p><ul><li>Bootstrap StatefulSet starts first and creates the initial cluster</li><li>Voter StatefulSet init containers wait for bootstrap pod health checks to pass</li><li>Once conure-bootstrap-0 is ready, voter pods start and join the cluster</li><li>The cluster reaches full consensus with all members participating</li></ul><p><strong>Key Benefits:</strong></p><ul><li><strong>Deterministic Startup</strong>: Clear separation between bootstrap and joining phases</li><li><strong>Race Condition Prevention</strong>: Init containers ensure proper ordering</li><li><strong>Operational Simplicity</strong>: Bootstrap pod can be safely removed after cluster formation</li><li><strong>Rolling Update Safety</strong>: Updates work correctly without disrupting consensus</li></ul><p>This design pattern is essential for production distributed systems because it eliminates the complexity of bootstrap detection logic while ensuring reliable cluster formation.</p><p><strong>Network Delays</strong>: Init containers use retries and timeouts to handle network latency during startup.</p><p><strong>Graceful Degradation</strong>: If bootstrap detection fails, nodes default to join mode, which is safer than accidentally creating multiple clusters.</p><h3>Service Discovery and Auto-Join</h3><p>ConureDB supports Kubernetes-native service discovery:</p><pre>// Auto-discovery using Kubernetes DNS<br>seeds := []string{<br>    &quot;conure-0.conure-headless:8081&quot;,<br>    &quot;conure-1.conure-headless:8081&quot;, <br>    &quot;conure-2.conure-headless:8081&quot;,<br>}<br><br>// Automatic cluster joining<br>if !cfg.Bootstrap {<br>    go joinCluster(cfg.NodeID, cfg.RaftAddr, 2*time.Second, 0)<br>}</pre><p><strong>Benefits of Kubernetes Integration:</strong></p><ul><li><strong>Automatic scaling</strong>: Easy to add/remove nodes</li><li><strong>Health monitoring</strong>: Kubernetes health checks and restart policies</li><li><strong>Load balancing</strong>: Automatic traffic distribution</li><li><strong>Rolling updates</strong>: Zero-downtime deployments</li><li><strong>Resource management</strong>: CPU and memory limits/requests</li></ul><h3>Conclusion: Lessons Learned and Future Directions</h3><p>Building ConureDB from scratch was an incredible learning experience that reinforced several key principles:</p><ul><li><strong>Simplicity wins</strong>: Choosing B-trees over more exotic data structures paid off in implementation simplicity and debuggability.</li><li><strong>Consensus is complex</strong>: Even with a mature Raft library, distributed consensus introduces many edge cases and operational complexities.</li><li><strong>Copy-on-write is powerful</strong>: COW semantics simplified concurrency while maintaining good performance characteristics.</li><li><strong>Operational tooling matters</strong>: The database is only as good as its deployment, monitoring, and management tools.</li><li><strong>Performance optimization is iterative</strong>: Starting with a correct implementation and optimizing bottlenecks as they appear is more effective than premature optimization.</li></ul><h3>Final Thoughts</h3><p>Building a database from scratch demystifies the “magic” behind systems we use every day. While ConureDB may not replace production databases anytime soon, the journey of building it provides invaluable insights into distributed systems, storage engines, and the engineering trade-offs that shape our digital infrastructure.</p><p>The complete source code, documentation, and deployment guides are available on <a href="https://github.com/conuredb/conuredb">GitHub</a>, ready for anyone brave enough to continue this journey of database exploration and learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=842e70ad7d95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Rust by Building a Load Balancer: Insights from EchidnaLB]]></title>
            <link>https://neelanjanmanna.medium.com/exploring-rust-by-building-a-load-balancer-insights-from-echidnalb-7db7a624e476?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7db7a624e476</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sun, 27 Oct 2024 17:10:06 GMT</pubDate>
            <atom:updated>2024-10-27T17:10:06.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuPHO9Nc4hc7MsFn-SukYg.jpeg" /></figure><p>I have been tinkering with Rust for some time now and it was about time to delve deeper into it. <strong>EchidnaLB</strong> is my first experience working with a Rust project. While I initially found the language’s emphasis on safety and its unique approach to variable borrowing and ownership challenging, the learning curve was well worth it. Rust’s low-level control, especially compared to languages like Go and Java, was both demanding and rewarding. Although the build times can be slower as compared to many other higher-level languages, Rust’s performance and reliability are very well appreciated. Now, let’s dive into the project itself: a custom load balancer!</p><p>Load balancing plays a crucial role in the pursuit of fast, efficient web services. <a href="https://github.com/echidna-lb/echidna-lb"><strong>EchidnaLB</strong></a><strong> </strong>— a layer 7 load balancer written in Rust — aims to address these needs with performance and reliability at its core. Leveraging Rust’s memory safety and concurrency strengths, EchidnaLB offers modern load balancing tailored for high-traffic applications.</p><p><a href="https://github.com/echidna-lb/echidna-lb">GitHub - echidna-lb/echidna-lb: A layer 7 load-balancer for HTTP backend servers written in Rust.</a></p><h3>Key Features</h3><p>EchidnaLB is built to support <strong>HTTP/1.1 and HTTP/2</strong> protocols, ensuring compatibility across various clients and backend services. It provides robust <strong>IPv4 and IPv6 support</strong>, allowing flexibility in networking environments, and offers <strong>TLS termination</strong> with configurable SSL certificates for secure HTTPS handling. Additionally, the load balancer supports multiple algorithms, including:</p><ul><li><strong>Round Robin</strong></li><li><strong>Least Connections</strong></li><li><strong>Weighted Round Robin</strong></li><li><strong>IP Hashing</strong></li><li><strong>Least Latency</strong></li></ul><p>Each method addresses unique traffic distribution needs, making it adaptable for various use cases.</p><h4>Configurability &amp; Health Checks</h4><p>EchidnaLB’s configuration file allows users to define backend servers, choose load-balancing algorithms, and set worker counts for optimal performance. Health checks are also configurable, with options for interval and custom route paths, helping monitor and maintain backend server health effectively.</p><h4>Performance &amp; Scalability</h4><p>Built with Rust, EchidnaLB harnesses high concurrency without sacrificing memory safety, making it an excellent choice for scalable, high-performance applications. The support for both HTTP protocols and IP versions further extends its adaptability in distributed systems.</p><h3>Hands-On: Running EchidnaLB Locally</h3><p>Here’s a step-by-step guide for locally setting up three backend servers, configuring EchidnaLB, and testing its load-balancing capabilities.</p><p>For the demo, it is assumed that you’ve already downloaded one of the many pre-built binaries from the <a href="https://github.com/echidna-lb/echidna-lb/releases">GitHub release page</a>. Otherwise, you’d have to <a href="https://github.com/echidna-lb/echidna-lb?tab=readme-ov-file#build-yourself">build it yourself</a>.</p><h4>Step 1: Set Up the Backend Servers</h4><p>We’ll start three simple HTTP servers locally. Each server will respond with “Hello, World! From: &lt;server-port&gt;&quot;.</p><p>Create a Rust file, server.rs, with the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a817e164fe14844c726a883aad519029/href">https://medium.com/media/a817e164fe14844c726a883aad519029/href</a></iframe><p>To start three instances of this server:</p><ol><li>Open three terminal windows.</li><li>In each terminal, run the following commands, replacing &lt;port&gt; with 3001, 3002, and 3003 respectively:</li></ol><pre>PORT=&lt;port&gt; cargo run</pre><p>For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FCRp-RJsAKspXfRPyj9MeQ.png" /></figure><h4>Step 2: Create the Load Balancer Config File</h4><p>Next, we’ll configure EchidnaLB to balance requests across these servers. A very minimal config will be used for this demo.</p><p>Create a YAML file, config.yaml, with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e61bc823efff8a229b025d1d35132c5d/href">https://medium.com/media/e61bc823efff8a229b025d1d35132c5d/href</a></iframe><p>This file specifies the load balancing algorithm (Round Robin), the port where EchidnaLB will listen, and details for the backend server endpoints.</p><h4>Step 3: Start EchidnaLB</h4><p>Now we’re ready to launch EchidnaLB with this configuration.</p><p>In a terminal, navigate to your EchidnaLB project directory and run the following after appropriately updating the config file path:</p><pre>./echidna-lb --config ./config.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgNFtMEHmCptIQevtb5xEQ.png" /></figure><p>EchidnaLB should start up, listening on port 9000.</p><h4>Step 4: Test the Load Balancer</h4><p>With everything running, you can now test the load balancer by making requests to it:</p><pre>curl http://127.0.0.1:9000</pre><p>Since we’re using the Round Robin algorithm, you’ll see the response cycling between the three backend servers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzqtcKrHiQ1PCRPG68DBIg.png" /></figure><h3>Reflecting on the Experience</h3><p>Building EchidnaLB was a great introduction to Rust’s strengths and nuances. I learned to appreciate Rust’s meticulous handling of memory and data safety, and I’m excited to leverage it for more system-level programming. EchidnaLB combines performance, safety, and flexibility — qualities I’m eager to explore further as I continue working with Rust.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7db7a624e476" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extending  kubectl Utility With Plugins]]></title>
            <link>https://neelanjanmanna.medium.com/extending-kubectl-utility-with-plugins-9f332f9fb80?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f332f9fb80</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 15:40:25 GMT</pubDate>
            <atom:updated>2022-09-15T15:51:27.277Z</atom:updated>
            <content:encoded><![CDATA[<h3>Extending kubectl Utility With Plugins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWhimiG4o35Jd6pOdpy3Zw.png" /></figure><h3>Introduction</h3><p>We can all agree on the fact that Kubernetes is quite expansive. It has become the de facto tool for cloud-native application deployment because of its flexibility. One often uses the kubectl CLI to interact with their Kubernetes cluster, and while kubectl is quite extensive in itself with regard to all the operations you can perform with it, its usability can be further extended using plugins.</p><p>kubectl plugins can extend the usability of the CLI tool by adding functional capabilities specific to a Kubernetes application or in general providing more features that can be accessed as kubectl sub-commands. The best part is it does not require editing kubectl’s source code or recompiling it, thereby making the plugins truly modular in the sense that they can be simply plugged in and used.</p><h3>Installing Plugins</h3><p>Although there’s more than one way to distribute and install kubectl plugins, the simplest way is to use <a href="https://krew.sigs.k8s.io/"><strong>krew</strong></a>. It is a package manager for kubectl and makes the installation and management of kubectl plugins a cakewalk. To get started, you would have to first install krew on your machine. You can refer to <a href="https://krew.sigs.k8s.io/docs/user-guide/setup/install/">this</a> installation document to do so.</p><p>Once installed, you shall first index all the available krew plugins, a list of which can be accessed <a href="https://krew.sigs.k8s.io/plugins/">here</a>. To do so, run the following command:</p><pre>kubectl krew update</pre><p>Did you notice how krew itself is a kubectl plugin as well? A plugin to manage all other plugins! If you’re willing to, you can check its source code <a href="https://github.com/kubernetes-sigs/krew">here</a>.</p><p>Now, you can view the list of available plugins with the following command:</p><pre>kubectl krew search</pre><p>To install any plugin, say the <a href="https://github.com/rajatjindal/kubectl-whoami"><strong>whoami</strong></a> plugin, run the following command:</p><pre>kubectl krew install whoami</pre><p>Once installed, you can simply follow the usage guide to run the plugin commands as such:</p><pre>kubectl whoami</pre><p>You can also update, uninstall, and add a custom index for the plugins using krew among other things, for which you can refer to the <a href="https://krew.sigs.k8s.io/docs/">documentation</a>.</p><h3>Developing Plugins</h3><p>Developing plugins for kubectl is a fairly simple process, if we abstract out the business logic of the plugin. The plugin itself is nothing but a standalone CLI application that uses appropriate commands to achieve its intended functionalities. So the question remains how do we add this standalone CLI as a kubectl plugin? Let’s find out.</p><p>I have made a demo plugin called kubectl-count, which we will be using for this small demo. The complete source code can be found here: <a href="https://github.com/neelanjan00/kubectl-count">https://github.com/neelanjan00/kubectl-count</a>. In a nutshell, this plugin allows you to count the instances of Kubernetes resources present in your cluster. So for example, you can count the number of pods in a namespace, the number of nodes in your cluster, or the total number of deployments in all the namespaces of your cluster.</p><p>The source code of the plugin is pretty straightforward, we use the <a href="https://github.com/spf13/cobra">Cobra CLI</a> to create a boilerplate CLI project.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a20783f4d2a9e3ec84fcdfdbc49c6e1/href">https://medium.com/media/6a20783f4d2a9e3ec84fcdfdbc49c6e1/href</a></iframe><p>Then, we add the commands for each of the supported Kubernetes resources for which we will count the resources. We do so using the Kubernetes client-go library. For example, this is the logic for the command for counting the number of pods.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2300fc7edc012c0b939496596e4a946d/href">https://medium.com/media/2300fc7edc012c0b939496596e4a946d/href</a></iframe><p>We are using flags here for specifying the namespace, the label selector, and all-namespaces, just as you would do in kubectl. Also, the aliases for the pods command such as po and pod are also available to be used. Similarly, we have created definitions for other commands as well, corresponding to the resources that they represent.</p><p>Lastly, if you’re wondering how we’re obtaining the Kubernetes client here, we use the existing .kubeconfig file in the machine and use it to generate a client.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29a9a2ef8873d05d804b43beb15606c0/href">https://medium.com/media/29a9a2ef8873d05d804b43beb15606c0/href</a></iframe><p>We can simply run this CLI program as it is like the following:</p><pre>go run kubectl-client.go po</pre><p>The above command should give the number of pods in your default namespace, given that you have a valid Kubernetes cluster config file on your machine.</p><p>Now that we have our plugin ready, how shall we use it along with kubectl? To do so, first, we need to build a binary for this go program and then we need to name the binary file (or an executable shell file, in case you want to make your plugins in bash) file such that the root sub-command of the plugin is preceded by kubectl-. For example, since the root sub-command for this plugin is count, the name of the binary will be kubectl-count. After this, we just need to move this file to the /usr/local/bin directory.</p><p>The naming convention we used here will allow kubectl to search for this plugin and any other plugin that you will be installing, given that the directory /usr/local/bin is in your system path. Now you can simply use this plugin to run the command that you had previously run to count the pods like so:</p><pre>kubectl count po</pre><h3>Conclusion</h3><p>kubectl plugins are an excellent way to increase the usability of the CLI tool. While there’s an ever-soaring list of plugins that you can simply plug and play using krew, it’s also pretty feasible to develop your own plugins which can abridge the lack of any functionality in kubectl without complicating the developer experience or the end-user experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f332f9fb80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Install Drone CI Server in Kubernetes]]></title>
            <link>https://neelanjanmanna.medium.com/how-to-install-drone-ci-server-in-kubernetes-1d1392fea8c4?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d1392fea8c4</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sun, 04 Sep 2022 13:10:07 GMT</pubDate>
            <atom:updated>2022-09-04T13:10:07.351Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*b3VqijuUHh4WxYyL1rzVWg.png" /></figure><p>In this blog, we’ll setup a Drone CI server in Kubernetes using Helm. If you’re a beginner and are lost in the labyrinths of the documentations and GitHub READMEs, this simple blog will bootstrap you in just a few minutes.</p><h3>Introduction</h3><p>Drone by <a href="https://harness.io/">Harness</a> is a continuous integration service that enables you to conveniently set up projects to automatically build, test, and deploy as you make changes to your code. Drone integrates seamlessly with Github, Bitbucket and Google Code as well as third party services such as Heroku, Dotcloud, Google AppEngine and more.</p><p>While Drone also has a cloud SaaS platform, we’ll be focusing on setting up the hosted version on Kubernetes. By default, Drone CI is configured to run as a Docker container and is meant to be setup in a server or a VM. In fact, all the CI build steps are performed using individual Docker containers.</p><p>However it is also possible to setup the Drone CI server on a Kubernetes cluster. Drone also provides a Helm chart to do so, which we will be using here. However this mode of installation is not present in the primary <a href="https://docs.drone.io/">user documentation</a>, hence the requirement of this blog. The Helm chart we will be using is present in <a href="https://github.com/drone/charts">this</a> GitHub repository.</p><h3>Pre-Requisites</h3><ul><li><strong>A Kubernetes Cluster</strong>: Any Kubernetes cluster of sufficient capacity can be used. I will be using a GKE cluster of three e2-micro VMs for the setup.</li><li><strong>Helm</strong>: Ensure that you have Helm installed on your local machine.</li><li><strong>GitHub Account</strong>: For this blog, we’ll be using GitHub for creating an OAuth application for the Drone server.</li><li><strong>kubectl</strong>: We’ll be using kubectl to access some of the Kubernetes cluster info.</li></ul><h3>Step-1: Setting up a GitHub OAuth Application for Drone</h3><p>Before deploying Drone in Kubernetes, we need to create an OAuth application so that the users can authorise access of their GitHub repositories to Drone. We are using GitHub for this purpose here, however this can be achieved using other major Git hosting service providers as well like BitBucket, GitLab among others. Checkout the <a href="https://docs.drone.io/server/overview/">docs</a> for the full list.</p><p>Go to <a href="https://github.com/settings/developers">https://github.com/settings/developers</a> and create a new OAuth application and choose <strong>New OAuth App</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Faz2Eqb-jINoYdGOUaCozQ.png" /><figcaption>Register a new GitHub OAuth application</figcaption></figure><p>Put application name as <strong>Drone</strong>.</p><p>For the Homepage URL, we need to provide the Drone server application endpoint. Depending on how we’re configuring the access to the application in Kubernetes i.e. either using a NodePort or a LoadBalancer type of service, we will obtain the URL.</p><p>We’ll be using a NodePort type of service for the purpose of this demo. Currently, we don’t have the Drone server deployed in Kubernetes and hence the NodePort service isn’t available to us just yet. However, we can choose an unused NodePort value right away which will be later used for the server deployment. I will be using the port <strong>32000</strong>, as an example.</p><p>With the NodePort value decided upon, we can simply use it with the external IP of any of the Kubernetes nodes to obtain the Homepage URL. In other words, this will be the endpoint of the Drone server once we install it. To get the external IP of the nodes, we can use the following command:</p><pre>kubectl get nodes -o wide | awk &#39;{ print $7 }&#39;</pre><p>You’ll obtain a similar result:</p><pre>EXTERNAL-IP</pre><pre>35.238.118.197</pre><pre>34.68.253.216</pre><pre>35.222.5.86</pre><p>You may choose any one of the external IPs available to you. For this demo I’ll be using the <strong>35.238.118.197 </strong>IP. Hence, our Homepage URL becomes <strong>http://35.238.118.197:32000 </strong>and for the authorisation callback URL, we will use <strong>http://35.238.118.197:32000/login</strong>. Optionally, feel free to add any description that seems fit to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQnbcAQ8VfmIaevS6IgPtw.png" /></figure><p>Once done, choose <strong>Register application</strong>. This should register your Drone application with the GitHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByQtw4YyNIwHqgWIIKH4AQ.png" /><figcaption>Registration successful</figcaption></figure><p>Now all we need to do is obtain a client secret key so that it can be used by Drone to authorize the login requests via GitHub OAuth. To do so, choose <strong>Generate a new client secret</strong>. This should prompt GitHub to authenticate you so that client secret key creation can be validated. Once done, you’ll obtain the secret key. Copy and save the key somewhere else as you won’t be able to access it from GitHub the next time, when we’ll make use of it to configure the Drone server. Also take a note of the client ID.</p><h3>Step-2: Configuring the Drone server Helm chart</h3><p>Before we configure the deployment options, we will add the Helm repo. To do so, execute the following commands:</p><pre>helm repo add drone <a href="https://charts.drone.io/">https://charts.drone.io</a></pre><pre>helm repo update</pre><p>Once done, we can configure the server deployment options. To do so, we’ll use a YAML file, which can be obtained <a href="https://github.com/drone/charts/blob/master/charts/drone/values.yaml">here</a>. Download and save the file by the name values.yaml and open it using any text editor. We’ll be doing a minimal configuration setup in this demo, though you can explore all the other configuration options as well.</p><p>First, we’ll modify the Kubernetes service to be used for the server deployment. As decided earlier, we’ll be using a NodePort type of service, therefore service.type field value will be NodePort. Also, for the service.nodePort field, we will use the value 32000.</p><pre>...<br>service:</pre><pre>  type: NodePort</pre><pre>  port: 8080</pre><pre>  annotations: {}</pre><pre>  nodePort: 32000</pre><pre>...</pre><p>Next, we’ll configure the Drone specific options, which will be provided as environment variables. The first env is DRONE_SERVER_HOST, for which we will provide the external IP that we had selected earlier. Ensure that the IP address value is wrapped in quotes, to specify it as a string.</p><p>For the DRONE_SERVER_PROTO env, provide the value http as we will be using the HTTP protocol for the Drone server requests and runner polling.</p><p>For the DRONE_RPC_SECRET env, we need to provide a secret value that the Drone server and runners will share and use for authenticating the requests. To generate this value, you may use the following command to get a 32 character length secret:</p><pre>openssl rand -hex 16</pre><p>You can then provide this value as a string for the env.</p><p>Lastly, we will provide the GitHub OAuth application client ID and client secret key which we had created in the first step for the DRONE_GITHUB_CLIENT_ID and DRONE_GITHUB_CLIENT_SECRET envs respectively.</p><pre>...</pre><pre>env:</pre><pre>  DRONE_SERVER_HOST: &quot;35.238.118.197&quot;</pre><pre>  DRONE_SERVER_PROTO: http</pre><pre>  DRONE_RPC_SECRET: &quot;727b7fe17e8de56689f46943c76d25f8&quot;<br>  <br>  DRONE_GITHUB_CLIENT_ID: 620a9d86236b7470558a</pre><pre>  DRONE_GITHUB_CLIENT_SECRET: 35e31b8fddd16c2cb3b681e2a4b84a1ee9c8ce57</pre><pre>...</pre><p>Save and close the file.</p><h3>Step-3: Create the Drone server deployment in Kubernetes</h3><p>Open a terminal in the directory where your values.yaml file is located.</p><p>Firstly we’ll create the namespace where the deployment will take place with the following command. I will be using the drone namespace here.</p><pre>kubectl create namespace drone</pre><p>Next, run the following command which will install the Drone server in Kubernetes:</p><pre>helm install --namespace drone drone drone/drone -f values.yaml</pre><p>It will take a while for the command to execute and setup all the Kubernetes resources. Upon its completion, you’ll notice a similar output:</p><pre>NAME: drone<br>LAST DEPLOYED: Sun Sep  4 17:39:07 2022<br>NAMESPACE: drone<br>STATUS: deployed<br>REVISION: 1<br>TEST SUITE: None<br>NOTES:<br>1. Get the application URL by running these commands:<br>  export NODE_PORT=$(kubectl get --namespace drone -o jsonpath=&quot;{.spec.ports[0].nodePort}&quot; services drone)<br>  export NODE_IP=$(kubectl get nodes --namespace drone -o jsonpath=&quot;{.items[0].status.addresses[0].address}&quot;)<br>  echo <a href="http://$NODE_IP:$NODE_PORT">http://$NODE_IP:$NODE_PORT</a></pre><p>This indicates that the server is successfully deployed in Kubernetes. We can validate that all the Kubernetes resources have been successfully setup using the following command:</p><pre>kubectl get all --namespace drone</pre><p>This should give you a similar output:</p><pre>NAME                         READY   STATUS    RESTARTS   AGE<br>pod/drone-77cd496d5d-jf7gc   1/1     Running   0          3m26s</pre><pre>NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE<br>service/drone   NodePort   10.28.177.200   &lt;none&gt;        8080:32000/TCP   3m31s</pre><pre>NAME                    READY   UP-TO-DATE   AVAILABLE   AGE<br>deployment.apps/drone   1/1     1            1           3m32s</pre><pre>NAME                               DESIRED   CURRENT   READY   AGE<br>replicaset.apps/drone-77cd496d5d   1         1         1       3m34s</pre><h3>Step-4: Access Drone server UI</h3><p>Before accessing the Drone server UI, make sure that if there’s a Firewall for the Kubernetes nodes then there’s an ingress rule allowing all the http traffic on port <strong>32000</strong>.</p><p>This would allow us to reach the server endpoint using the NodePort and node external IP. For example, I will be accessing <strong>http://35.238.118.197:32000 </strong>URL in my browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8B0d55buJXylyRyXWopUA.png" /><figcaption>Drone CI server UI</figcaption></figure><p>We can successfully access the Drone server UI now. Choose <strong>CONTINUE </strong>to proceed with the user authentication. Upon doing so, you’ll be prompted for GitHub OAuth authorization. This is by virtue of the OAuth application that we had created earlier in GitHub. Once done, you’ll be prompted to fill up your user details. Once done, choose <strong>SUBMIT</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9tzWLQTKB80FCPr2aFDhXg.png" /><figcaption>Update user details</figcaption></figure><p>You’ll be then able to access the Drone server dashboard, where all your repositories can be accessed, by virtue of the GitHub OAuth which gives permission to the Drone application to access all your repos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mbF0NFFp0xeihou47z-vlQ.png" /><figcaption>User GitHub repositories</figcaption></figure><h3>Summary</h3><p>In summary, we observed how Drone CI can be setup in Kubernetes using Helm. We created a GitHub OAuth application to authorize the users and provide access to the repositories. Then, we configured the server deployment by providing the relevant details using a config file and finally we used that config file to create the server deployment in Kubernetes using Helm, which allowed us to access the Drone server UI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d1392fea8c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP IAM Integration for LitmusChaos with Workload Identity]]></title>
            <link>https://medium.com/litmus-chaos/gcp-iam-integration-for-litmuschaos-with-workload-identity-50b7cb082491?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/50b7cb082491</guid>
            <category><![CDATA[litmuschaos]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[workload-identity]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sun, 04 Sep 2022 05:52:47 GMT</pubDate>
            <atom:updated>2022-09-04T05:52:47.120Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wjuo-fPRZUQj-4ma6A3iLQ.png" /></figure><p>In this blog, we’ll take a look at how to do a GCP IAM integration for LitmusChaos using Workload Identity for executing the GCP experiments in a keyless manner when using the Google Kubernetes Engine (GKE) as the execution plane.</p><h3>Introduction</h3><p>To execute LitmusChaos GCP experiments, one needs to authenticate with GCP using a service account before trying to access the target resources. Usually, you have only one way of providing the service account credentials to the experiment, using a service account key, but if you’re using a GKE cluster you have a keyless medium of authentication as well. Therefore you have two ways of providing the service account credentials to your GKE cluster:</p><ul><li><strong>Using Secrets:</strong> As you would normally do, you can create a secret containing the GCP service account in your GKE cluster, which gets utilized by the experiment for authentication to access your GCP resources.</li><li><strong>IAM Integration:</strong> When you’re using a GKE cluster, you can bind a GCP service account to a Kubernetes service account as an IAM policy, which can be then used by the experiment for keyless authentication using <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity">GCP Workload Identity</a>. We’ll discuss more on this method in the following sections.</li></ul><h3>Why use Workload Identity for GCP authentication?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmyaNfpfmeC90n0bUTLExA.png" /><figcaption>Places where a service account key might exist</figcaption></figure><p>A Google API request can be made using a GCP IAM service account, which is an identity that an application uses to make calls to Google APIs. You might create individual IAM service accounts for the users who execute GCP experiments to enforce role-based access control, then download and save the keys as a Kubernetes secret that you manually rotate. Not only is this time-consuming, but service account keys only last ten years (or until you manually rotate them). An unaccounted-for key could give an attacker extended access in the event of a breach or compromise. Using service account keys as secrets is not an optimal way of authenticating GKE workloads due to this potential blind spot and the management cost of key inventory and rotation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*REe_OptsdqI0Sh122Ijn9g.png" /><figcaption>Workload Identity in a nutshell</figcaption></figure><p>Workload Identity allows you to restrict the possible “blast radius” of a breach or compromise while enforcing the principle of least privilege across your environment. It accomplishes this by automating workload authentication best practices, eliminating the need for workarounds, and making it simple to implement recommended security best practices.</p><ul><li>Your tasks will only have the permissions they require to fulfill their role with the principle of least privilege. It minimizes the breadth of a potential compromise by not granting broad permissions.</li><li>Unlike the 10-year lifetime service account keys, credentials supplied to the Workload Identity are only valid for a short time, decreasing the blast radius in the case of a compromise.</li><li>The risk of unintentional disclosure of credentials due to a human mistake is greatly reduced because Google controls the namespace service account credentials for you. It also eliminates the need for you to manually rotate these credentials.</li></ul><h3>Enabling service accounts to access GCP resources</h3><h4>STEP 1: Enable Workload Identity</h4><p>You can enable Workload Identity on clusters and node pools using the Google Cloud CLI or the Google Cloud Console. Workload Identity <strong>must</strong> be enabled at the cluster level before you can enable Workload Identity on node pools. Workload Identity can be enabled for an existing cluster as well as a new cluster.</p><p>To enable Workload Identity on a new cluster using the console, choose to create a GKE cluster, and aside from the usual configuration, simply enable the Workload Identity under security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZzVg8rLDk1O5YQkFNRVzQ.png" /><figcaption>Enable Workload Identity in GCP Cloud Console</figcaption></figure><p>You can also use the gcloud tool to create the Kubernetes cluster with Workload Identity enabled using the following command:</p><pre>gcloud container clusters create CLUSTER_NAME \    --region=COMPUTE_REGION \    --workload-pool=PROJECT_ID.svc.id.goog</pre><p>Replace the following:</p><ul><li>CLUSTER_NAME: the name of your new cluster.</li><li>COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.</li><li>PROJECT_ID: your Google Cloud project ID.</li></ul><p>You can enable Workload Identity on an existing Standard cluster by using the gcloud CLI or the Cloud Console. Existing node pools are unaffected, but any new node pools in the cluster use Workload Identity by the modification of the same setting as shown above in the Cloud Console. Alternatively, you can use the following command to achieve it:</p><pre>gcloud container clusters update CLUSTER_NAME \<br>   --region=COMPUTE_REGION \<br>   --workload-pool=PROJECT_ID.svc.id.goog</pre><p>Replace the following:</p><ul><li>CLUSTER_NAME: the name of your new cluster.</li><li>COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.</li><li>PROJECT_ID: your Google Cloud project ID.</li></ul><h4>STEP 2: Configure LitmusChaos to use Workload Identity</h4><p>Assuming that you already have LitmusChaos installed in your GKE cluster as well as the Kubernetes service account you want to use for your GCP experiments, execute the following steps.</p><ol><li>Get Credentials for your cluster.</li></ol><pre>gcloud container clusters get-credentials CLUSTER_NAME</pre><p>Replace CLUSTER_NAME with the name of your cluster that has Workload Identity enabled.</p><p>2. Create an IAM service account for your application or use an existing IAM service account instead. You can use any IAM service account in any project in your organization. For Config Connector, apply the IAMServiceAccount object for your selected service account. To create a new IAM service account using the gcloud CLI, run the following command:</p><pre>gcloud iam service-accounts create GSA_NAME \<br>--project=GSA_PROJECT</pre><p>Replace the following:</p><ul><li>GSA_NAME: the name of the new IAM service account.</li><li>GSA_PROJECT: the project ID of the Google Cloud project for your IAM service account.</li></ul><p>Alternatively, you can also use the Cloud Console UI to create a new GCP IAM Service Account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsnVn1bWfZYu1tz-e6goJQ.png" /><figcaption>IAM Service Accounts in GCP Cloud Console</figcaption></figure><p>3. Ensure that this service account has all the roles required for interacting with the Compute Engine resources including VM Instances and Persistent Disks according to the GCP experiments that you’re willing to run. You can grant additional roles either using the Cloud Console or using the following command:</p><pre>gcloud projects add-iam-policy-binding PROJECT_ID \    <br>--member &quot;serviceAccount:GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com&quot; \<br>--role &quot;ROLE_NAME&quot;</pre><p>Replace the following:</p><ul><li>PROJECT_ID: your Google Cloud project ID.</li><li>GSA_NAME: the name of your IAM service account.</li><li>GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.</li><li>ROLE_NAME: the IAM role to assign to your service account, like roles/spanner.viewer.</li></ul><p>4. Allow the Kubernetes service account to be used for the GCP experiments to impersonate the GCP IAM service account by adding an <a href="https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/add-iam-policy-binding">IAM policy binding</a> between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.</p><pre>gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \<br>   --role roles/iam.workloadIdentityUser \<br>   --member &quot;serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]&quot;</pre><p>Replace the following:</p><ul><li>GSA_NAME: the name of your IAM service account.</li><li>GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.</li><li>KSA_NAME: the name of the Kubernetes service account to be used for LitmusChaos GCP experiments. If you’re using ChaosCenter, then by default the litmus-admin service account is used for all the experiments.</li><li>NAMESPACE: the namespace in which the Kubernetes service account to be used for LitmusChaos GCP experiments is present. If you’re using ChaosCenter, then by default litmus is the namespace.</li></ul><p>5. Annotate the Kubernetes service account to be used for LitmusChaos GCP experiments with the email address of the GCP IAM service account.</p><pre>kubectl annotate serviceaccount KSA_NAME \<br>   --namespace NAMESPACE \<br>iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com</pre><p>Replace the following:</p><ul><li>KSA_NAME: the name of the Kubernetes service account to be used for LitmusChaos GCP experiments. If you’re using ChaosCenter, then by default the litmus-admin service account is used for all the experiments.</li><li>NAMESPACE: the namespace in which the Kubernetes service account to be used for LitmusChaos GCP experiments is present. If you’re using ChaosCenter, then by default litmus is the namespace.</li><li>GSA_NAME: the name of your IAM service account.</li><li>GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.</li></ul><h4>STEP 3: Update ChaosEngine Manifest</h4><p>When creating a new workflow with GCP experiments, edit the manifest YAML and add the following value to the ChaosEngine manifest field .spec.experiments[].spec.components.nodeSelector to schedule the experiment pod on nodes that use Workload Identity:</p><p>iam.gke.io/gke-metadata-server-enabled: &quot;true&quot;</p><p>As an example, say you’re adding the GCP VM Instance Stop experiment, you’d make the following change:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*id8oMvx3S32DFeZHH1WTrg.png" /><figcaption>Update the nodeSelector field for an experiment in the ChaosEngine Manifest</figcaption></figure><h4>STEP 4: Update ChaosExperiment Manifest</h4><p>Remove cloud-secret at .spec.definition.secrets in the ChaosExperiment manifest as we are not using a secret to provide our GCP Service Account credentials.</p><p>As an example, say you’re adding the GCP VM Instance Stop experiment, you’d remove the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5iYYrPyPGqJT2aqffFs8lw.png" /><figcaption>Remove the Kubernetes secret for GCP SA from ChaosEngine Manifest</figcaption></figure><p>Now you can run your GCP experiments with a keyless authentication provided by GCP using Workload Identity.</p><h3>Summary</h3><p>To conclude, we were able to set up a keyless authentication medium for the GCP experiments executing in a GKE that belongs to the target GCP environment. We saw how we can leverage Workload Identity for the GKE node pools to assign an IAM identity to our Kubernetes workloads, which was then used for the authentication of LitmusChaos GCP experiments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50b7cb082491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/litmus-chaos/gcp-iam-integration-for-litmuschaos-with-workload-identity-50b7cb082491">GCP IAM Integration for LitmusChaos with Workload Identity</a> was originally published in <a href="https://medium.com/litmus-chaos">Litmus-Chaos</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Did I Contribute to the LitmusChaos Project for Hacktoberfest 2021]]></title>
            <link>https://neelanjanmanna.medium.com/why-did-i-contribute-to-the-litmuschaos-project-for-hacktoberfest-2021-1eaacdba0a03?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1eaacdba0a03</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[litmuschaos]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 09:43:06 GMT</pubDate>
            <atom:updated>2021-12-06T09:43:06.843Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Av_vkV5Snp3cPDDtevTOA.png" /></figure><p>For the eighth edition of Hacktoberfest, I chose to contribute to <a href="https://litmuschaos.io/">LitmusChaos</a>, a CNCF sandboxed project for Cloud-Native Chaos Engineering. It was a month-long celebration of making Chaos Engineering simpler for all the SREs and Developers who aspire to make their services more resilient. So if you are a software developer like me, then why should you consider contributing to the <a href="https://github.com/litmuschaos/litmus">LitmusChaos project</a>?</p><h4>1. Buzzing Community</h4><p>The Litmus community boasts of some 1000+ community members who profoundly promote the Cloud-Native resiliency paradigm shift by redefining the Chaos Engineering experience for everyone. It is the community members who bring out the best qualities of Chaos Engineering for their distinct use cases by leveraging the diverse set of chaos experiments offered by LitmusChaos. The regular community meetups and sync-ups highlight these interesting use cases, which helps every community member to learn from each other.</p><h4>2. Excellent Code Quality</h4><p>LitmusChaos is an awesome project for beginners who want to kick-start their journey into the world of open-source as the LitmusChaos project has been developed and contributed by all kinds of developers belonging to a varied number of organizations. This has enabled a robust project codebase that adheres to the best coding practices and is easy to understand and contribute. Further, the LitmusChaos project makes use of a wide range of open-source tools such as Kubernetes, GraphQL, Argo Workflows, React JS, etc. which allows every developer to contribute to something of their own choice.</p><h4>3. Awesome Documentation</h4><p>The Litmus project features extensive <a href="https://docs.litmuschaos.io/">documentation</a> that encompasses every single aspect of the project. It aids the developers who want to gain a deeper insight into the project and want to contribute to it with its detailed user documentation, experiment documentation, and API reference. Contributing to the documentation and tutorials is also an excellent option, which helps LitmusChaos to be used by more community members and end-users.</p><h4>4. Flexible Litmus SDK</h4><p><a href="https://medium.com/@ispeakc0de/getting-started-with-litmus-sdk-f2d1fc8c8dea">Litmus SDK</a> allows developers to define their chaos experiments for the LitmusChaos framework. It helps developers to easily bootstrap the experiment files where the SDK is responsible for generating all the requisite artifacts and the developer is only responsible for defining the experiment business logic. The best part about Litmus SDK is that it’s available in multiple programming languages such as Go, Python, and Ansible, allowing developers to develop chaos experiments in any programming language of their choice.</p><h4>5. Mentorship Programs</h4><p>Apart from Hacktoberfest, LitmusChaos also takes part in all the major open-source mentorship programs such as Google Summer of Code (GSoC), GitHub India Externship, The Linux Foundation Mentorship Program (LFX Mentorship), and Google Summer of Docs (GSoD) among the others. These mentorship programs provide a very lucrative opportunity to the mentees for not only contributing to the open-source but also to learn and gain recognition as a quality developer since you’ll be making very significant contributions to the Litmus project under the guidance of a LitmusChaos project maintainer.</p><h3>Join the Litmus Community:</h3><p>Want to get help with queries, learnings, &amp; contributions? Join the Litmus community on slack. To join the slack community please follow the following steps:<br><strong>Step 1:</strong> Join the Kubernetes slack using the following link: <a href="https://slack.k8s.io/">https://slack.k8s.io/</a><br><strong>Step 2:</strong> Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: <a href="https://slack.litmuschaos.io/">https://slack.litmuschaos.io/</a></p><p>Looking forward to seeing you in the world of Open Source!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1eaacdba0a03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP VM Disk Loss Experiment for LitmusChaos]]></title>
            <link>https://medium.com/litmus-chaos/gcp-vm-disk-loss-experiment-for-litmuschaos-881e8edcc4b5?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/881e8edcc4b5</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sun, 25 Jul 2021 14:51:35 GMT</pubDate>
            <atom:updated>2021-08-13T11:32:56.745Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfGgphZRr5SJqWm0kQf9Ag.png" /></figure><p>In this beginner-friendly blog, we’ll be going through the GCP VM Disk Loss experiment for LitmusChaos. GCP VM Disk Loss experiment causes detachment of a non-boot persistent storage disk from a GCP VM instance for a specified duration of time and later re-attaches the disk to its respective VM instance. The broad objective of this experiment is to extend the principles of cloud-native chaos engineering to non-Kubernetes targets while ensuring resiliency for all kinds of targets, be it Kubernetes or non-Kubernetes ones, as a part of a single chaos workflow for the entirety of a business.</p><p>At the time of writing this blog, the experiment is available only as a technical preview in the chaos hub, but in the upcoming releases, the experiment will surely become an integral part of the chaos hub. That being said, we can still access and execute the experiment without any problem, as I am about to show you in this blog.</p><h3>Pre-Requisites</h3><p>Before we begin with the steps of the experiment, let’s check the pre-requisites for performing this experiment:</p><ol><li>A GCP project containing the target Persistent Storage Disks attached to their respective VM instances</li><li>A GCP Service Account having project level Editor or Owner permissions</li><li>A Kubernetes cluster with Litmus 2.0 installed</li></ol><h3>STEP 1: Updating The Chaos Hub</h3><p>Browse and log in to your Chaos Center. You should be on the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzLBHmn3oBPVgI5h4Y2TPA.png" /></figure><p>Select ChaosHubs. Here you’d be able to see the default Chaos Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sWZo_yfY7kXhe9trvel16w.png" /></figure><p>Choose to Edit the default Chaos Hub and instead of the v1.13.x branch, choose the master branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AfHG59P1SNOYqE5sFHUmIQ.png" /></figure><p>Click Submit Now. Now you’d be able to access all the experiments, even those under the technical preview. To confirm that the experiments have been added successfully, click on Chaos Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sb5sIOS1NN0P96urC5rbGA.png" /></figure><p>You should see the GCP Experiments listed here. Now we are all set to begin the steps of the experiment.</p><h3>STEP 2: Setting Up the Chaos Experiment</h3><p>We’d be using the <a href="https://docs.litmuschaos.io/docs/next/gcp-vm-disk-loss/">experiment docs</a> to help us with a few steps.</p><p>In this demo, we will inject chaos into two non-boot persistent storage disks named test-disk and test-disk-1, attached to the VM instances named test-instance and test-instance-1 respectively, the disks are located in the zones us-central1-a and us-central1-b respectively, and belong to the GCP project “Litmus GCP Instance Delete” with the ID of litmus-gcp-instance-delete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBRTDWgbD90b8FEcPvbCPg.png" /></figure><p>Please notice that both the disks are initially attached to their respective VM instances, before the injection of the chaos. Now that we have our disks ready, we can set up our experiment. Before scheduling the chaos experiment, we need to make the GCP Service Account credentials available to Litmus, so that the instances can be shut down and later started as part of the experiment. To do that, we’ll make a Kubernetes secret named secret.yaml as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb4705fc5d57c09542465fbd222c94d3/href">https://medium.com/media/fb4705fc5d57c09542465fbd222c94d3/href</a></iframe><p>The format of this secret is also available in the experiment docs. Make sure the name of the secret is cloud-secret and replace the respective fields of the secret with your own service account credentials. Once done, apply the secret in the litmus namespace using the command:</p><pre>kubectl apply -f secret.yaml -n litmus</pre><p>Once the secret is applied, we’re all set to schedule our experiment from the Chaos . In Dashboard, click on the Schedule a Workflow button. In the workflow creation page, choose the self-agent and click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9DdkclS0Y02bz1xjz8lMOA.png" /></figure><p>In the Choose a Workflow page, select “Create a new workflow using the experiments from MyHub” and select Chaos Hub in the dropdown. Then click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAjvh0O642_JMYPS-DZENQ.png" /></figure><p>In the Workflow Settings page, fill in the workflow name and description of your choice. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bi2b0I80dkeqOXt5U41s1A.png" /></figure><p>In the Tune Workflow page, click on “Add a new experiment” and choose gcp/gcp-vm-disk-loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2kdtzLJI2m47GPOHI_7n1A.png" /></figure><p>Click Done. Notice that the experiment has been added to the experiment graph diagram. Now click on “Edit YAML”. Here we will edit the workflow manifest to specify the experiment resource details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlVE5sJhUdJXsk7_oq61Rg.png" /></figure><p>Scroll down to the manifest of the ChaosExperiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2BRfKJPwPffHD07dCxZsQA.png" /></figure><p>Notice that the name of the secret that we had previously created is being passed to the ChaosExperiment to be mounted at the path /tmp/.</p><p>Scroll further down and similarly fill in the relevant experiment details in the manifest of the ChaosEngine as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FcyyleBaKJ8eXVxQEdLCig.png" /></figure><p>Please take note that the zone for each target disk is to be mentioned in DISK_ZONES in the same order of the DISK_VOLUME_NAMES. Similarly, the device name for each target disk is to be mentioned in DEVICE_NAMES in the same order of the DISK_VOLUME_NAMES. If you like, feel free to modify the parameters of the experiment such as the RAMP_TIME, TOTAL_CHAOS_DURATION, etc. As you would have noticed, some of the experiment tunables are common for both the ChaosEngine and ChaosExperiment, and the values of ChaosExperiment get overridden by that of the values of the ChaosEngine if they differ in both the manifests. Once you have made the changes, click Save Changes. We’ve now specified all the experiment details and are ready to go to the next step. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXXQ420X5fK2NmDitWI11Q.png" /></figure><p>In the Reliability Score, we will use the default score of 10. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zfbXioIzq1nGgyF0K-B-vA.png" /></figure><p>In Schedule, click Schedule Now. Click Next. On the Verify and Commit page verify all the details and once satisfied click on Finish. We’ve successfully scheduled our chaos experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nZ5-XUjx8nZjpRNs67psGA.png" /></figure><h3>STEP 3: Observing the Chaos</h3><p>Click on Go to Workflow and choose the workflow that we just created. Here we can observe the different steps of the workflow execution including chaos experiment installation, chaos injection, and chaos revert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMRH-6hpqbJ_ZlMgjqgxAg.png" /></figure><p>You can also determine if the chaos injection has taken place and as a result, the disks have detached from their respective VM instances or not from the GCP Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nb9qvAB1uTvS-GVTacpOdw.png" /></figure><p>We can also view the Table View for the experiment logs as the experiment proceeds through the various steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBQSQjagHjap2C6l-YvRZA.png" /></figure><p>Once completed, the workflow graph should have executed all the steps successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZhvkkeL44CNfNO-mHugnkw.png" /></figure><p>We have got a 100% Resiliency Score. We can also check the ChaosResult verdict which should say the experiment has passed. The Probe Success Percentage should be 100% as all our disks re-attached successfully post their detachment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g9isFtuKIk9qfZYKAQbfSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bWmNH990s7OSZYLR64lifw.png" /></figure><p>Again you can check in the GCP console if the disks have been re-attached or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TVypKu_2hSIciXq5WpSimQ.png" /></figure><p>We can also perform post chaos analysis for the experiment results in the Analytics section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKX3vfjSJp3S-GuQCT1OmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFoI_6KZMePZRlG2LWbX1w.png" /></figure><p>In conclusion of this blog, we saw how we can perform the GCP VM Disk Loss chaos experiment using Litmus Chaos 2.0. This experiment is only one of the many experiments for the Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, Azure, VMWare, and many more, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used by them.</p><p>Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. To join the Litmus community:<br><strong>Step 1:</strong> Join the Kubernetes slack using the following link: https://slack.k8s.io/<br><strong>Step 2:</strong> Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: <a href="https://slack.litmuschaos.io/">https://slack.litmuschaos.io/</a></p><p>Show your ❤️ with a ⭐ on our <a href="https://github.com/litmuschaos/litmus">Github</a>. To learn more about Litmus, check out the <a href="https://docs.litmuschaos.io/">Litmus documentation</a>. Thank you! 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=881e8edcc4b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/litmus-chaos/gcp-vm-disk-loss-experiment-for-litmuschaos-881e8edcc4b5">GCP VM Disk Loss Experiment for LitmusChaos</a> was originally published in <a href="https://medium.com/litmus-chaos">Litmus-Chaos</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP VM Instance Stop Experiment for LitmusChaos]]></title>
            <link>https://medium.com/litmus-chaos/gcp-vm-instance-stop-experiment-for-litmuschaos-3ea5b58266b2?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ea5b58266b2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[litmuschaos]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Sat, 24 Jul 2021 10:42:56 GMT</pubDate>
            <atom:updated>2021-08-13T11:51:06.812Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CP6mn8x8Jcozi1ssvID16w.png" /></figure><p>This blog is a beginner-friendly guide for the GCP VM Instance Stop chaos experiment for LitmusChaos. The experiment causes the shutdown of one or more GCP VM instances for a specified duration of time and later restarts them. The broad objective of this experiment is to extend the principles of cloud-native chaos engineering to non-Kubernetes targets while ensuring resiliency for all kinds of targets, be it Kubernetes or non-Kubernetes ones, as a part of a single chaos workflow for the entirety of a business.</p><p>At the time of writing this blog, the experiment is available only as a technical preview in the chaos hub, but in the upcoming releases, the experiment will surely become an integral part of the chaos hub. That being said, we can still access and execute the experiment without any problem, as I am about to show you in this blog.</p><h3>Pre-Requisites</h3><p>Before we begin with the steps of the experiment, let’s check the pre-requisites for performing this experiment:</p><ol><li>A GCP project containing the target VM instances</li><li>A GCP Service Account having sufficient permissions to stop or start the VM Instances</li><li>A Kubernetes cluster with Litmus 2.0 installed</li></ol><h3>STEP 1: Updating The Chaos Hub</h3><p>Browse and log in to your Litmus portal. You should be on the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UT3U9G55m954-R7zArDA4w.png" /></figure><p>Select ChaosHubs. Here you’d be able to see the default ChaosHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elFfaTBr6HQsurUKzSS97A.png" /></figure><p>Choose to Edit the default Chaos Hub and instead of the v1.13.x branch, choose the master branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHx_njzV2zVPFSdUs58Kig.png" /></figure><p>Click Submit Now. Now you’d be able to access all the experiments, even those under the technical preview. To confirm that the experiments have been added successfully, click on Chaos Hub and view the Chaos Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRvEuqOBjPO9RhN42UaeTw.png" /></figure><p>You should see the GCP Experiments listed here. Now we are all set to begin the steps of the experiment.</p><h3>STEP 2: Setting Up the Chaos Experiment</h3><p>We’d be using the <a href="https://docs.litmuschaos.io/docs/next/gcp-vm-instance-stop/">experiment docs</a> to help us with a few steps.</p><p>In this demo, we will inject chaos into two VM instances named test-instance and test-instance-1, belonging to the zones us-central1-a and us-central1-b respectively, belonging to the GCP project “Litmus GCP Instance Delete” with the ID of litmus-gcp-instance-delete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZW_7AW1lkJmL_SBUud6hWg.png" /></figure><p>Please notice that the instances are in a running state initially, before the injection of the chaos. Now that we have our instances ready, we can set up our experiment. Before scheduling the chaos experiment, we need to make the GCP Service Account credentials available to Litmus, so that the instances can be shut down and later started as part of the experiment. To do that, we’d make a Kubernetes secret named secret.yaml as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb4705fc5d57c09542465fbd222c94d3/href">https://medium.com/media/fb4705fc5d57c09542465fbd222c94d3/href</a></iframe><p>The format of this secret is also available in the experiment docs. Make sure the name of the secret is cloud-secret and replace the respective fields of the secret with your own service account credentials. Once done, apply the secret in the litmus namespace using the command:</p><pre>kubectl apply -f secret.yaml -n litmus</pre><p>Once the secret is applied, we’re all set to schedule our experiment from the Litmus portal. In Dashboard, click on the Schedule a Workflow button. In the workflow creation page, choose the self-agent and click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCVsGdY8JROEB-IYVYVSXw.png" /></figure><p>In the Choose a Workflow page, select “Create a new workflow using the experiments from MyHub” and select Chaos Hub in the dropdown. Then click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6nni9DM8-2pqz7inMG47g.png" /></figure><p>In the Workflow Settings page, fill in the workflow name and description of your choice. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-UQS5xX90d92gc_8bka7g.png" /></figure><p>In the Tune Workflow page, click on “Add a new experiment” and choose gcp/gcp-vm-instance-stop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5JCcExdUwirihZag-VLK_g.png" /></figure><p>Click Done. Notice that the experiment has been added to the experiment graph diagram. Now click on “Edit YAML”. Here we will edit the workflow manifest to specify the experiment resource details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_5K1OSQd8qaD4XIAdvL5Q.png" /></figure><p>Scroll down to the manifest of the ChaosExperiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zZUP9MHRG7rwpYlCpPFzWg.png" /></figure><p>Notice that the name of the secret that we had previously created is being passed to the ChaosExperiment to be mounted at the path /tmp/.</p><p>Scroll further down and similarly fill in the relevant experiment details in the manifest of the ChaosEngine as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1KKPZpITbIIVBsb1KNnAw.png" /></figure><p>Please take note that the zone for each target instance is to be mentioned in INSTANCE_ZONES in the same order of the VM_INSTANCE_NAMES. If you like, feel free to modify the other parameters of the experiment such as the RAMP_TIME, TOTAL_CHAOS_DURATION, etc. As you would have noticed, some of the experiment tunables are common for both the ChaosEngine and ChaosExperiment, and the values of ChaosExperiment get overridden by that of the values of the ChaosEngine if they differ in both the manifests. Once done, click Save Changes. We’ve now specified all the experiment details and are ready to go to the next step. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3eGS4zuAPeKoFpQ_8oRvGQ.png" /></figure><p>In the Reliability Score, we will use the default score of 10. Click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0bVb55ABK6yDB-Y6StswA.png" /></figure><p>In Schedule, click Schedule Now. Click Next. On the Verify and Commit page verify all the details and once satisfied click on Finish. We’ve successfully scheduled our chaos experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SPEcwOC1gxAgVwV0v_S_wQ.png" /></figure><h3>STEP 3: Observing the Chaos</h3><p>Click on Go to Workflow and choose the workflow that we just created. Here we can observe the different steps of the workflow execution including chaos experiment installation, chaos injection, and chaos revert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*taXCcBA8SEL_hkag6JmxSQ.png" /></figure><p>You can also determine if the chaos injection has taken place and as a result, the instances have shutdown or not from the GCP Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIvwN6J7tDpJn5RCuPGfNA.png" /></figure><p>We can also view the Table View for the experiment logs as the experiment proceeds through the various steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMeQ_JdSEEi3SRDlLrfooA.png" /></figure><p>Once completed, the workflow graph should have executed all the steps successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wFb7HH-ipZ3uVdRI76ebTA.png" /></figure><p>We can also check the ChaosResult verdict which should say the experiment has passed. The Probe Success Percentage should be 100% as all our instances restarted successfully post their shutdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O5t5gUjMyQEnXn4gU6hRAA.png" /></figure><p>Again you can check in the GCP console if the instances have restarted or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CPdg-cOiVjpNwW0JsZDxqA.png" /></figure><p>We can also perform post chaos analysis for the experiment results in the Analytics section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CV_uEUlWSr6-XTuBRn-4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*htQFa1oWJHtmPFHIU890Cg.png" /></figure><p>In conclusion of this blog, we saw how we can perform the GCP VM Instance Stop chaos experiment using Litmus Chaos 2.0. This experiment is only one of the many experiments for the Non-Kubernetes experiments in LitmusChaos, including experiments for AWS, Azure, VMWare, and many more, which are targeted towards making Litmus an absolute Chaos Engineering toolset for every enterprise regardless of the technology stack used by them.</p><p>Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. To join the Litmus community:<br><strong>Step 1:</strong> Join the Kubernetes slack using the following link: https://slack.k8s.io/<br><strong>Step 2:</strong> Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: <a href="https://slack.litmuschaos.io/">https://slack.litmuschaos.io/</a></p><p>Show your ❤️ with a ⭐ on our <a href="https://github.com/litmuschaos/litmus">Github</a>. To learn more about Litmus, check out the <a href="https://docs.litmuschaos.io/">Litmus documentation</a>. Thank you! 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ea5b58266b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/litmus-chaos/gcp-vm-instance-stop-experiment-for-litmuschaos-3ea5b58266b2">GCP VM Instance Stop Experiment for LitmusChaos</a> was originally published in <a href="https://medium.com/litmus-chaos">Litmus-Chaos</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Litmus 2.0 in Google Kubernetes Engine]]></title>
            <link>https://medium.com/litmus-chaos/getting-started-with-litmus-2-0-in-google-kubernetes-engine-42dddb264a12?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/42dddb264a12</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Fri, 02 Jul 2021 13:26:50 GMT</pubDate>
            <atom:updated>2021-08-13T11:03:49.351Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started with LitmusChaos 2.0 in Google Kubernetes Engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjV-3SHxwdzSamjcI4aEyg.png" /></figure><p>This is a quick tutorial on how to get started with Litmus 2.0 in Google Kubernetes Engine. We’ll first set up our basic GKE cluster throughout this blog, then install Litmus 2.0 in the cluster, and finally, execute a simple chaos workflow using Litmus.</p><p>But before we kick off the demonstration, let’s have a brief introduction to Litmus. <a href="https://litmuschaos.io/">Litmus</a> is a toolset to perform cloud-native Chaos Engineering. It provides tools to orchestrate chaos on Kubernetes to help developers and SREs find weaknesses in their application deployments. Litmus can be used to run chaos experiments initially in the staging environment and eventually in production to find bugs, vulnerabilities. Fixing the weaknesses leads to increased resilience of the system. Litmus adopts a “Kubernetes-native” approach to define chaos intent in a declarative manner via custom resources, although Litmus is not limited to only Kubernetes targets while injecting chaos and is able to target a plethora of non-Kubernetes targets as well, such as bare-metal infrastructure, public cloud infrastructure, hybrid cloud infrastructure, containerized services, etc. to fulfill the chaos engineering needs of an entire business, and not just a specific application microservice.</p><h3>Pre-Requisites to the Demo</h3><ol><li>A GCP project with sufficient permission for accessing GKE</li></ol><h3>Step-1: Setup the GKE Cluster</h3><p>To set up the GKE cluster, we first need to enable the Kubernetes Engine API in our GCP project. To do that, we can access the APIs &amp; Services Dashboard from the GCP console and search for the Kubernetes Engine API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2r4jdrIuLGsTW_xo1Ze5kg.png" /></figure><p>We can then enable the Kubernetes Engine API. Now we are all set to launch our GKE cluster. To do that, we will first go to the Kubernetes Engine dashboard in the GCP console and then choose to Create the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKxtj85kPd6Xf7ew2j3ZFw.png" /></figure><p>Go for a standard cluster and choose to Configure it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ivIqlysLZ3rP_2zy_reBtA.png" /></figure><p>Here you’d get all the options to configure your Kubernetes cluster. You can either choose to configure your own cluster as per your preference or you can simply go for My First Cluster option under the Cluster set-up Guide, which will set up a basic three-node cluster for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ER03wDAfQ9z4efTQ5s6g6g.png" /></figure><p>It will take a while for your cluster resources to set up and undergo a health check, after which it would finally be ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCiLOzED8t9nEA3Tu-8nXA.png" /></figure><p>We can now connect to our cluster and proceed to install litmus. To do that, we need to choose our cluster and then choose the Connect option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVFVSZVfTMWMy2gaOnfi5A.png" /></figure><p>Next, choose the Run In Cloud Shell option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jq0Is2xIvn2eooe8oOZqrQ.png" /></figure><p>This will open up a cloud shell terminal in your console. A cloud SDK command would be already there in your terminal which you need to execute. This command will configure the Kubectl to configure itself for the cluster we just created. If prompted, choose accept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-zbtP9wSV7VcyvtXhoajQ.png" /></figure><p>Now we are all set to install Litmus.</p><h3>STEP 2: Install Litmus</h3><p>We’d follow the instructions given in the <a href="https://litmusdocs-beta.netlify.app/docs/introduction">Litmus 2.0 documentation</a> to install Litmus. Here we will be installing Litmus in namespace mode using Helm. To do that, the very first step is to add the litmus helm repository using the following command:</p><pre>helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/</pre><p>This will add litmuschaos repository to the list of Helm chart repositories. We can verify the repository using the command:</p><pre>helm repo list</pre><p>You should see litmuschaos repository listed here. Next, we will create the litmus namespace as all the litmus infra components will be placed in this namespace. We will do this using the command:</p><pre>kubectl create ns litmus</pre><p>Once this is done, we can proceed to install the Litmus control plane using the following command:</p><pre>helm install chaos litmuschaos/litmus-2-0-0-beta --namespace=litmus --devel --set portalScope=namespace</pre><p>You should get a similar message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMnOfsKCUcdTFwXEWjx_6Q.png" /></figure><p>Next, we can proceed to install the Litmus CRDs. Cluster-Admin or an equivalent user with the right permissions are required to install the CRDs. We’d use the following command to apply the CRDs:</p><pre>kubectl apply -f https://raw.githubusercontent.com/litmuschaos/litmus/master/litmus-portal/litmus-portal-crds.yml</pre><p>And we’re done with installing Litmus. Just one final step before we can access the Litmus portal. If we use the command kubectl get svc -n litmus we’ll be able to list down all the services running in the litmus namespace. Here you should be able to see the litmusportal-frontend-service and the litmusportal-server-service, both of which should be NodePort services, and their corresponding TCP ports should be something like 9091:xxxxx/TCP where xxxxx is the node port of the respective service. To access the portal, we need to expose both the node ports of the frontend service and the server service by applying a firewall rule. For the litmusportal-server-service we can expose only the first NodePort as there are two of them. We can do that using the following two commands:</p><pre>gcloud compute firewall-rules create frontend-service-rule --allow tcp:&lt;NODE_PORT&gt;</pre><pre>gcloud compute firewall-rules create server-service-rule --allow tcp:&lt;NODE_PORT&gt;</pre><p>Replace the &lt;NODE_PORT&gt; in the first command with your frontend service’s node port and similarly replace the &lt;NODE_PORT&gt; in the first command with your server service’s node port. Once done, we’re all set to access the Litmus portal. To do that, use the command kubectl get nodes -o wide to list all the pods in your cluster. You should see something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DOT0E1kS-kJ_aWX8Aj5j-A.png" /></figure><p>For any of the nodes, copy its external IP and paste it into your browser URL section, followed by a :xxxxx where xxxxx corresponds to your node port. You should then be directed to the Litmus home screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DKLkV01jnnjVilby6mtewA.png" /></figure><p>Log in using the username admin and password litmus. Then you’d be asked to set up a password, which you’d use for any subsequent login. Once done, you’d be able to access the Litmus portal dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WCvu2RffIvxO9tWc6ukELQ.png" /></figure><h3>Step 3: Run a Chaos Workflow</h3><p>For this demo, we’d see how do chaos workflows execute in Litmus with the help of a predefined chaos workflow template called podtato-head. <a href="https://github.com/cncf/podtato-head">podtato-head</a> is a simple microservice-based application that can be deployed in the default namespace for the purpose of this demo, in which we will try to inject pod-delete chaos.</p><p>Once the podtato-head application is deployed in the default namespace, choose Schedule a Workflow. In the workflows dashboard, choose the self agent and then choose Next. Choose the Create a Workflow option from pre-defined workflow templates and choose podtato-head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBmN0Fsmw5FamvJX7V7aTQ.png" /></figure><p>Choose Next. Here we can define the experiment name, description, and namespace. Leave the default values and choose Next. Here we can tune the workflow by editing the experiment manifest and adding/removing or arranging the experiments in the workflow. The podtato-head template comes with its own defined workflow so simply choose Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PZ8AkosqNIHESenfR5QAgw.png" /></figure><p>Now we can set a reliability score per chaos experiment in our workflow. Since this workflow has only pod-delete chaos experiment, we can set a score for that in between 1–10. Then choose Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0NyrzqdjAbZsUuFPClNPw.png" /></figure><p>Here we can schedule our workflows to be executed in a determined frequency or we can simply schedule it to execute right away. Choose Schedule now. Choose Next. Here we can review the workflow details and make changes if required before finally executing the workflow. Choose Finish. You have successfully started the Chaos Workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eiFUcOY1hnW84w5M_nnvw.png" /></figure><p>Choose Go to Workflow and you’d see the running workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fSRwcYfywGwkiCu0Kxy8Q.png" /></figure><p>Select the workflow and you’d be able to see the workflow execute in its various stages in the graphical chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hdDu5Hxo4u7D7FgGDTBX4Q.png" /></figure><p>Wait for the workflow execution to complete, once completed the graph would appear as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Imn4mQYGRsHjHW__MLlLQ.png" /></figure><p>We can analyze the Table View tab for every step as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQeLj-efOQ2oKn0Dy3Qfmw.png" /></figure><p>Choose View Logs &amp; Results to access the chaos result of the pod-delete experiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3qvepJ0rmcSyHwjVLScFg.png" /></figure><p>You can also observe the analytics of the workflow from the workflow dashboard page which shows a graphical analysis of the workflow run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHKxARyWigg7vDnAefIH3Q.png" /></figure><p>In conclusion of this blog, we saw how to set up a Google Kubernetes Engine cluster, how to install Litmus 2.0, and finally how to execute a chaos workflow using the Litmus portal. Though this is only the tip of the iceberg, I really hope you learned something new today, which will be helpful for your further journey in chaos engineering.</p><p>Come join me at the Litmus community to contribute your bit in developing chaos engineering for everyone. To join the Litmus community: <br><strong>Step 1:</strong> Join the Kubernetes slack using the following link: <a href="https://slack.k8s.io/">https://slack.k8s.io/</a><br><strong>Step 2:</strong> Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: <a href="https://slack.litmuschaos.io/">https://slack.litmuschaos.io/</a></p><p>Show your ❤️ with a ⭐ on our <a href="https://github.com/litmuschaos/litmus">Github</a>. To learn more about Litmus, check out the <a href="https://docs.litmuschaos.io/">Litmus documentation</a>. Thank you! <a href="https://emojipedia.org/folded-hands/">🙏</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42dddb264a12" width="1" height="1" alt=""><hr><p><a href="https://medium.com/litmus-chaos/getting-started-with-litmus-2-0-in-google-kubernetes-engine-42dddb264a12">Getting Started with Litmus 2.0 in Google Kubernetes Engine</a> was originally published in <a href="https://medium.com/litmus-chaos">Litmus-Chaos</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Practical Guide to Containerisation and Chaos Engineering with LitmusChaos 2.0]]></title>
            <link>https://medium.com/litmus-chaos/a-beginners-practical-guide-to-containerisation-and-chaos-engineering-with-litmuschaos-2-0-5f4f3cf2a55d?source=rss-dcf285ed58a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f4f3cf2a55d</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Neelanjan Manna]]></dc:creator>
            <pubDate>Mon, 31 May 2021 20:00:31 GMT</pubDate>
            <atom:updated>2021-07-27T12:24:37.084Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part-2: A Beginner’s Practical Guide to Containerisation and Chaos Engineering with LitmusChaos 2.0</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMCqtt7bUfkhu5plPX4i2w.png" /></figure><h3>Part 2: Chaos Engineering 101, Using LitmusChaos 2.0 for a custom workflow chaos experiment</h3><p>This blog is part two of a two blog series that details how to get started with containerization using Docker and Kubernetes for deployment, and later how to perform chaos engineering using LitmusChaos 2.0. Find Part-1 of the blog <a href="https://neelanjanmanna.medium.com/a-beginners-practical-guide-to-containerisation-and-chaos-engineering-with-litmuschaos-2-0-eeb2ba859189">here</a>.</p><p>Say you have deployed your E-Commerce application using Kubernetes and you’re very satisfied with how flexible and stable your application deployment has come to be. During the testing of the application, it had checked all the boxes and you’re very confident that your application deployment is all set to face the peak hours of the sale next week, which will have customers all over the country trying to buy products using your application. But alas, right on the peak hours of the sale, your customers face a service outage. What went wrong? You don’t have any idea, since on a superficial level nothing seems to be out of order. Could this infelicitous situation have been avoided? Yes, using chaos engineering.</p><p>In this blog, we will fundamentally explore chaos engineering; starting from what is chaos engineering, why chaos engineering is a necessity, how is it different from testing, processes, and principles of chaos engineering, introduction to cloud-native chaos engineering, a bird’s eye view of LitmusChaos, and finally, we will perform a pod-delete chaos experiment with a custom chaos workflow on the application that we had deployed in the previous blog.</p><h3><strong>What is Chaos Engineering?</strong></h3><p>As Wikipedia defines it:</p><blockquote><strong>Chaos engineering</strong> is the discipline of experimenting on a software system in production in order to build confidence in the system’s capability to withstand turbulent and unexpected conditions.</blockquote><p>In our previous example of the E-Commerce application which faced a service downtime on account of the sharp rise in the number of users trying to simultaneously access the application, that situation could have been avoided by identifying the factors that were contributing to the service downtime.</p><p>Chaos Engineering emphasizes <em>experiments</em>, otherwise called <em>hypotheses</em>, and then compares the results to a defined steady-state. It is also regarded as the science of “breaking things on purpose” to identify the unforeseen weaknesses of the system before they wreak havoc in production. More than just fault injection, chaos engineering is an attempt at understanding the factors that contribute to the instability of a system, and gaining insights from the behavior of the system as a whole when it is subjugated to an adverse situation, with the ultimate goal of making systems more resilient in the production environment.</p><p>As an example, a distributed system could be checked for resiliency by randomly disabling the services responsible for the functioning of the system and analyzing its impact on the system as a whole.</p><h3>Why Chaos Engineering?</h3><p><strong>Recognize the dangers and consequences: </strong>By allowing you to create an experiment and quantify how it affects your business, chaos engineering allows you to understand the influence of turbulent conditions on important applications. Companies can make informed judgments and react proactively to avoid or prevent losses when they understand what’s at risk.</p><p><strong>Reaction to an incident: </strong>Because distributed systems are so complicated, there are several ways for things to go wrong. The notion of disaster recovery and business continuity is critical for firms in highly regulated contexts, such as the financial industry, because even a single instant of outage can be costly. These industries may rehearse, prepare, and put mechanisms in place for real-life situations by conducting chaotic experiments. When an incident occurs, chaos engineering allows teams to have the correct level of awareness, plans, and visibility.</p><p><strong>Application Security &amp; Observability: </strong>Chaos experiments help you figure out where your systems’ monitoring and observability capabilities are lacking, as well as your team’s ability to respond to crises. Chaos engineering will help you identify areas for improvement and motivate you to make your systems more visible, resulting in better telemetry data.</p><p><strong>System Reliability: </strong>Chaos engineering enables firms to create dependable and fault-tolerant software systems while also increasing your team’s trust in them. The more reliable your systems are, the more confident you can be in their ability to perform as expected.</p><h3>Is it the same as testing?</h3><p>In one word, no.</p><p>A failure test looks at a single situation and determines whether or not a property is true. A test like this breaks the system in a predetermined way. The outcomes are often binary and do not reveal any additional information about the program, which is essential to understand the root cause of the problem.</p><p>Chaos Engineering’s purpose is to generate fresh information about the system when it is subjugated to adversity. One can learn more about the system’s behaviors, attributes, and performance because the scope is broader and the outcomes are unpredictable. Therefore it allows us to better understand the limitations of our system and act on them.</p><h3>Process of conducting Chaos Engineering</h3><p>By and large, chaos engineering can be abstracted into the following set of processes:</p><ol><li><strong>Define the steady-state hypothesis:</strong> You should begin by imagining what could go wrong. Start with a failed injection and forecast what will happen when it’s live.</li><li><strong>Confirm the steady-state and perform several realistic simulations:</strong> Test your system using real-world scenarios to observe how it reacts to different stressors and events.</li><li><strong>Collect data and monitor dashboards:</strong> You must assess the system’s dependability and availability. It’s ideal to employ key performance indicators that are linked to consumer success or usage. We want to see how the failure compares to our hypothesis, therefore we’ll look at things like latency and requests per second.</li><li><strong>Changes and issues should be addressed:</strong> After conducting an experiment, you should have a good notion of what is working and what needs to be changed. We can now predict what will cause an outage and precisely what will cause the system to fail.</li></ol><h3>Principles of Chaos Engineering</h3><p>An elaborate description of the same is the <a href="https://principlesofchaos.org/">Principles of Chaos</a> manifesto, which describes the core concerns that chaos engineering should address:</p><p><strong>Understand your system’s normal state:</strong> Define your system’s steady state. Any chaotic experiment uses a system’s regular behavior as a reference point. You will have a better understanding of the effects of faults and failures if you understand the system when it is healthy.</p><p><strong>Use of realistic bugs and failures: </strong>All experiments should be based on plausible and realistic settings. When a real-life failure is injected, it becomes clear which processes and technologies need to be upgraded.</p><p><strong>Production-level testing:</strong> Only by running the test in a production setting can you see how disruptions influence the system. Allow your team to experiment in a development environment if they have little or no experience with chaotic testing. Once the production environment is ready, test it.</p><p><strong>Control the radius of the blast:</strong> A chaotic test’s blast radius should always be kept as small as possible. Because these tests are conducted in a live setting, there is a potential that they will have an impact on end-users.</p><p><strong>Automating chaos:</strong> Chaos experiments may be automated to the same degree as your CI/CD pipeline. Continuous chaos allows your team to continuously improve current and future systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHbTcsEAqA6RpxT_bhjv-w.png" /></figure><h3>Introduction to Cloud-Native Chaos Engineering</h3><p>Businesses love cloud: About a third of companies’ IT budget goes to cloud services and the global public cloud computing market is set to exceed $330 billion in 2021. The groundbreaking shift towards cloud-native software products needs to be supplemented with the right set of tools to ensure that they are resilient against all the possible adverse situations that may arise in production.</p><p>Enter cloud-native chaos engineering, the best of both worlds. In its core essence, it&#39;s all about performing chaos engineering in a cloud-native or Kubernetes-first way. There are four major principles that define how cloud-native a chaos engineering tool or framework is:</p><ol><li><strong>CRDs for Chaos Management: </strong>For coordinating chaos on Kubernetes, the framework should have explicitly defined CRDs. These CRDs provide standard APIs for provisioning and managing chaos in large-scale production systems. These are the elements that make up a chaotic workflow orchestration system.</li><li><strong>Open Source: </strong>To enable larger community engagement and examination, the framework must be totally open-source under the Apache License 2.0. The number of applications that are migrating to the Kubernetes platform is uncountable. Only the Open Chaos model will thrive and gain the requisite adoption at such a wide scale.</li><li><strong>Extensible and Pluggable: </strong>The framework should be integrable with the vast number of existing cloud-native applications, essentially built as a component that can be easily plugged in for chaos engineering within an application and can be easily plugged out as well.</li><li><strong>Broad Community adoption: </strong>The chaos will be carried out against well-known infrastructures such as Kubernetes, applications such as databases, and infrastructure components like as storage and networking. These chaos experiments can be utilized again, and a large community can help identify and contribute to more high-value scenarios. As a result, a Chaos Engineering system should have a central hub or forge where open-source chaos experiments can be shared and code-based collaboration is possible.</li></ol><p>LitmusChaos<strong> </strong>is a cloud-native Chaos Engineering framework for Kubernetes that fulfills all the four criteria listed above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rnHIj_oz14J_PwM9qlwJYA.jpeg" /><figcaption>Image Courtsey: <a href="https://medium.com/litmus-chaos/multiarch-support-in-litmuschaos-4ee37c45d0f5">Sarita Behera</a></figcaption></figure><h3>A Bird’s Eye View of LitmusChaos</h3><h4>What is Litmus</h4><p><a href="http://litmuschaos.io/">Litmus</a> is a toolset to do cloud-native Chaos Engineering. It helps both Developers and SREs automate the chaos experiments at different stages within the DevOps pipeline like development, during CI/CD, &amp; in production. Fixing the weaknesses leads to increased resilience of the system.</p><p>Litmus adopts a “Kubernetes-native” approach to define chaos intent in a declarative manner via custom resources. It broadly defines Kubernetes chaos experiments into two categories: application or pod-level chaos experiments and platform or infra-level chaos experiments. The former includes pod-delete, container-kill, pod-cpu-hog, pod-network-loss, etc., while the latter includes node-drain, disk-loss, node-cpu-hog, etc. It is largely developed under Apache License 2.0 license at the project level.</p><p>Before we understand the architecture of Litmus, let us understand a few terminologies:</p><ol><li><strong>Chaos Experiment:</strong> Chaos Experiments are the building blocks of the Litmus architecture. Users can develop the desired chaos workflow by choosing from freely available chaos experiments or by creating new ones.</li><li><strong>Chaos Workflow: </strong>A chaos workflow is a lot more than a chaos experiment. It helps the user define the intended result, observe the result, analyze the overall system behavior, and decide whether the system needs to be changed to improve resilience. For a normal development or operations team, LitmusChaos provides the infrastructure required to design, use, and manage chaotic workflows. Litmus’ teaming and GitOps features considerably aid in the collaborative control of chaotic processes within teams or software organizations.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwb1lUg99sVRANP3rd5dtQ.jpeg" /></figure><h4>Litmus Architecture</h4><p>Litmus components can be classified into two parts:</p><ol><li>Portal</li><li>Agents</li></ol><p><strong>Portal </strong>is a set of Litmus components that act as Cross-Cloud Chaos Control plane (WebUI) which is being used to orchestrate and observe the chaos workflows on Agents.</p><p><strong>Agent </strong>is the set of Litmus components that induces Chaos using the chaos workflows on the K8s cluster component.</p><h4>Portal Components</h4><ul><li><strong>Litmus WebUI:</strong> Litmus UI provides a web user interface, where users can construct and observe the chaos workflow at ease. Also this act as a cross-cloud chaos control plane that is</li><li><strong>Litmus Server:</strong> Litmus Server act as middleware which is used to handle API request from the user interface, store the config and results into the DB. This also acts as an interface to communicate between the requests and scheduling the workflow to Agent.</li><li><strong>Litmus DB:</strong> Litmus DB act as a config store for chaos workflows and their results.</li></ul><h4>Agent Components</h4><ul><li><strong>Chaos Operator:</strong> Chaos-Operator watches for the ChaosEngine CR and executes the Chaos-Experiments mentioned in the CR. Chaos-Operator is namespace scoped. By default, it runs in litmus namespace.</li><li><strong>CRDs:</strong> During installation, the following three CRDs are installed on the Kubernetes cluster: chaosexperiments.litmuschaos.io, chaosengines.litmuschaos.io, and chaosresults.litmuschaos.io.</li><li><strong>Chaos Experiment: </strong>Chaos Experiment is a CR and is available as YAML files on <a href="https://hub.litmuschaos.io/">Chaos Hub</a>.</li><li><strong>Chaos Engine: </strong>ChaosEngine CR connects experiments to applications. The user must construct ChaosEngine YAML by giving the app label and experiments, as well as the CR.</li><li><strong>Chaos Results:</strong> The results of a ChaosExperiment with a namespace scope are stored in the ChaosResult resource. The experiment itself creates or updates it in runtime. It contains critical information such as the ChaosEngine reference, Experiment State, Experiment Verdict (on completion), and key application/result properties. It can also be used to collect metrics.</li><li><strong>Chaos Probes: </strong>Litmus probes are pluggable tests that can be defined for any chaotic experiment within the ChaosEngine. These checks are carried out by the experiment pods based on the mode they are defined in, and their success is used to determine the experiment’s judgment (along with the standard “in-built” checks).</li><li><strong>Chaos Exporter: </strong>Metrics can be exported to a Prometheus database if desired. The Prometheus metrics endpoint is implemented by Chaos-Exporter.</li><li><strong>Subscriber: </strong>Subscriber is a component on the Agent side that communicates with the Litmus Server component to obtain Chaos process data and return the results.</li></ul><h3>Demo: Performing the pod-delete experiment with a custom chaos workflow on a Kubernetes deployment</h3><p>Let’s get a hang of Litmus with a very simple chaos experiment: the pod-delete experiment. Essentially, we’d like to see that whether our Kubernetes deployment from the last blog is resilient against the event of accidental pod-deletion.</p><p>Here’s what we’d do; firstly we will install Litmus, then we’d define our custom workflow, and finally we’d analyze the results of our experiment. Simple, isn’t it?</p><p>Our application deployment can be viewed using the kubectl get deployments command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpyGgHKGv3pCyA36CTTabQ.png" /></figure><p>and the associated pods can be viewed using kubectl get pods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3CbUjMfVW6hqzomgpl-aQ.png" /></figure><p>Before installing Litmus, let us check the pre-requisites:</p><ol><li>Kubernetes 1.15 or later</li><li>A persistent volume of 20GB (Recommended)</li><li>Helm3 or Kubectl</li></ol><p>Once we’re good with these, let’s follow the installation steps. Litmus can be installed either using Helm or using Kubectl. Let’s try to install it using Helm:</p><h4>STEP 1: Create a Litmus namespace in Kubernetes</h4><p>Litmus installs within the litmus namespace. So let us create the namespace with the command: kubectl create namespace litmus</p><h4>STEP 2: Add the Litmus Helm Chart</h4><p>Clone the litmus repo and move into the cloned directory using the command: git clone https://github.com/litmuschaos/litmus-helm &amp;&amp; cd litmus-helm</p><h4><strong>STEP 3: Install Litmus</strong></h4><p>The helm chart will install all the CRDs, required service account configuration, and chaos-operator required both for the core services as well as the portal to run. Use the command helm install litmuschaos — namespace litmus ./charts/litmus-2–0–0-beta/</p><p>And that’s it, you’re all set up to use Litmus! You can verify the installation by viewing all the resources installed under the litmus namespace using kubectl get all -n litmus command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ed2ylmNDs1MaRbPoao7_6A.png" /></figure><p>It might take a while for all the resources to get ready and eventually you’d see something as depicted above. Here we can observe that we have three <em>pods </em>running: litmus-frontend, litmus-backend, and mongo. These comprise the Litmus WebUI, Litmus Server, and Litmus DB respectively, as discussed earlier.</p><p>We also have three <em>services </em>for our application, namely litmusportal-frontend-service, litmusportal-backend-service, and mongo-service. These services maintain the endpoints for the <em>pods</em> which we saw earlier.</p><p>These <em>pods</em> are created using the two <em>deployments </em>namely litmusportal-frontend and litmusportal-backend, which are also responsible for specifying the <em>replicasets </em>for the same.</p><p>Lastly, the Mongo DB database being a stateful resource uses a <em>statefulset </em>named<em> </em>mongo<em> </em>to persist the data contained by the DB even if the mongo <em>pod </em>dies and restarts.</p><p>Once all these resources are ready, we can proceed to the Litmus portal. For this, we will try to access the <em>NodePort</em> service litmusportal-frontend-service. The <em>nodePort </em>assigned to litmusportal-frontend-service in my machine has a mapping of 9091:30628 where 9091 is the specified <em>targetPort</em> while 30628 is the assigned <em>nodePort</em>.</p><p>If you’re using Minikube, you can port-forward any unused port of your choice to the litmusportal-frontend-service in order to access it at that port. For example, if I wish to access the litmusportal-frontend-service at the 3000 port, I’d use the command kubectl port-forward svc/litmusportal-frontend-service 3000:9091 -n litmus. Once done, simply access the Litmus portal at http://127.0.0.1:3000.</p><p>If you’re using any other Kubernetes platform, you can directly access the litmus portal at the <em>nodePort</em>, given you have a firewall rule allowing ingress at that port. For example, I have a nodePort of 30628, hence I can directly access the Litmus portal at http://127.0.0.1:30628.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-EBZxJGoQ8YQFkEQ5X7Xw.png" /></figure><p>The default username is admin and the default password is litmus. Once you log in, you’d be prompted to enter a new password. Once that’s done, you’d find yourself in the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESqD9pbxN8w-2rfzVpKKKQ.png" /></figure><p>Let’s take a quick tour of the portal. The Dashboard is where you see the different workflows which were either currently executing or had been previously executing, the number of agents connected to the portal, and the number of projects and invitations for collaboration. Inside Workflows, you will be able to get an elaborate view of the running and past workflow, along with their respective analytics and logs. ChaosHubs is where you can access the chaos experiments, by default you can access all the experiments listed under the Litmus’ ChaosHub but you can also set it up with your own hub. Under Analytics you’d be able to get a vivid analysis of the different aspects of your chaos workflows, using the graphs. Finally, Settings allow you to modify your personal information, automate workflows using GitOps, etc.</p><p>Let’s proceed on with the workflow creation for our pod-delete experiment. In Dashboard, click on the Schedule a Workflow button and choose the self-agent, since the application we’re targetting is deployed within the same Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YC0HEGqAjgzt1b3_nYtAPQ.png" /></figure><p>Click Next. Select “Create a new workflow using the experiments from MyHub” and from the dropdown, choose “Chaos Hub”. This is because the experiment we’re trying to perform here i.e. the pod-delete experiment, is a part of the Chaos Hub and we can directly use it, without the need to define it ourselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiT3AhFrMSBINekc9BvrGQ.png" /></figure><p>Click Next. Under the Workflow Settings, rename your workflow to any name of your choice. Don’t alter the namespace since our Litmus installation is supposed to use the litmus namespace only. Add a description of your own choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vj4Wz7GnoE3l5l15Fgmv5A.png" /></figure><p>Click Next. Inside Tune Workflow, currently, you’d be able to view only one step listed in the flow chart and that is “install-chaos-experiments”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYEQ7efbd7qswwU_qTV4-Q.png" /></figure><p>Click on Add a new experiment and choose generic/pod-delete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EinYdPuGhg7eyNLpxv4kcg.png" /></figure><p>Click Done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXb30oN3jFSbO0XFnLYy0Q.png" /></figure><p>As you can observe, in the flow chart a second step “pod-delete” has been listed now. Although for the purpose of this demo we’d keep the workflow simple with only one experiment, one can design their entire workflow by adding more experiments as per the desired sequence. Its as simple as that!</p><p>Now, we’d specify the target deployment on which the chaos workflow will be executed. Click on Edit YAML and scroll down to line number 134 in the editor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oK4GE6K1PBBU4pyfTeEnKg.png" /></figure><p>Here, we need to specify the appLabel of our hello-world application by overriding the default app=nginx value. To check the label of our deployment, we can use the kubectl get deployments --show-labels command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GfHOgbIXkS2g7LWotcnkeQ.png" /></figure><p>The label of our deployment is app=hello-world so we’d simply replace app=nginx with app=hello-world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6FCkE7WouTfjC_quKZ88A.png" /></figure><p>Click Save Changes. Back in the portal, keep Revert Schedule as TRUE. This ensures that all the changes made to the system during the execution of the workflow will get reverted post-completion of the workflow. Click Next.</p><p>In Reliability Score, we can add a numeric score between 1 to 10 for each of our experiments. Litmus will simply use this weightage while calculating the resiliency score at the end of the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dskugwWlsj2B0dJUSkXzHw.png" /></figure><p>Click Next. Inside Schedule, we can schedule our workflows to be executed in a determined frequency or we can simply schedule it to execute right away. Choose “Schedule now”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bTKajnGdzAa4Yzwr-8_ovA.png" /></figure><p>Click Next. Inside Verify and Commit, you can review your workflow details and make changes if required before finally executing the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIHMkM-LULK9TZoIlSqC8w.png" /></figure><p>Once you’re satisfied with the configuration, click Finish. You have successfully started the Chaos Workflow now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-im6XW6sgaYnE4oyUMJPA.png" /></figure><p>Click Go to Worklow and choose the currently running workflow from the Workflows dashboard. You can view the graphical view of the workflow as it is actively performing the pod-delete experiment on our deployed application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSUxFmyaHyjCO2B7h_NurQ.png" /></figure><p>In effect, this experiment will first install the chaos experiment resources in the cluster, causing one of the three pods in our application deployment to be randomly chosen and deleted, and finally perform a cleanup of the chaos experiment resources.</p><p>The application deployment is therefore expected to spin up a new pod in replacement of the forcefully deleted pod for the experiment to be successful. Otherwise, if a new pod fails to spin up, then our system is definitely not resilient.</p><p>After a few minutes, you’d be able to see that the experiment has successfully completed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iX5eYeYwaKKhkZYUEc3ePw.png" /></figure><p>All the steps of the experiment were successful and our system was able to successfully cope up even in the scenario of forceful pod-delete. We can further analyze the details regarding the experiment inside the Table View:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyE7JX0iJv7fIhtD4ZCCtw.png" /></figure><p>As we can see, we have obtained a 100% resilience score, and the other pieces of information of our experiment are also visible here. Furthermore, we can analyze the workflow analytics to better understand the impact of the workflow upon our system. Click Back and in the Workflow dashboard, click on the options ellipsis icon:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1JXI5e0AIonB1YvpCr2DjA.png" /></figure><p>Choose Show the analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*zloC_QTsfUqwWbVHTB08kg.png" /></figure><p>The analytics graph shows the sequence of chaos experiments, the resiliency score, and the state of other parameters over the timeline of the execution of the workflow.</p><p>In conclusion of this blog series, let us appreciate that how we went all the way from containers to chaos engineering, starting from learning how to Dockerize a Node.js application, deploying our Docker container using Kubernetes, and finally using LitmusChaos to perform a pod-delete chaos experiment on our application.</p><p>Once again, welcome to the world of containers and chaos engineering. Come join me at the <a href="https://github.com/litmuschaos/litmus">Litmus community</a> to contribute your bit in developing chaos engineering for everyone. Stay updated on the latest Litmus trends through the Kubernetes <a href="https://kubernetes.slack.com/messages/CNXNB0ZTN">Slack</a> channel (Look for #litmus channel).</p><p>Don’t forget to share these resources with someone who you think might benefit from them. Thank you. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f4f3cf2a55d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/litmus-chaos/a-beginners-practical-guide-to-containerisation-and-chaos-engineering-with-litmuschaos-2-0-5f4f3cf2a55d">A Beginner’s Practical Guide to Containerisation and Chaos Engineering with LitmusChaos 2.0</a> was originally published in <a href="https://medium.com/litmus-chaos">Litmus-Chaos</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>