<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveen kumar on Medium]]></title>
        <description><![CDATA[Stories by Naveen kumar on Medium]]></description>
        <link>https://medium.com/@navin205?source=rss-f68b27754c86------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ITakD8elKjSycOVV.jpg</url>
            <title>Stories by Naveen kumar on Medium</title>
            <link>https://medium.com/@navin205?source=rss-f68b27754c86------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:09:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@navin205/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[You’re not ready for Kubernetes if you don’t understand Linux.]]></title>
            <link>https://medium.com/@navin205/youre-not-ready-for-kubernetes-if-you-don-t-understand-linux-1572aef4594b?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/1572aef4594b</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Tue, 13 Jan 2026 11:49:51 GMT</pubDate>
            <atom:updated>2026-01-13T11:49:51.678Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stop Over-Complicating Linux: This is How You Learn Linux for DevOps</h3><p><strong>You cannot succeed in DevOps without solid Linux fundamentals.</strong></p><p>I see too many aspiring DevOps engineers jumping straight into container orchestration without understanding the operating system that runs everything underneath.</p><p>Thanks for reading Devops with Naveen! Subscribe for free to receive new posts and support my work.</p><p>Subscribe</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8ySjkMd0kIEfcvVm.png" /></figure><h3>Why Linux Matters in DevOps</h3><p>Every production server you’ll work with runs Linux. Every container runs on Linux. Every cloud instance runs Linux. When things break at 3 AM (and they will), you need to diagnose issues using Linux commands, not GUI tools.</p><p>Here’s what happened to me early in my career: I was confident with Docker and Kubernetes basics, got my first DevOps job, and then faced a production incident. The application was down, customers were complaining, and I had to debug a memory leak on a Ubuntu server with nothing but a terminal. I spent 2 hours Googling basic commands while the system burned.</p><p>Don’t be that person.</p><h3>The Linux Skills That Actually Matter</h3><p>Forget about memorizing 500 commands. Focus on the concepts and tools that you’ll use daily in production environments. Here are the 10 essential areas every DevOps engineer must master:</p><h3>1. Process Management: Your Applications Under the Hood</h3><p>When your application crashes, consumes too much CPU, or becomes unresponsive, you need to understand processes. Think of processes as individual workers in a factory — each doing a specific job.</p><h3>The Commands You’ll Actually Use</h3><p><strong>See what’s running:</strong></p><pre># The holy trinity of process monitoring<br>ps aux                    # Snapshot of all processes<br>top                       # Real-time view (like Task Manager)<br>htop                      # Enhanced version (install it everywhere)</pre><p>When I run ps aux on a production server, I’m looking for:</p><ul><li>High CPU usage processes (%CPU column)</li><li>Memory hogs (%MEM column)</li><li>Processes that shouldn’t be running</li></ul><p><strong>Example scenario:</strong> Your web server is slow. Run top and you see a rogue process eating 90% CPU. Now you know exactly what to kill.</p><p><strong>Kill misbehaving processes:</strong></p><pre># Find the troublemaker first<br>ps aux | grep nginx</pre><pre># Kill it gently<br>kill 1234</pre><pre># Kill it with force (when gentle doesn’t work)<br>kill -9 1234</pre><pre># Nuclear option - kill all instances<br>killall nginx</pre><p><strong>Pro tip:</strong> Always try kill before kill -9. The gentle kill allows the process to clean up properly.</p><h3>Process Hierarchy Matters</h3><pre># See the family tree of processes<br>pstree                    # Visual process tree<br>pstree -p                 # Include process IDs</pre><p>This shows you parent-child relationships. Kill a parent process, and its children die too. Essential for understanding how applications spawn sub-processes.</p><h3>2. Networking: How Your Services Talk</h3><p>In a microservices world, everything is connected. Applications talk to databases, APIs call other APIs, load balancers distribute traffic. When networking breaks, everything breaks.</p><h3>Network Interface Management</h3><pre># Modern way to check network setup<br>ip addr show              # Show all network interfaces<br>ip link show              # Show network devices</pre><pre># The old way (still works)<br>ifconfig                  # Shows interface configuration</pre><p><strong>Real scenario:</strong> Your application can’t reach the database. First thing I check: ip addr show to ensure the network interface is up and has the right IP address.</p><h3>Port Monitoring — The Detective Work</h3><pre># Who’s using what port?<br>netstat -tulpn            # Show all listening ports with processes<br>ss -tulpn                 # Modern replacement (faster)</pre><pre># Specific port investigation<br>lsof -i :80               # What’s using port 80?<br>lsof -i :3306             # Check if MySQL is running</pre><p><strong>Example workflow:</strong> Application won’t start, claiming “port already in use.” Run lsof -i :8080 and discover another process is squatting on your port. Kill it or change your app’s port.</p><h3>Network Troubleshooting Arsenal</h3><pre># Is the server reachable?<br>ping google.com           # Basic connectivity test<br>ping -c 4 server.com      # Send only 4 packets</pre><pre># Trace the network path<br>traceroute google.com     # Show every router hop</pre><pre># DNS investigation<br>dig google.com            # Detailed DNS lookup<br>nslookup google.com       # Simple DNS check</pre><p><strong>Production debugging story:</strong> API calls were timing out randomly. traceroute revealed packet loss at a specific router. Network team fixed the routing issue.</p><h3>Firewall Management</h3><pre># Ubuntu/Debian firewall (UFW - User Friendly Firewall)<br>sudo ufw status           # Check firewall status<br>sudo ufw enable           # Turn on firewall<br>sudo ufw allow 22         # Allow SSH<br>sudo ufw allow 80/tcp     # Allow HTTP traffic<br>sudo ufw deny 8080        # Block specific port</pre><pre># Traditional iptables (more complex but powerful)<br>sudo iptables -L          # List all rules<br>sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT  # Allow HTTPS</pre><h3>3. File System Navigation</h3><p>You’ll spend countless hours navigating server file systems. Efficiency here saves hours daily.</p><h3>Navigation Mastery</h3><pre># Basic movement<br>pwd                       # Where am I?<br>cd /var/log              # Go to logs directory<br>cd ..                    # Go up one level<br>cd ~                     # Go home<br>cd -                     # Go to previous directory</pre><p><strong>Pro workflow:</strong> I always start with pwd to know my location, then use ls -la to see what’s available.</p><h3>File and Directory Operations</h3><pre># Creating things<br>mkdir -p /path/to/deep/directory    # Create nested directories<br>touch config.yaml                   # Create empty file</pre><pre># Copying and moving<br>cp app.py app.py.backup            # Backup before changes<br>cp -r /source/dir /destination/     # Copy entire directories<br>mv oldname.txt newname.txt          # Rename files</pre><pre># Removing (be careful!)<br>rm filename                         # Delete file<br>rm -rf directory                    # Delete directory and contents (DANGEROUS!)</pre><p><strong>Safety tip:</strong> Always use ls to verify what you’re about to delete. I’ve seen entire projects wiped because someone was in the wrong directory.</p><h3>File Viewing Techniques</h3><pre># Quick file inspection<br>cat config.yaml           # Show entire file<br>head -20 app.log          # First 20 lines<br>tail -50 error.log        # Last 50 lines<br>tail -f application.log   # Follow log in real-time (lifesaver for debugging)</pre><pre># Paginated viewing<br>less large-file.txt       # Navigate large files</pre><p><strong>Real scenario:</strong> Application is throwing errors. I run tail -f /var/log/app.log in one terminal while reproducing the issue in another. I can see errors appear in real-time.</p><h3>4. File Permissions: The Security Foundation</h3><p>Nothing is more frustrating than “Permission denied” errors. Understanding permissions prevents security issues and deployment failures.</p><h3>Understanding the Permission Matrix</h3><pre>rwx rwx rwx<br>│   │   └── Others (everyone else)<br>│   └────── Group (file’s group)  <br>└────────── Owner (file creator)</pre><pre>r = read (4)    - Can view file content<br>w = write (2)   - Can modify file<br>x = execute (1) - Can run file as program</pre><h3>Permission Management in Practice</h3><pre># Symbolic notation (readable)<br>chmod u+x script.sh       # Give owner execute permission<br>chmod g-w file.txt        # Remove group write permission<br>chmod o+r document.pdf    # Give others read permission<br>chmod a+x binary          # Give everyone execute permission</pre><pre># Numeric notation (faster once you learn it)<br>chmod 755 script.sh       # rwxr-xr-x (common for scripts)<br>chmod 644 config.txt      # rw-r--r-- (common for config files)<br>chmod 600 private.key     # rw------- (private keys)<br>chmod 777 file.txt        # rwxrwxrwx (AVOID THIS - security risk!)</pre><p><strong>Production example:</strong> Deploy a script to production, but it won’t run. Check with ls -la script.sh and see it lacks execute permission. Fix with chmod +x script.sh.</p><h3>Ownership Management</h3><pre># Change file ownership<br>sudo chown user:group file.txt      # Change both user and group<br>sudo chown jenkins config.yaml      # Change only owner<br>sudo chown :docker script.sh        # Change only group</pre><pre># Recursive ownership changes<br>sudo chown -R nginx:nginx /var/www/html   # Change entire directory tree</pre><h3>5. File System Usage: Avoiding the Disk Space Disaster</h3><p>Running out of disk space crashes applications. Monitoring disk usage prevents 90% of storage-related incidents.</p><h3>Disk Space Monitoring</h3><pre># Check available space<br>df -h                     # Human-readable disk usage<br>df -i                     # Check inode usage (can run out even with space)</pre><pre># Find what’s eating your disk<br>du -h /var/log            # Directory size<br>du -sh *                  # Size of each item in current directory<br>du -h --max-depth=1 /     # Top-level directory sizes</pre><p><strong>Emergency scenario:</strong> Server alerts show 95% disk usage. Run du -sh /* to find the culprit directory, then dive deeper with du -h --max-depth=1 /var/log to find massive log files.</p><h3>Finding Space Hogs</h3><pre># Find large files<br>find / -size +100M -type f 2&gt;/dev/null    # Files larger than 100MB<br>find /var/log -name “*.log” -size +50M    # Large log files</pre><pre># Find old files (potential cleanup candidates)<br>find /tmp -type f -mtime +30               # Files older than 30 days</pre><h3>6. Search and Filter: Finding Needles in Haystacks</h3><p>Production environments generate massive amounts of data. Efficient searching separates good DevOps engineers from frustrated ones.</p><h3>File Finding Mastery</h3><pre># Find files by name<br>find /var/log -name “*.log”           # All log files<br>find /etc -name “*nginx*”             # Anything with nginx in name<br>find / -type f -name “config.yaml”    # Specific filename anywhere</pre><pre># Find by size and date<br>find /var/log -size +100M             # Large log files<br>find /tmp -mtime +7                   # Files older than 7 days<br>find /etc -type f -perm 777           # World-writable files (security risk)</pre><h3>Text Processing Powerhouse</h3><pre># grep - your text searching best friend<br>grep “ERROR” /var/log/app.log         # Find errors in logs<br>grep -r “database” /etc/              # Recursively search for “database”<br>grep -i “warning” *.log               # Case-insensitive search<br>grep -n “failed” app.log              # Show line numbers<br>grep -v “DEBUG” app.log               # Exclude debug messages</pre><pre># Real-world log analysis<br>grep “500” /var/log/nginx/access.log | wc -l     # Count 500 errors<br>grep “$(date ‘+%Y-%m-%d’)” /var/log/app.log      # Today’s logs only</pre><p><strong>Debugging workflow:</strong> Application is slow. I run grep -i “slow\|timeout\|error” /var/log/app.log | tail -20 to see recent issues.</p><h3>Advanced Text Processing</h3><pre># awk - column extraction magic<br>awk ‘{print $1}’ /var/log/nginx/access.log        # Extract IP addresses<br>awk -F: ‘{print $1}’ /etc/passwd                  # Extract usernames<br>awk ‘$9 == 404 {print $1}’ /var/log/nginx/access.log  # IPs with 404 errors</pre><pre># sed - stream editing<br>sed ‘s/old/new/g’ config.txt          # Replace all occurrences<br>sed -n ‘10,20p’ large-file.txt        # Print lines 10-20</pre><pre># cut - simple column extraction<br>cut -d: -f1 /etc/passwd               # First field using : delimiter<br>cut -c1-10 filename                   # Characters 1-10 of each line</pre><pre># sort and unique analysis<br>sort /var/log/ips.txt | uniq -c | sort -nr    # Count and sort unique IPs</pre><h3>7. Package Management: Installing and Managing Software</h3><p>Different Linux distributions use different package managers. Knowing them prevents “software not found” headaches.</p><h3>APT (Ubuntu/Debian) — The Most Common</h3><pre># Update package database first (always!)<br>sudo apt update                       # Refresh package lists<br>sudo apt upgrade                      # Upgrade installed packages</pre><pre># Install software<br>sudo apt install nginx                # Install web server<br>sudo apt install htop git curl       # Multiple packages at once</pre><pre># Remove software<br>sudo apt remove nginx                 # Remove package<br>sudo apt purge nginx                  # Remove package and configs<br>sudo apt autoremove                   # Clean up orphaned dependencies</pre><pre># Search for packages<br>apt search docker                     # Find docker-related packages<br>apt list --installed | grep python   # List installed Python packages</pre><h3>YUM/DNF (RedHat/CentOS/Fedora)</h3><pre># YUM (older RHEL/CentOS systems)<br>sudo yum update                       # Update all packages<br>sudo yum install docker               # Install Docker<br>sudo yum remove docker                # Remove Docker</pre><pre># DNF (newer Fedora/RHEL systems)<br>sudo dnf update                       # Update packages<br>sudo dnf install podman               # Install container runtime<br>sudo dnf search kubernetes            # Search for packages</pre><p><strong>Production tip:</strong> Always run apt update before apt install to ensure you get the latest package versions.</p><h3>8. System Configuration: Making Your Environment Work</h3><p>Understanding system configuration prevents mysterious failures and enables automation.</p><h3>System Information Commands</h3><pre># Know your system<br>uname -a                              # Complete system info<br>lscpu                                 # CPU information<br>free -h                               # Memory usage<br>lsblk                                 # Block devices (disks)<br>df -h                                 # Disk usage</pre><pre># OS and version info<br>cat /etc/os-release                   # OS version details<br>hostnamectl                           # Hostname and system info</pre><h3>Shell Configuration Mastery</h3><pre># Profile files (loaded in order)<br>/etc/profile                          # System-wide login profile<br>~/.bash_profile                       # User login profile<br>~/.bashrc                             # User interactive shell config</pre><pre># Create useful aliases<br>alias ll=’ls -la’                     # Long listing<br>alias la=’ls -la’                     # All files with details<br>alias grep=’grep --color=auto’        # Colored grep output<br>alias k=’kubectl’                     # Kubernetes shortcut</pre><pre># Make aliases permanent<br>echo “alias ll=’ls -la’” &gt;&gt; ~/.bashrc<br>source ~/.bashrc                      # Reload configuration</pre><p><strong>Pro productivity tip:</strong> I create aliases for commonly used commands. My .bashrc has aliases for kubectl, docker, and complex find commands I use regularly.</p><h3>Environment Variables</h3><pre># View environment<br>env                                   # All environment variables<br>echo $PATH                            # Show PATH variable<br>echo $HOME                            # Home directory</pre><pre># Set temporary variables<br>export API_KEY=”your-key-here”        # For current session</pre><pre># Set permanent variables<br>echo ‘export API_KEY=”your-key-here”’ &gt;&gt; ~/.bashrc<br>source ~/.bashrc                      # Reload to activate</pre><h3>9. System Monitoring: Keeping Your Finger on the Pulse</h3><p>Monitoring prevents problems. When you can see resource usage patterns, you can predict and prevent failures.</p><h3>Load and Performance Monitoring</h3><pre># System overview<br>uptime                                # Load average and uptime<br>w                                     # Who’s logged in and doing what<br>top                                   # Real-time process monitoring<br>htop                                  # Enhanced process viewer</pre><pre># Memory analysis<br>free -h                               # Memory usage summary<br>cat /proc/meminfo                     # Detailed memory info<br>vmstat 1 5                            # Virtual memory stats (1 sec intervals, 5 times)</pre><pre># I/O performance<br>iostat 1 5                            # Disk I/O statistics<br>sar 1 5                               # System activity report</pre><p><strong>Monitoring workflow:</strong> When a server feels slow, I run top to see CPU usage, free -h for memory, and iostat to check if disk I/O is the bottleneck.</p><h3>Swap Usage Analysis</h3><pre># Check swap status<br>swapon --show                         # Active swap partitions<br>cat /proc/swaps                       # Swap usage details<br>free -h                               # Memory and swap summary</pre><p><strong>Performance tip:</strong> If swap usage is high, your system needs more RAM or has a memory leak.</p><h3>10. Service Management: Controlling Your Applications</h3><p>Modern Linux uses systemd for service management. Master this, and you can control any application in production.</p><h3>Systemd Service Control</h3><pre># Basic service operations<br>sudo systemctl start nginx            # Start web server<br>sudo systemctl stop nginx             # Stop web server  <br>sudo systemctl restart nginx          # Restart (stop then start)<br>sudo systemctl reload nginx           # Reload config without restart</pre><pre># Service status and health<br>systemctl status nginx                # Detailed service status<br>systemctl is-active nginx             # Quick active check<br>systemctl is-enabled nginx            # Check if starts at boot</pre><pre># Enable/disable services<br>sudo systemctl enable nginx           # Start automatically at boot<br>sudo systemctl disable nginx          # Don’t start at boot</pre><p><strong>Real scenario:</strong> Deploy a new application version. Instead of killing processes manually, I run sudo systemctl restart myapp for a clean restart that respects the service configuration.</p><h3>Service Discovery and Troubleshooting</h3><pre># List services<br>systemctl list-units --type=service   # All services<br>systemctl list-units --failed         # Only failed services<br>systemctl list-unit-files --type=service | grep enabled  # Boot-enabled services</pre><pre># Log investigation<br>journalctl -u nginx                    # Service-specific logs<br>journalctl -u nginx --since “1 hour ago”  # Recent logs<br>journalctl -f -u nginx                 # Follow logs in real-time</pre><h3>Legacy Service Management</h3><pre># Traditional service command (still works)<br>sudo service nginx start              # Start service<br>sudo service nginx status             # Check status<br>sudo /etc/init.d/nginx restart        # Direct init script</pre><h3>The Commands You’ll Use Every Day</h3><p>Here are the commands I use most frequently in production:</p><h3>Daily Operations</h3><pre># Process monitoring<br>ps aux | grep python                  # Find Python processes<br>top -p $(pgrep nginx)                 # Monitor specific processes<br>htop                                  # Interactive process manager</pre><pre># Network debugging<br>ss -tulpn | grep :80                  # Check web server port<br>ping -c 3 database-server             # Test connectivity<br>curl -I <a href="https://api.example.com/">https://api.example.com</a>       # Check API health</pre><pre># File operations<br>tail -f /var/log/app.log              # Follow application logs<br>find /var/log -name “*.log” -mtime -1 # Today’s log files<br>grep -r “ERROR” /var/log/ | tail -10  # Recent errors</pre><pre># System health<br>df -h                                 # Disk space check<br>free -h                               # Memory usage<br>systemctl status docker               # Service status</pre><h3>Emergency Debugging</h3><pre># High CPU investigation<br>top -n 1 -b | grep -A 5 “^top”       # Quick CPU snapshot<br>ps aux --sort=-%cpu | head -10        # Top CPU users</pre><pre># Memory issues<br>ps aux --sort=-%mem | head -10        # Top memory users<br>cat /proc/meminfo | grep Available    # Available memory</pre><pre># Disk space emergency<br>du -sh /* | sort -hr | head -10       # Largest directories<br>find /var/log -name “*.log” -size +100M  # Large log files</pre><h3>Best Practices That Will Save Your Career</h3><ol><li><strong>Always backup before making changes</strong> cp config.yaml config.yaml.backup</li><li><strong>Test commands safely first</strong> # Test what files will be deleted find /tmp -name “*.tmp” -type f # Then actually delete them find /tmp -name “*.tmp” -type f -delete</li><li><strong>Use version control for configuration files</strong> cd /etc/nginx sudo git init sudo git add . sudo git commit -m “Initial nginx config”</li><li><strong>Monitor logs in real-time during deployments</strong> # One terminal for deployment sudo systemctl restart myapp # Another terminal for monitoring tail -f /var/log/myapp.log</li><li><strong>Learn keyboard shortcuts</strong></li></ol><ul><li>Ctrl+C: Cancel current command</li><li>Ctrl+Z: Suspend current command</li><li>Ctrl+R: Search command history</li><li>Tab: Auto-complete commands and paths</li></ul><h3>What’s Next?</h3><p>Master these Linux fundamentals, and you’ll have the foundation for everything else in DevOps. But here’s what most people get wrong: they learn these commands in isolation.</p><p>The real power comes from combining them:</p><pre># Complex real-world command combining multiple concepts<br>ps aux | grep nginx | grep -v grep | awk ‘{print $2}’ | xargs sudo kill -9</pre><p>This command:</p><ol><li>Lists all processes (ps aux)</li><li>Finds nginx processes (grep nginx)</li><li>Excludes the grep command itself (grep -v grep)</li><li>Extracts process IDs (awk ‘{print $2}’)</li><li>Kills all those processes (xargs sudo kill -9)</li></ol><p>Once you understand each piece, you can create powerful one-liners that solve complex problems.</p><h3>Coming Up Next</h3><p>Now that you have solid Linux fundamentals, you’re ready for:</p><ul><li><strong>Docker and Containerization</strong> — Package your applications</li><li><strong>Kubernetes Orchestration</strong> — Manage containers at scale</li><li><strong>Infrastructure as Code</strong> — Automate everything</li><li><strong>CI/CD Pipelines</strong> — Automate deployments</li></ul><p>But remember: everything builds on these Linux fundamentals. When your Kubernetes pod won’t start, you’ll use kubectl exec to get a shell and debug using these same Linux commands.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1572aef4594b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practice to Prevent OOMKilled Status]]></title>
            <link>https://medium.com/@navin205/best-practice-to-prevent-oomkilled-status-6481b63a6137?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/6481b63a6137</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 18:35:07 GMT</pubDate>
            <atom:updated>2024-12-27T18:36:10.840Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Best Practice to Prevent OOMKilled Status in Kubernetes</strong></h3><p>👉 Properly Setting Memory Requests and Limits <br>You have to set requests and limits for your pods:</p><p>1 — No Requests No Limits: If there are no requests or limits, one pod can consume all the resources on the node and prevent the other pods from requiring resources. This is not an ideal case.</p><p>2 — No Requests but Have Limits: In this case, k8s automatically sets the requests the same as limits, and each pod has guaranteed resources as limits.</p><p>3 — Having Requests and Limits: If both requests and limits are set, each pod gets a guaranteed number of resources and can go up to the defined limit.</p><p>4 — Having Requests but No Limits: Each pod can get guaranteed resources as requests are set and can go up to many resources as no limits are set.</p><p>You can choose any case of setting requests and limits for your pod according to your requirements, but it’s always preferred to set requests and limits on your pods so that they can’t use more or fewer resources than needed.</p><p>If you don’t mention limits, it can consume unlimited resources, leading to the OOMKilled event, and if you mention the fewer limits that are required by the pod to work, that also will not work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6481b63a6137" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshoot and Fix Kubernetes -Resource Exhausted error #1]]></title>
            <link>https://medium.com/@navin205/troubleshoot-and-fix-kubernetes-resource-exhausted-error-1-98b1cefbe255?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/98b1cefbe255</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 18:12:53 GMT</pubDate>
            <atom:updated>2024-12-27T18:12:53.491Z</atom:updated>
            <content:encoded><![CDATA[<h3>Regular monitoring and proactive resource management are essential to maintaining a stable Kubernetes environment.</h3><p><strong>Why am getting “Resource Exhausted” error in Kubernetes?</strong></p><h3>Possible Causes</h3><ol><li><strong>Insufficient Node Resources:</strong> The cluster nodes do not have enough CPU or memory to schedule new pods.</li><li><strong>Improper Resource Requests and Limits:</strong> Pods are requesting more resources than available or no resource limits are defined, leading to over-utilization.</li><li><strong>High Cluster Workload:</strong> The cluster is running too many pods, causing resource contention.</li><li><strong>Evictions:</strong> Nodes evict pods to reclaim resources due to resource pressure.</li></ol><h3>Step-by-Step Resolution</h3><h4>☞ Identify the Cause</h4><ul><li><strong>Check Pod Events:</strong></li><li><em>kubectl describe pod &lt;pod-name&gt;</em></li></ul><p>Look for messages like Insufficient CPU or Insufficient memory.</p><ul><li><strong>Check Node Status:</strong></li><li><em>kubectl get nodes</em></li><li><em>kubectl describe node &lt;node-name&gt;</em></li><li>Review node resource capacity and utilization.</li><li><strong>Monitor Resource Usage:</strong> Use metrics-server or tools like Prometheus and Grafana to monitor CPU and memory usage across the cluster.</li></ul><h4>☞ Verify Resource Requests and Limits</h4><ul><li>Ensure that each pod has proper resource requests and limits defined.</li><li>resources:</li><li>requests:</li><li>memory: “128Mi”</li><li>cpu: “500m”</li><li>limits:</li><li>memory: “256Mi”</li><li>cpu: “1000m”</li><li>Check existing pod configurations:</li><li>kubectl get pods -o yaml</li></ul><h4>☞ Adjust Resource Requests and Limits</h4><ul><li><strong>Modify Resource Configuration:</strong> Update resource requests and limits in the pod specification to match actual needs.</li></ul><p><em>kubectl edit deployment &lt;deployment-name&gt;</em></p><h4>☞ Scale the Application</h4><ul><li>If the workload is high, scale up the application using replicas:</li><li><em>kubectl scale deployment &lt;deployment-name&gt; — replicas=&lt;number&gt;</em></li></ul><h4>☞ Add More Nodes to the Cluster</h4><ul><li><strong>For On-Prem Clusters:</strong> Add more nodes manually.</li></ul><p><strong>For Cloud Providers:</strong> Enable cluster auto-scaling. Example (GKE):</p><p><em>gcloud container clusters update &lt;cluster-name&gt; — enable-autoscaling — min-nodes=&lt;min&gt; — max-nodes=&lt;max&gt;</em></p><h4>☞ Implement Resource Quotas</h4><ul><li>Limit resource usage per namespace to prevent overuse:</li></ul><p>apiVersion: v1 kind: ResourceQuota metadata: name: resource-quota namespace: &lt;namespace&gt; spec: hard: requests.cpu: “4” requests.memory: “8Gi” limits.cpu: “8” limits.memory: “16Gi”</p><h4>☞ Enable Cluster Autoscaler</h4><ul><li>Use Kubernetes Cluster Autoscaler to dynamically add nodes when resources are exhausted.</li></ul><h4>☞ Optimize Existing Resources</h4><ul><li>Identify and terminate unused pods.</li><li>Reduce resource limits for less critical applications.</li></ul><h4>☞ Monitor and Prevent Future Issues</h4><ul><li>Set up alerting for high resource utilization.</li><li>Use Vertical Pod Autoscaler (VPA) or Horizontal Pod Autoscaler (HPA) for dynamic scaling.</li></ul><p>By identifying the root cause, ensuring proper resource configuration, and scaling your cluster appropriately, you can resolve and prevent “Resource Exhausted” errors. Regular monitoring and proactive resource management are essential to maintaining a stable Kubernetes environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98b1cefbe255" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshoot and Fix Kubernetes CrashLoopBackoff error]]></title>
            <link>https://medium.com/@navin205/troubleshoot-and-fix-kubernetes-crashloopbackoff-error-07dcfe677221?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/07dcfe677221</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 18:06:49 GMT</pubDate>
            <atom:updated>2024-12-27T18:06:49.372Z</atom:updated>
            <content:encoded><![CDATA[<p>When a Kubernetes container repeatedly fails to start, it enters a ‘CrashLoopBackOff’ state, indicating a restart loop within a pod. This error often occurs due to various issues preventing the container from launching properly.</p><p>To confirm this error, execute</p><p><strong><em>kubectl get pods</em></strong> and</p><p>verify if the pod status is ‘<strong>CrashLoopBackOff</strong>’.</p><h3>How Does CrashLoopBackOff Work?</h3><p>By default, a pod’s restart policy is <strong>Always</strong>, meaning it should always restart on failure (Options: Never/OnFailure/Always). Depending on the restart policy defined in the pod template, Kubernetes might try to restart the pod multiple times.</p><p>When a Pod state is displaying CrashLoopBackOff, it means that it’s currently waiting the indicated time before restarting the pod again.</p><p>Every time the pod is restarted, Kubernetes waits for a longer and longer time, known as a <strong>backoff delay</strong>. The delay between restarts is exponential (10s, 20s, 40s, …) and is capped at five minutes. During this process, Kubernetes displays the CrashLoopBackOff error.</p><h3>Causes of CrashLoopBackOff</h3><p><strong>CrashLoopBackOff </strong>can occur when a pod fails to start for some reason, because a container fails to start up properly and repeatedly crashes. Let’s review the common causes of this issue.</p><h3>Resource Overload or Insufficient Memory</h3><p>One of the common causes of the CrashLoopBackOff error is resource overload or insufficient memory. Kubernetes allows setting memory and CPU usage limits for each pod, which means your application might be crashing due to insufficient resources. This might happen due to:</p><ol><li>memory leaks in your program,</li><li>misconfigured resource requests and limits, or</li><li>your application requires more resources than are available on the node.</li></ol><p><em>When the node that the pod is running on doesn’t have enough resources, the pod can be evicted and moved to a different node. If none of the nodes have sufficient resources, the pod can go into a </em><strong><em>CrashLoopBackOff </em></strong><em>state.</em></p><p>To resolve this issue, you need to understand the resource usage of your application and set the appropriate resource requests and limits. You can use the kubectl describe pod [pod_name] command to check if the pod was evicted due to insufficient memory.</p><p>You can also monitor the memory and CPU usage of your pods using Kubernetes metrics server or other monitoring tools like Prometheus. If your application is consistently using more resources than allocated, you might need to optimize your application, allocate more resources, or change resources:limits in the Container’s resource manifest.</p><h3>Errors When Deploying Kubernetes</h3><p>A common reason pods in your Kubernetes cluster display a CrashLoopBackOff message is that Kubernetes springs deprecated versions of Docker. You can reveal the Docker version using -v checks against the containerization tool.</p><p>A best practice for fixing this error is ensuring you have the latest Docker version and the most stable versions of other plugins. Thus, you can prevent deprecated commands and inconsistencies that trip your containers into start-fail loops.</p><p>When migrating a project into a Kubernetes cluster, you might need to roll back several Docker versions to meet the incoming project’s version.</p><h3>Issue with Third-Party Services (DNS Error)</h3><p>Sometimes, the CrashLoopBackOff error is caused by an issue with one of the third-party services. If this is the case, upon starting the pod you’ll see the message:</p><p><em>send request failure caused by: Post</em></p><p>Check the syslog and other container logs to see if this was caused by any of the issues we mentioned as causes of CrashLoopBackoff (e.g., locked or missing files). If not, then the problem could be with one of the third-party services.</p><p>To verify this, you’ll need to use a debugging container. A debug container works as a shell that can be used to login into the failing container. This works because both containers share a similar environment, so their behaviors are the same. Here is a link to one such shell you can use: ubuntu-network-troubleshooting.</p><p>Using the shell, log into your failing container and begin debugging as you normally would. Start with checking kube-dns configurations, since a lot of third-party issues start with incorrect DNS settings.</p><h3>Missing Dependencies</h3><p>The CrashLoopBackOff status can activate when Kubernetes cannot locate runtime dependencies (i.e., the var, run, secrets, kubernetes.io, or service account files are missing). This might occur when some containers inside the pod attempt to interact with an API without the default access token.</p><p>This scenario is possible if you manually create the pods using a unique API token to access cluster services. The missing service account file is the declaration of tokens needed to pass authentication.</p><p>You can fix this error by allowing all new –mount creations to adhere to the default access level throughout the pod space. Ensure that new pods using custom tokens comply with this access level to prevent continuous startup failures.</p><h3>Changes Caused by Recent Updates</h3><p>If you constantly update your clusters with new variables that spark resource requirements, they will likely encounter CrashLoopBackOff failures.</p><p>Suppose you have a shared master setup and run an update that restarts all the pod services. The result is several restart loops because Kubernetes must choose a master from the available options. You can fix this by changing the update procedure from a direct, all-encompassing one to a sequential one (i.e., applying changes separately in each pod). This approach makes it easier to troubleshoot the cause of the restart loop.</p><p>In some cases, CrashLoopBackOff can occur as a settling phase to the changes you make. The error resolves itself when the nodes eventually receive the right resources for a stable environment.</p><h3>Container Failure due to Port Conflict</h3><p>Let’s take another example in which the container failed due to a port conflict. To identify the issue you can pull the failed container by running docker logs [container id].</p><p>Doing this will let you identify the conflicting service. Using netstat, look for the corresponding container for that service and kill it with the kill command. Delete the kube-controller-manager pod and restart.</p><h3>Troubleshoot CrashLoopBackOff</h3><p>The best way to identify the root cause of the error is to start going through the list of potential causes and eliminate them one by one, starting with the most common ones first.</p><h3>1. Check for “Back Off Restarting Failed Container”</h3><p>Run kubectl describe pod [name].</p><p>☑If you get a Liveness probe failed and Back-off restarting failed container messages from the kubelet, as shown below, this indicates the container is not responding and is in the process of restarting.</p><pre>From       Message<br>-----      -----<br>kubelet    Liveness probe failed: cat: can’t open ‘/tmp/healthy’: No such file or directory<br>kubelet    Back-off restarting failed container</pre><p>☑If you get the back-off restarting failed container message this means that you are dealing with a temporary resource overload, as a result of an activity spike. The solution is to adjust periodSeconds or timeoutSeconds to give the application a longer window of time to respond.</p><p>If this was not the issue, proceed to the next step.</p><h3>2. Check Logs From Previous Container Instance</h3><p>If Kubernetes pod details didn’t provide any clues, your next step should be to pull information from the previous container instance.</p><p>You originally ran kubectl get pods to identify the Kubernetes pod that was exhibiting the CrashLoopBackOff error. You can run the following command to get the last 5 log lines from the pod:</p><p>kubectl logs --previous --tail 5</p><p>Search the log for clues showing why the pod is repeatedly crashing. If you cannot resolve the issue, proceed to the next step.</p><h3>3. Check Deployment Logs</h3><p>Run the following command to retrieve the kubectl deployment logs:</p><p>kubectl logs -f deploy/ -n</p><p>This may also provide clues about issues at the application level.</p><p>Failing all the above, the next step is to bash into the CrashLoop container to see exactly what happened.</p><h3>Tips for Prevention</h3><p>In most cases, restarting the pod and deploying a new version will resolve the problem and keep the application online.</p><p>However, it is important to identify the root cause of the CrashLoopBackOff error and prevent it in the first place.</p><p>Best practices that can help you prevent the CrashLoopBackOff error.</p><h3>1. Configure and Recheck Your Files</h3><p>A misconfigured or missing configuration file can cause the CrashLoopBackOff error, preventing the container from starting correctly. Before deployment, make sure all files are in place and configured correctly.</p><p>In most cases, files are stored in /var/lib/docker. You can use commands like ls and find to verify if the target file exists.</p><h3>2. Be Vigilant With Third-Party Services</h3><p>If an application uses a third-party service and calls made to a service fail, then the service itself is the problem. Most errors are usually due to an error with the <em>SSL certificate or network issues</em>, so make sure those are functioning correctly.</p><p>You can log into the container and manually reach the endpoints using curl to check.</p><h3>3. Check Your Environment Variables</h3><p>Incorrect environment variables are a common cause of the CrashLoopBackOff error. A common occurrence is when containers require Java to run, but their environment variables are not set properly.</p><p>Use env to inspect the environment variables and make sure they’re correct.</p><h3>4. Check Kube-DNS</h3><p>The application may be trying to connect to an external service, but the kube-dns service is not running.</p><p>You just need to restart the kube-dns service so the container can connect to the external service.</p><h3>5. Check File Locks</h3><p>As mentioned before, file locks are a common reason for the CrashLoopBackOff error. Ensure you inspect all ports and containers to see that none are being occupied by the wrong service. If they are, kill the service occupying the required port.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07dcfe677221" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Azure DevOps? Services, Advantages & Tutorial]]></title>
            <link>https://medium.com/@navin205/what-is-azure-devops-services-advantages-tutorial-4624b8110eab?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/4624b8110eab</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 13:12:32 GMT</pubDate>
            <atom:updated>2024-10-16T13:12:32.195Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will take a look at Azure DevOps.</p><p>We will first show how to set up Azure DevOps through the Azure portal and take a look at each service it provides. We will look at some popular extensions for Visual Studio Code and show how to set them up, describe some advantages and disadvantages of using Azure DevOps, and finally, the alternatives to Azure DevOps.</p><p>Let’s dive in!</p><h3>What is Azure DevOps?</h3><p>Azure DevOps is a unified platform from Microsoft that combines a modern set of Developer services together. It can be used as a cloud Software-as-a-service (SaaS) offering or be hosted on an on-premise server.</p><p>Microsoft describes Azure DevOps as:</p><p>Azure DevOps provides developer services for allowing teams to plan work, collaborate on code development, and build and deploy applications. Azure DevOps supports a collaborative culture and set of processes that bring together developers, project managers, and contributors to develop software. It allows organizations to create and improve products at a faster pace than they can with traditional software development approaches</p><h3>Setting up Azure DevOps in The Cloud</h3><p>To access Azure DevOps in the Azure portal, search for DevOps in the service bar and select ‘Azure DevOps Organisations’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*SiQvae2ohR9PPXnf" /></figure><p>Click on ‘My Azure DevOps Organizations,’ then ‘create new organization’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*4CJn8T09XQIGfWYL" /></figure><p>Give your organization a unique name, and then create your first project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*NVwHrMxxq6PWXkug" /></figure><p>Hit ‘create project,’ and you’re done!</p><p>💡 You might also like:</p><ul><li><a href="https://spacelift.io/blog/devops-best-practices">16 DevOps Best Practices to Follow</a></li><li><a href="https://spacelift.io/blog/ci-cd-tools">Top Most Useful CI/CD Tools for 2023</a></li><li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves">Common Infrastructure Challenges and How to Solve Them</a></li></ul><h3>Azure DevOps Services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/0*JFU8ltMFhCWVe8gM" /></figure><p>When you are greeted by the Azure DevOps dashboard, the available services will be shown on the left menubar. Each service is tightly integrated with other services in the suite.</p><p>Let’s take a look at each service listed on the menu.</p><h3>Azure Boards</h3><p><a href="https://learn.microsoft.com/en-us/azure/devops/boards/get-started/what-is-azure-boards?view=azure-devops">Boards</a> enable you to track your work with a set of Agile tools. Among other capabilities, they can help you track progress, bugs, and issues using a Kanban board. They enhance collaboration and transparency within different teams, allowing for real-time updates and visibility into the status of different tasks and projects, and also have integrations available for Slack and MS Teams.</p><p>Boards basically provide a visual representation of the work, making it easy to set up priorities and ensuring the team is focused on delivering value efficiently.<br>Boards integrate seamlessly with other Azure DevOps tools, offering an end-to-end experience from tracking and planning until actual code deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/0*YsiLFUBI5C0d1udc" /></figure><h3>Azure Repos</h3><p><a href="https://learn.microsoft.com/en-us/azure/devops/repos/get-started/what-is-repos?view=azure-devops">Repos</a> provides Git repositories for you to store and version control your code. You can also use Team Foundation Version Control (TFVC), allowing teams to choose the best version control system for their needs.</p><p>Azure DevOps Repos offer advanced features such as branch policies, pull requests, and comments on open pull requests to enhance collaboration between teams. These features enforce more code reviews, build validations, and status checks, enabling teams to be more efficient without sacrificing development time.</p><p>Building branch policies and having a pull-request approach to merging code changes enhances security around your repositories, ensuring that no changes can be pushed directly to the main branch, thus minimizing the chances of human error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/0*8KYXKSreiPJd23Pu" /></figure><h3>Azure Pipelines</h3><p><a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/what-is-azure-pipelines?view=azure-devops">Pipelines</a> allow you to set up Continuous Integration and Continuous Delivery <a href="https://spacelift.io/blog/ci-cd-pipeline">pipelines (CI/CD)</a>, enabling teams to automate the building, testing, and deployment of application. These pipelines are written in <a href="https://spacelift.io/blog/yaml">YAML</a>, but you can also create them visually, allowing teams to choose the best approach for their workflows.</p><p><a href="https://spacelift.io/blog/azure-pipelines">Azure DevOps Pipelines</a> ensure that code changes are automatically integrated and delivered to various environments, improving the speed and reliability of the software delivery process.</p><p>One other key feature of Pipelines is the fact that you can create reusable pipelines that can be easily leveraged in others, making your pipeline-related code DRY and ensuring a smooth workload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/0*uiEkGFKrHXYNpIEP" /></figure><h3>Azure Test Plans</h3><p><a href="https://learn.microsoft.com/en-us/azure/devops/test/overview?view=azure-devops">Test plans</a> enable continuous testing and exploratory testing of your applications to ensure quality through a suite of tools.</p><p>These comprehensive tools help teams plan, track, and manage their testing efforts to ensure the delivery of high-quality software.</p><p>Teams can create and manage test plans, author and run manual test cases, and track progress and results in real time. This is really helpful in the software development lifecycle (SDLC), as test cases ensure code cannot be merged without passing all of the checkboxes defined.</p><p>With support for both manual and automated testing, Azure DevOps Test Plans is versatile, catering to various testing methodologies and practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/0*IcDdaO0HKzXCpKVC" /></figure><h3>Azure Artifacts</h3><p><a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/artifacts-overview?view=azure-devops">Artifacts</a> is a package repository allowing teams to share packages such as Maven, npm, and NuGet and integrate these packages into their build pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/0*w464ZtsC9QDzCMYX" /></figure><p>By using Artifacts, teams can efficiently manage the dependencies used in their codebase and ensure that all team members and CI/CD pipelines have access to the same set of packages, promoting consistency across different environments and stages of development.</p><p>It also allows teams to use packages from public repositories in addition to their private feeds. This is crucial for teams that depend on open-source packages but also want to ensure the availability and security of their dependencies.</p><p>Aside from those listed, you can also use the Wiki, which is useful for documenting projects, team dashboards for getting a project overview and visualizing metrics, and configuring notifications to let your team members know when they have been tagged or assigned work.</p><h3>Azure DevOps Advantages and Disadvantages</h3><p>Using Azure DevOps provides many advantages.</p><p>Azure DevOps is a SaaS service, so it provides Maintenance-free operations, including elastic scale, and as we have shown in this article, can be quickly set up. Access to cloud build and deployment servers is also provided. It is reliable and backed by a 99.9% SLA and 24×7 support. The development roadmap for Azure DevOps is published publically for full transparency so you can get full visibility of the new features coming soon.</p><p>It has a tight RBAC permission model integrated with Azure AD, enabling you to grant permission to particular parts on Azure DevOps as required.</p><p>However, if you need your data to stay within your on-premise organization, you may need to choose the Azure DevOps server version, which would sit within your trusted network. This will take longer to set up and incur management and hosting overheads. You will also need to provision build and deployment servers, meaning more servers and overhead. If you are currently running Team Foundation Server on-premises, Azure DevOps would be a natural upgrade if cloud hosting was out of the question.</p><p>Azure DevOps can support continuous delivery to any cloud, including AWS and Google Cloud. It can be used for development across any platform (Linux, macOS, and Windows) or language (including Node.js, Python, Java, PHP, Ruby, C/C++, .Net, Android, and iOS apps). Azure DevOps is not just aimed at organizations building and shipping .Net code.</p><h3>Azure DevOps Pricing</h3><p><a href="https://azure.microsoft.com/en-us/pricing/details/devops/azure-devops-services/">Azure DevOps pricing</a> is very flexible. You can either choose only the services you are interested in, or you can go for a bundle of all the services. If you opt for the basic bundle (which includes everything apart from tests), the first five users are free, and then you pay $6/month/user, or if you go for the full service, you will pay $52/month/user.</p><h3>Most Popular Extensions For Azure DevOps</h3><p>Azure DevOps supports extensions for integrating with other services and enhancing the functionality of some of the Azure DevOps services. Because of this, Azure DevOps is extremely flexible. It is not limited to tools in the Microsoft ecosystem.</p><p>You can browse and install hundreds of extensions from the <a href="https://marketplace.visualstudio.com/azuredevops/">Visual Studio Marketplace</a>.</p><p>Useful extensions include:</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=louisgerard.slackposter">Slack</a> — If your team uses Slack for collaboration, you can hook Azure DevOps notifications into a slack channel.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vsts.vss-services-teams">Microsoft Teams</a> — You can use this extension to get notifications for pipeline runs, board updates, and more.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens">Replace Tokens</a> — Azure Pipelines extension that replace tokens in <strong>text</strong> files with variable values.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=blueprint.vsts-open-work-items-in-excel">Azure DevOps Open in Excel</a> — This free extension adds a button that opens work items and query results in Excel directly from Azure DevOps.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms.vss-code-search">Code Search </a>— Code Search is a free extension that provides fast, flexible, and accurate search across all your code.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=SonarSource.sonarqube">Sonarqube </a>— Detect bugs, vulnerabilities and code smells across project branches and pull requests. It can be integrated with the Sonarqube community edition or commercial versions.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks">Terraform</a> — <a href="https://spacelift.io/blog/how-to-install-terraform">Install Terraform</a> and run Terraform commands to manage resources on Azure, AWS, and GCP.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscs-rm.vss-services-ansible">Ansible</a> — This extension executes an <a href="https://spacelift.io/blog/ansible-playbooks">Ansible playbook</a> using a specified inventory via the command line interface.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=chef-software.chef-azdo">Chef</a> — Tasks for performing common Chef operations against the Chef Automate platform.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=tsuyoshiushio.k8s-endpoint">Kubernetes </a>— Kubernetes extension with k8s endpoint. Which enables the kubectl, helm, and istioctl command.</li><li><a href="https://marketplace.visualstudio.com/items?itemName=piraces.bicep-tasks">Bicep</a> — Provides Azure DevOps tasks to install and run Microsoft Bicep CLI commands (cross-platform)</li></ul><h3>Azure DevOps Alternatives</h3><p>There are many alternatives to Azure DevOps, the biggest amongst them including:</p><ul><li>GitHub</li><li>GitLab</li><li>Bitbucket</li></ul><p>These provide similar capabilities in terms of Code repositories, CI/CD, and Agile tools, each with its own pros/cons.</p><p>Note that Microsoft owns GitHub, which is expected to replace Azure DevOps eventually but still lacks key features. However, most features are being included in GitHub Enterprise. It’s unlikely that Azure DevOps is going away in short to medium term. As of 2021, Microsoft itself held 98% of its code on Azure DevOps, which 80,000 internal users access.</p><p>If you need any help managing your infrastructure, <a href="https://docs.spacelift.io/integrations/source-control/azure-devops.html">Spacelift supports using Azure DevOps</a> as the source of code for your <a href="https://docs.spacelift.io/concepts/stack/">stacks</a> and <a href="https://docs.spacelift.io/vendors/terraform/module-registry">modules</a>. It is an alternative to using homegrown solutions on top of a generic CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure. You can check it for free by <a href="https://spacelift.io/free-trial">creating a trial account</a>.</p><h3>Key Points</h3><p>Azure DevOps is a mature, feature-rich platform that combines a modern set of Developer services together.</p><p>For more information, check out the <a href="https://learn.microsoft.com/en-us/azure/devops/get-started/?view=azure-devops">Azure DevOps documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4624b8110eab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Ways to Manage Terraform at Scale — Best Practices]]></title>
            <link>https://medium.com/@navin205/5-ways-to-manage-terraform-at-scale-best-practices-b3887736b98e?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3887736b98e</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 13:07:35 GMT</pubDate>
            <atom:updated>2024-10-16T13:07:35.344Z</atom:updated>
            <content:encoded><![CDATA[<h3>5 Ways to Manage Terraform at Scale — Best Practices</h3><p>Learning Terraform <em>can</em> be easy. The HCL configuration language is simple enough to start with basic resource creation straight away.</p><p>But —</p><p>Once the code that defines your infrastructure grows in complexity, maintainability, and dependency, an uphill battle can ensue. And that is why working out an efficient deployment workflow is crucial for further development.</p><p>In this article you will see five approaches to managing Terraform workflows at scale, what their benefits and downsides are, and what problems they address.</p><p><strong>Note:</strong> New versions of Terraform are placed under the BUSL license, but everything created before version 1.5.x stays open-source. <a href="https://opentofu.org/">OpenTofu</a> is an open-source version of Terraform that expands on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6.</p><h3>1) Work Locally</h3><p>A quick and easy way to start deploying resources using the Infrastructure as Code pattern is to use Terraform locally from the <strong>same machine you’re developing the code on</strong>. The only requirement is to have access to the target provider and an installed Terraform binary.</p><p>This approach makes it much easier to leverage commands related to day-to-day operations (such as state import, move, remove) or to get outputs than if doing it within the implementations where you have no direct access to the underlying provider.</p><p>Even though this is the easiest way to work with Terraform, this is not a best practice, because as soon as you scale and you have multiple DevOps Engineers working on your project, it will become very complicated.</p><p>You can also use all of the features of Terraform, such as remote state and state locking, to work with your teammates all-at-once. If you collaborate using a version control system and do not have continuous integration in place, but want to keep your code at a decent level of quality, there are <strong>tools to help you.</strong></p><p>One of these is <a href="https://github.com/antonbabenko/pre-commit-terraform">pre-commit-terraform</a> that will run a set of selected tools to lint the code you’re creating before you even commit it. Even if you have continuous integration available, it’s a handy way to avoid wasting time waiting for results from pull request status checks.</p><p>What are the <strong>drawbacks of this approach</strong>? Each action is manual and must be performed directly by the developer, which leads to several issues.</p><p>In a situation where multiple people are working on the same codebase, you could often encounter a scenario such as this one:</p><p><em>Person A applies their changes to the environment and follows up with a pull request. At the same time, Person B would like to deploy their changes but is blocked as the codebase is not up-to-date with Person A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. When working in larger teams, this will often result in a lot of wasted time just waiting and rebasing on a codebase to apply the changes.</em></p><p>If you happen to write unit tests for <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work">Terraform modules</a> (and you should be), you’re sure to know they need to be executed manually from your computer. Running them every time you change something in your configuration can be tedious and time-consuming. If you do not yet write tests for your modules, you’d be well advised to start looking at <a href="https://docs.spacelift.io/vendors/terraform/module-registry#tests">Spacelift modules tests</a> or <a href="https://terratest.gruntwork.io/docs/getting-started/quick-start/">Terratest</a>.</p><p>Privileged access to an underlying vendor is required, which can easily result in a compromised environment. It’s much harder to keep restricted access within this model, as Terraform must be allowed to create, change, and destroy resources. The same rule applies to a scenario where access to private (internal) resources is required, e.g., every developer has to set up a VPN connection to interact with them.</p><p>Moreover, every person working with the codebase needs access to the Terraform state, which creates a security concern due to how Terraform state works. Even if you’re pulling sensitive data from an external system such as Vault, it will be stored within the state file in plaintext once used in a resource. A threat actor could simply run `terraform state pull` to <strong>access all the secrets</strong>!</p><p>Keeping all these issues in mind, let’s take a look at a more automated approach.</p><h3>2) Implement Homegrown Automation</h3><p>Implementing homegrown automation means incorporating Terraform workflow into your continuous integration / continuous deployment process. A specific example would need to directly tie to a CI/CD tool you are using, so let’s try to keep the example as generic as possible.</p><p>As the applying actor is the CI/CD system, the hard requirement of privileged access for developers is gone. You can grant this type of access to the execution layer while developers keep read-only permissions. This is true for private (internal) resources, and if your code is being executed on an agent that resides within the target infrastructure with the proper permissions having been set. In a model where everything has to go through the <a href="https://spacelift.io/blog/terraform-in-ci-cd">CI/CD pipeline</a>,<strong> visibility of changes is improved</strong> as the system logs produced during the operation execution are available in real-time.</p><p>Linting, compliance checks, and automated unit tests can be moved to pull request status checks, and it’s up to you to configure these steps of the pipeline.</p><p>But —</p><p>If your CI/CD is executing jobs concurrently, you will end up with race conditions causing pull requests to fail non-deterministically.</p><p>It is not difficult to imagine a situation where two operators open a pull request within several seconds of one another. The pipeline gets triggered immediately for both, and one pull request will be marked as successful, while the other will be marked as failed. This situation will definitely happen if you are using state locking (it is a best practice and you really should be using it).</p><p>The explanation is simple: the first pull request has locked the state and therefore the second one could not, so it failed. This issue can be resolved by configuring queued runs, where only one pipeline can run at a time. However, this is not so simple with some CI/CD products.</p><p>The biggest issue and concern with this approach is the effort you have to put in to build a complete pipeline. If you consider requirements such as:</p><ul><li>planning on pull requests</li><li>linting</li><li>unit tests</li><li>compliance checks</li><li>applying once merged</li><li>periodical drift detection</li><li>registry for all your versioned modules</li><li>modules sharing across different parts of the organization</li><li>shared variable contexts/reusable values</li></ul><p>It might be tempting to implement solutions for those requirements by yourself. However, taking into account the time and effort required for doing it, you might want to take a look at alternatives, such as open-source tools.</p><p>💡 You might also like:</p><ul><li><a href="https://spacelift.io/blog/spacelift-module-registry">How to Use Spacelift Module Registry</a></li><li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift">Why DevOps Engineers Recommend Spacelift</a></li><li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves">Infrastructure Problems that Spacelift Solves</a></li></ul><h3>3) Use Open Source</h3><p>One of the most popular open-source tools to use with Terraform is Atlantis, which describes itself as “Terraform pull request automation”.</p><p>Atlantis provides a way for teams to collaborate on Terraform code changes by creating a pull request workflow. With Atlantis, developers can create, review, and merge Terraform code changes using familiar Git-based workflows. Atlantis also provides features such as automated plan and apply workflows, automatic branch cleanup, and integration with popular collaboration tools like Slack and GitHub.</p><p>It doesn’t offer a dedicated User Interface, you will need to use the one provided by your VCS provider, and if you want to use more advanced features like Policy as Code or Drift Detection, you will need to create the integrations inside of your pipeline, as it doesn’t come with these out of the box.</p><p>You can read more about it here: <a href="https://spacelift.io/blog/alternative-to-atlantis">Alternative to Atlantis</a>.</p><h3>4) Use Terraform Cloud</h3><p><a href="https://spacelift.io/blog/what-is-terraform-cloud">Terraform Cloud</a> is an application provided by the company behind Terraform itself — HashiCorp. It is available for free for up to 5 users. Two plans are available for more users: “Team &amp; Governance” and “Business”. As the product is created by Terraform authors themselves, any new feature that makes it to Terraform will be quickly available within Terraform Cloud as well. Neat!</p><p>See the comparison: <a href="https://spacelift.io/blog/atlantis-terraform">Atlantis vs. Terraform Cloud</a>.</p><h3>Advantages of Terraform Cloud</h3><p>Advanced security compliance mechanisms available through Sentinel allow you to enforce organization standards even before the code actually ships and creates resources with the chosen provider context.</p><p>Another important feature is a module registry that acts as an <strong>artifact repository</strong> for all your modules. It can be compared to JFrog Artifactory, Sonatype Nexus, or other similar software. Having every module in a separate repository and versioning allows you to control code promotion in a granular fashion or quickly roll back to if it allows you to go back to the most recent version only, then we should use “the”, but if it lets you go back to some other earlier version then we should use “a” previous version.</p><p>Having a centralized place to store all the pieces also enables you to <strong>quickly browse through</strong> available modules, their documentation, inputs, outputs, and used resources. It is a great single pane of glass in terms of module management. However, it can only be used within the same Terraform Cloud organization and cannot be shared. This might be troublesome for operations teams that base their separation on the organization level.</p><p>There’s also the <strong>remote execution backend</strong>, which is an exclusive feature for Terraform Cloud. It allows you to work with Terraform on your local computer as you would usually do, upload your local codebase, and run a plan against it. It is a very efficient way of quickly verifying if what you are working on is actually working.</p><h3>Disadvantages of Terraform Cloud</h3><p>Along with so many benefits, there are some drawbacks as well. Any linting or module unit tests need to be implemented inside your own continuous integration system. This requires an additional effort the development team has to make to sustain their code quality.</p><p>Integrations are limited to what Run Tasks are supporting at any given time, meaning that you cannot really own your workflow. You will need to manage with what you are receiving from Terraform Cloud.</p><p>Another disadvantage is the fact that you cannot control the steps that are happening before and after an operation (init/plan/apply). Run tasks can be added before or after a plan and also before an apply, but you cannot add them before an init or after an apply, which again shows a lack of flexibility, and you will need to build pipelines in Github Action or whatever other CI/CD tool you prefer, to do some simple checks before running your code. The only thing that you can actually do, is run remote Terraform command execution using the CLI, which is not that helpful overall.</p><p>The biggest drawback is the fact that <strong>Terraform Cloud doesn’t support any other products apart from Terraform</strong>. Of course, this is not its purpose, but as many organizations are using Kubernetes and Ansible in conjunction with Terraform, this would mean that you need to adopt other tools like <a href="https://spacelift.io/blog/argocd">ArgoCD</a> and Ansible Tower to manage your workflows end to end.</p><h3>5) Use Spacelift</h3><p>And here we are. You can <a href="https://spacelift.io/terraform-cloud-alternative">compare Spacelift directly to Terraform Cloud</a>. They share a lot of functionality with each other in terms of the actual deployment execution, but also in the area of compliance.</p><p>While Terraform Cloud uses Sentinel mostly (the HashiCorp approach to implementation for compliance as code), Spacelift leverages Open Policy Agent. It is beneficial to have a similar way of working with policies for companies that already incorporate OPA in their workflow (e.g., for Kubernetes). As there is <strong>no need to learn another syntax</strong>, operators can focus their efforts on providing compliance, instead of learning yet another tool.</p><p>With <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#custom-inputs">Spacelift’s custom inputs</a>, you can easily integrate any tool inside your workflow, but what really takes this to the next level is the fact that you can even write policies for it.</p><p>Do you want <em>tfscan </em>as part of your workflow? Install it and run it in a <a href="https://docs.spacelift.io/concepts/configuration/runtime-configuration/index.html#before_-and-after_-hooks">before_init hook</a> and save the output to a file with the following format &lt;key&gt;.custom.spacelift.json (where key can be anything you like). Access the data in a plan policy under input.third_party_metadata.custom.&lt;key&gt; and you can write whatever policy you want. Don’t know what data is getting exposed?</p><p>Use <a href="https://docs.spacelift.io/concepts/policy/#policy-workbench">Spacelift’s policy workbench</a> and play around with the data until you easily understand what policies you can write based on the output of the integrated tool.</p><p>Spacelift also allows you to automate your workflow even further. By having a built-in continuous integration for modules, you can shift your linters or unit tests really quickly into one place dedicated to Terraform. It helps you reduce the effort required to implement integration workflow by letting you utilize a solution that you are already using.</p><p><a href="https://docs.spacelift.io/concepts/stack/stack-dependencies">Stack Dependencies</a> can help you easily deploy everything that you need in one go and building these dependencies is just one click away. As Stack Dependencies are Directed Acyclic Graphs (DAG), one stack can depend on multiple stacks and it can be depended by multiple stacks, but loops cannot exist. This unlocks the possibility of creating a sophisticated workflow without having to build complex pipelines.</p><p>Sometimes you need to work with a Terraform state directly to import something, for example. In scenarios where you are leveraging Terraform Cloud, it is often impossible to do this without accessing the provider yourself. In contrast, Spacelift gives you the <strong>ability to run literally any command</strong> within the context of a particular codebase via <a href="https://docs.spacelift.io/concepts/run/task.html">tasks</a>. This way, you do not need to grant any additional permissions to import, move, or remove resources directly.</p><p>By utilizing <a href="https://docs.spacelift.io/concepts/configuration/context.html">contexts</a>, you can simply share anything you like across multiple stacks — whether it is a set of environment variables or a file. This is something that becomes much more valuable once your codebase complexity increases.</p><p>As mentioned in the previous paragraph, drift detection is something that becomes much more important once your infrastructure grows. With Spacelift, it is possible to <strong>schedule periodic drift detection</strong> on any stack. Going even further, you can enable automated remediation if any changes are found compared to what you have in your codebase.</p><h3>Best practices for managing Terraform at scale</h3><p>Regardless of how you decide to manage your Terraform code base, some best practices apply:</p><ul><li><strong>Use modules</strong> — Modules in Terraform allow you to package a collection of resources as a single logical unit. By doing so, you can create reusable components, making your codebase DRY and consistent. Modularizing code also improves code clarity and reduces duplication.</li><li><strong>Leverage Dynamic Blocks</strong> — Dynamic Blocks in Terraform let you conditionally create block constructs based on input variables or other logic. This can help reduce repetition in your code and adapt configurations to different scenarios or environments without significant structural changes.</li><li><strong>Use loops and conditionals</strong> — <strong>Count</strong> and <strong>for_each</strong>, allow you to iterate over lists or maps to create resources dynamically. This is particularly useful for scenarios where resource quantities or configurations are different based on input variables. Additionally, conditionals using the <strong>ternary(</strong><strong>? :)</strong> operator can enable or disable configurations based on certain criteria. You can also leverage the <strong>for </strong>loop when you are building expressions and you also have the ability to use <strong>if statements</strong> in the loop, allowing you to build any expression you want based on your input.</li><li><strong>Take advantage of variable validations</strong> — Starting from Terraform 0.13, you can include validation rules for input variables, ensuring that the values provided meet certain conditions before Terraform runs your configuration. This can prevent common configuration errors and ensure that provided input uses the expected formats or constraints.</li><li><strong>Manage your state remotely with locking enabled</strong> — Using remote backends like AWS S3 paired with DynamoDB for state locking enables safer collaboration among team members. Remote state management ensures state consistency, while state locking prevents concurrent modifications, reducing the risk of state corruption or unintended infrastructure changes.</li><li><strong>Scan your code for security vulnerabilities</strong> — Security is mandatory, especially when defining infrastructure. Tools like <a href="https://spacelift.io/blog/what-is-checkov"><strong>Checkov</strong></a>,<a href="https://spacelift.io/blog/what-is-tfsec"><strong> tfsec</strong></a>, or <a href="https://spacelift.io/blog/what-is-terrascan"><strong>Terrascan</strong></a> can scan your Terraform code for potential security misconfigurations or non-compliant definitions. Regularly scanning and addressing these vulnerabilities reduces the attack surface of your deployed infrastructure.</li><li><strong>Implement Policies — Open Policy Agent (OPA) </strong>offers a flexible way to define and enforce policies across your Terraform configurations. Using a declarative language called Rego, you can create policies that restrict certain resource configurations, enforce naming conventions, or ensure compliance. Integrating OPA checks into your CI/CD pipeline helps maintain consistent adherence to these policies as your Terraform code evolves.</li><li><strong>Implement Linting</strong> — Linting refers to the process of analyzing code for potential errors, code inconsistencies, and ensuring it matches the best practices. In the context of Terraform, linting can help catch syntax erros, deprecated code usage, or misconfigurations before they cause issues during an apply.</li><li><strong>Test your code</strong> — Testing is a fundamental part of any software development project. Using tools such as Terratest or Kitchen-Terraform can help with writing integration tests for your Terraform configurations, ensuring your code is working properly before being deployed to the production environment/</li><li><strong>Write thorough documentation</strong> — Documentation is vital for understanding the purpose and usage of your Terraform configurations, especially when collaborating with others. Terraform-docs is a handy tool that can automatically generate documentation for your Terraform modules, providing an overview of inputs, outputs, providers, and more. Integrating this into your CI/CD pipeline ensures that your documentation stays up-to-date with your code changes.</li></ul><h3>Key Points</h3><p>There are many ways of working with Terraform. Each way is different in terms of complexity, and has a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most times, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may often exceed its potential benefits. It is much easier and quicker to leverage platforms such as <a href="https://spacelift.io/">Spacelift</a> to provide these features for you instead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3887736b98e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Generic CI/CD Tools Will Not Deliver Successful IaC]]></title>
            <link>https://medium.com/@navin205/why-generic-ci-cd-tools-will-not-deliver-successful-iac-6a65a2b41c32?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a65a2b41c32</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 13:04:49 GMT</pubDate>
            <atom:updated>2024-10-16T13:04:49.815Z</atom:updated>
            <content:encoded><![CDATA[<p>Infrastructure as code (IaC) is still in its infancy. According to the August 2022 <a href="https://www.gminsights.com/industry-analysis/infrastructure-as-code-market">Global Market Insights Inc</a>, the market is predicted to grow from $1B to $3.5B by 2030, at an annual rate of around 22%, and <a href="https://www.emergenresearch.com/industry-report/infrastructure-as-code-market">other reports</a> predict similar growth. As the market matures, the complexity of adoptions will increase, intensifying the need for adequate processes and tools.</p><h3>Requirements for successful IaC</h3><p>In this post, we focus on the four key elements that successful IaC delivers and generic CI/CD tooling does not:</p><ol><li><a href="https://spacelift.io/blog/infrastructure-as-code-with-generic-ci-cd#1-automation">Automation</a></li><li><a href="https://spacelift.io/blog/infrastructure-as-code-with-generic-ci-cd#2-continuous-processes">Continuous processes</a></li><li><a href="https://spacelift.io/blog/infrastructure-as-code-with-generic-ci-cd#3-infrastructure-orchestration-">Infrastructure orchestration</a></li><li><a href="https://spacelift.io/blog/infrastructure-as-code-with-generic-ci-cd#4-governance">Governance</a></li></ol><p>We outline how a dedicated solution can help businesses attain these four pillars and successfully manage their IaC, debunking the myths around dedicated IaC solutions, and concluding with the benefits of choosing a specialized platform for managing IaC successfully.</p><h3>1. Automation</h3><p>IaC emerged from the Automation element of the CALMS (Culture, Automation, Lean, Measurement, and Sharing) framework and is a pillar of the modern DevOps Approach. Automation has always been a goal of engineers, who previously used scripting languages like Bash or even programming languages to try to achieve it.</p><p>Ultimately, Configuration Management emerged as a way to manage operating systems configuration, application deployments, and configurations; and IaC was introduced to manage cloud infrastructure in a way that <a href="https://spacelift.io/blog/business-benefits-of-iac">maximized business value</a>.</p><p>Apart from delivering business benefits, IaC allowed Agile teams to manage their own infrastructure more easily and effectively. Interaction with infrastructure (even cloud infrastructure) was no longer siloed but could be equally distributed among multiple teams. However, this could also prompt issues that we will discuss later.</p><h3>2. Continuous processes</h3><p>IaC certainly introduced valuable benefits, but the process only automated single executions — not the entire process. Many organizations implemented CI/CD for applications, but infrastructure was treated separately, and automation (in the form of Configuration Management or IaC) was still executed manually. This led to significant issues, including inconsistent executions, incomplete understanding of the process inside teams, lack of communication, long delays, and lack of control. These made the process for infrastructure unreliable and unpredictable.</p><p>The solution was CI/CD processes and tools.</p><p>According to the <a href="https://cd.foundation/state-of-cd-2023/">State of CD report</a>, if developers are involved in DevOps activities, 29% of them manage and provision infrastructure programmatically, with different kinds of IaC tools. The report also states that when using multiple DevOps technologies, companies can decrease lead time, increase deployment frequency, and ultimately decrease the MTTR (Mean Time To Restore).</p><p>Combining IaC execution with <a href="https://spacelift.io/blog/ci-cd-pipeline">CI/CD pipelines</a> finally allowed organizations and teams to automate processes, not just executions.</p><p>Let’s look at the diagrams below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bdBnqd1G6gSBUN3i" /></figure><p>In the simplified picture of the process above, we clearly see the discontinuity between change and execution. We need two manual, unconnected steps: One is publishing the code of IaC, and the second concerns deploying to the target environment (in this case, AWS).</p><p>This leads to multiple problems, such as gaps in governance and compliance, broken processes, etc. CI/CD allows us to fix these gaps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Mu6AfQkLEAzuSo9" /></figure><p>Adding an automated process of execution through CI/CD not only completes the flow, but also removes manual steps from the process. We can clearly understand what change triggers execution. We can control the process by understanding the whole flow.</p><p>Combining those two approaches — for infrastructure and for delivery — created a <em>continuous</em> flow, which should be instant in terms of trigger. There is no delay between committing the code and action triggered in the CI/CD tool. In this way, the organization significantly reduces one of the Lean’s waste — <a href="https://theleanway.net/The-8-Wastes-of-Lean">waiting state</a>.</p><h3>3. Infrastructure orchestration</h3><p>In the CI/CD world, specialized tools like ArgoCD or Octopus focus on one function or one group of products, whereas generic CI/CD tools like Jenkins, CircleCI, and GitHub Actions cover the whole Continuous Integration/Continuous Delivery (or Deployment) field.</p><p>The main limitation of CI/CD tools derives from the design of the approach — not the specific tool — and is a default behavior of this type of tool. DevOps is pictured as an infinite loop, whereas the CI/CD pipeline is represented by a straight line. This line is a part of the infinite loop, of course, but the problem here is that one execution of the CI/CD pipeline is a single, encapsulated entity, which is not tightly connected to the whole lifecycle.</p><p>Let’s look at the simplified view of the generic pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*txpLLlfCAEia5bfD" /></figure><p>How do engineering teams build IaC code?</p><p>Well, there is no one answer to this question. However, in more complicated solutions we can observe multiple templates and even multiple stacks with very focused functions. Create a network layer, create a database layer, a storage layer and so on.</p><p>The diagram below illustrates this approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iRYwSONIrLWAZU10" /></figure><p>We see a complete disconnection between pipelines, which leads to multiple issues.</p><p>As the organization grows, so does the infrastructure. In the case of application-focused pipelines, it requires more work to properly manage the growing number of application pipelines.</p><p>Using the architecture patterns of microservices in the proper way eliminates this obstacle, but it also creates siloed areas of responsibility and management.</p><p>In the case of infrastructure, engineering teams apply many solutions, and there is almost always a disconnect between infrastructure elements in different stacks. As the infrastructure grows, the complexity of managing stacks also intensifies because more stacks are created and executed, and the increased connections between these stack executions must be managed.</p><p>Modern CI/CD tools allow us to trigger one pipeline from another or configure the pipeline execution in a flexible way, but this is just a workaround for the main problem — a lack of <em>orchestration.</em></p><h3>4. Governance</h3><p>Organizations should enforce three main goals for governance when thinking about IaC: compliance, security, and cost control. These elements are not relevant to scale. All organizations want to be secure and run their business in a cost-effective manner. However, the complexity of governance management grows as companies scale. Furthermore, companies operating in sectors such as finance, energy, or healthcare are governed by specific industry regulations.</p><p>Generic CI/CD tooling can be adapted to these restrictions from a technical standpoint, but this results in a siloed approach with risks for collaboration, communication, and understanding of the process.</p><p>Proper security for IaC is crucial. Infrastructure is highly sensitive to all vulnerabilities, not just as a potential vector of attack for cybercriminals, but also from the perspective of the risks for the infrastructure’s stability and scalability incorporated in its design. According to Prisma Cloud’s <a href="https://www.paloaltonetworks.com/state-of-cloud-native-security">State of Cloud-native Security Report 2023</a>, 90% of organizations struggle with detecting and fixing cyber threads within an hour. Of the respondents to this report, 78% stated that they are expected to have minimal learning for cloud security.</p><p>This should be an “aha” moment for organizations. We enforce a shift-left approach, which is appreciated by teams, but the point is that security for infrastructure is different from security for applications, and development teams shouldn’t be forced to learn a whole new area. And here we hit a huge drawback of generic CI/CD tools, where teams are fully responsible for their pipelines. Even with shared libraries etc., the team that establishes a new pipeline must be aware of the configuration.</p><p>💡 You might also like:</p><ul><li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift">Why DevOps Engineers Recommend Spacelift</a></li><li><a href="https://spacelift.io/blog/introducing-spacelift-self-hosted">Take Control with Spacelift Self-Hosted on AWS</a></li><li><a href="https://spacelift.io/blog/getting-started-with-spacelift-self-hosted">Why is it Worth Using Self-Hosted Solution</a></li></ul><h3>The solution: Use dedicated tools</h3><p>Generic CI/CD tools can achieve only very loose coupling between automation, continuous processes, orchestration, and governance.</p><p>This is because the main goal of CI/CD tools is the continuous integration and delivery of code to production systems. CI/CD utilizes automation, but from a process perspective, it doesn’t matter for the CI/CD tool what code it ships. CI/CD can orchestrate and govern the entire setup, but it is mostly manual work and requires specialized knowledge from all development teams.</p><p>This decoupling is shown in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*3KPF8bK_vGY0uaZ7" /></figure><p>Dedicated tools like Spacelift encapsulate the whole spectrum of infrastructure SDLC in one tool, as in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*fy8oFSlmjpUq3_1y" /></figure><p>With this in mind, organizations can control the whole process from all angles, like tooling, processes themselves, governance, security, and compliance.</p><h3>Debunk the myths about dedicated tools</h3><p>It is a common perception that organizations should limit the number of tools in the delivery chain. This does not mean that organizations should not use specialized tools for their chain of operations. We have to analyze the business value of the tool:</p><ul><li>Does the tool decrease the complexity of the SDLC?</li><li>What part of SDLC can be covered by the tool?</li><li>Does the organization have the skills required to use the tool properly?</li><li>Does the tool affect the lead time?</li><li>Does the tool affect service restoration?</li></ul><h4>Add the proper tool to improve lead time</h4><p>Chapter 4.3 of the State of the CD Report 2023 mentioned earlier confirms that organizations with diversified tooling are more likely to be top performers in terms of lead time for code changes. The important thing is to use a prudent approach to incorporating tools in the SDLC — not simply add more tools for the sake of it.</p><h4>Improve distribution of skills in your teams</h4><p>Another myth is that more tools means you’ll need more skills. If the organization considers all business needs and opportunities prudently, the opposite might be the case. If continuous automation is characterized by proper orchestration and governance, workloads are decreased for <a href="https://teamtopologies.com/key-concepts">stream-aligned teams</a>.</p><h4>Pay for the tool, not for work</h4><p>It is common to view tools, licenses, etc. as costs for the organization. However, if the tool decreases the amount of work needed, the organization lowers its operational costs. Upskilling all development teams to use specific IaC, CI/CD, and orchestration tools will be more time- and effort-consuming than using a dedicated tool within a dedicated team.</p><h4>Don’t be discouraged by vendor lock-in</h4><p>The vendor lock-in dilemma is raised frequently, especially in the cloud-native approach. The perception of avoiding vendor lock-in at any cost is unfortunate and might lead to unnecessary issues in processes. Every organization is subject to some vendor lock-in, by selecting a cloud vendor, software vendors, programming languages, or even a specific delivery methodology.</p><p>Generic CI/CD tools are simply not up to the task of fully controlling and monitoring infrastructure. As we discussed previously, they don’t know what technology they ship. They don’t know what happens before or after a pipeline is executed. The CI/CD pipeline is designed to deliver the code from VCS to production in an automated, controlled way.</p><p>Spacelift’s platform includes delivery pipelines but it is also aware of the technology the organization uses and is able to orchestrate and govern the whole infrastructure.</p><p>Let’s take a look at these elements one by one:</p><h3>Awareness of AUTOMATION</h3><p>Spacelift is aware of infrastructure technology. During the process of <a href="https://docs.spacelift.io/concepts/stack/creating-a-stack#configure-backend">creating the stack</a>, the proper backend technology must be selected — for example, Terraform. This determines the behavior of the run. Applying specific <a href="https://docs.spacelift.io/concepts/policy/">policies</a> also increases this awareness of the technology used in the process.</p><h3>CONTINUOUS processes</h3><p>As core functionality, Spacelift stacks are <a href="https://docs.spacelift.io/concepts/stack/creating-a-stack#integrate-vcs">connected to the Version Control System</a> of your choice, and each <a href="https://docs.spacelift.io/concepts/run/">run</a> can be executed after committing code to the repository. Furthermore, this functionality is also used for the pull requests process. This approach decreases the lead time significantly and allows control of the executions in a very granular way.</p><h3>ORCHESTRATE the SDLC</h3><p>Spacelift orchestrates all your stacks, executions and technologies through a couple of mechanisms implemented. With <a href="https://docs.spacelift.io/concepts/blueprint/">Blueprints,</a> organizations can enforce optimum infrastructure quality and security. <a href="https://docs.spacelift.io/concepts/spaces/">Spaces</a> allow organizations to control the executions and privileges in a very granular way. This is achieved by specific configuration of access for users, variables, and variable sets (called <a href="https://docs.spacelift.io/concepts/configuration/context">contexts</a>). To enforce orchestration, Spacelift offers <a href="https://docs.spacelift.io/concepts/policy/">policies</a>, <a href="https://docs.spacelift.io/concepts/stack/stack-dependencies">stack dependencies</a>, and <a href="https://docs.spacelift.io/concepts/stack/stack-settings#customizing-workflow">customizations of the workflow</a>.</p><h3>Control your infrastructure with properly implemented GOVERNANCE</h3><p>Finally, Spacelift allows organizations to implement and enforce the proper governance models.</p><p>Security of the process can be enhanced through Blueprints, Spaces, policies, and granular access mentioned previously. For Terraform, this can also be done through <a href="https://docs.spacelift.io/vendors/terraform/module-registry">Terraform registry</a>.</p><p>Execution can be secured easily by implementing security frameworks for the runs, and, most importantly, this implementation can be enforced by policies applied to stacks.</p><p>Implementation of cost control is possible for Terraform stacks using custom inputs and policies with Infracost. Finally, with <a href="https://docs.spacelift.io/concepts/resources">full view on the resources</a> deployed, the organization is able to understand and control all infrastructure resources.</p><h3>Key points</h3><p>Spacelift allows organizations to use one tool to conveniently govern and orchestrate the continuous automation of infrastructure. This allows businesses to increase the cohesion of their infrastructure, decrease the lead time, and restore time.</p><p>The table below sums up the advantages of using Spacelift as a dedicated tool for infrastructure management:</p><p><strong>Increase Decrease </strong>Infrastructure cohesion Lead time Deployment frequency Restore time Control over lifecycle Manual processes and control Security Learning curve Process’s awareness Tooling costs</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a65a2b41c32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy Infrastructure in CI/CD Using Terraform (Pipeline)]]></title>
            <link>https://medium.com/@navin205/how-to-deploy-infrastructure-in-ci-cd-using-terraform-pipeline-fedc994db1d1?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/fedc994db1d1</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 12:57:56 GMT</pubDate>
            <atom:updated>2024-10-16T12:57:56.631Z</atom:updated>
            <content:encoded><![CDATA[<p>Terraform by itself automates many tasks: it creates, changes, and versions your cloud resources. Although many teams run Terraform locally (sometimes with wrapper scripts), running Terraform in CI/CD can boost your organization’s performance and ensure consistent deployments.</p><p>In this article, you will see a review of the different approaches to integrating Terraform into generic deployment pipelines. Let’s start with the basics.</p><h3>What is Terraform</h3><p>Terraform is an infrastructure-as-code (IaC) tool created by HashiCorp that recently switched from an open-source license to a BSL one. It allows developers and infrastructure teams to define and provision infrastructure using a declarative configuration language.</p><p>It is stateful, maintaining a state file to track the current version of your infrastructure. This state allows Terraform to determine what changes need to be made to achieve the desired state defined in the configuration files.</p><p>Read more: <a href="https://spacelift.io/blog/terraform-use-cases">9 Terraform Use Cases</a></p><h3>Terraform workflow</h3><p>A Terraform workflow typically refers to the sequence of steps that engineers follow to write, plan, and apply infrastructure configurations using Terraform.</p><p>A standard workflow involves the following stages:</p><ul><li>Writing the HCL Code</li><li>Initializing the working directory using “terraform init”</li><li>Validating the configuration using “terraform validate”</li><li>Formatting the configuration code using “terraform fmt”</li><li>Planning the configuration changes using “terraform plan”</li><li>Applying the configuration changes using “terraform apply”</li><li>Tearing down the configuration using “terraform destroy</li></ul><p>Here’s how to deploy your infrastructure in CI/CD using Terraform:</p><h3>1. Store the Terraform Code</h3><p>Storing Terraform code in the same repository as the application code or maintaining a separate repository for the infrastructure? This question has no strict and clear answer, but here are some insights that may help you decide:</p><ul><li>The <a href="https://spacelift.io/blog/what-is-terraform">Terraform</a> and application code coupled together represent one unit, so it’s simple to maintain by one team;</li><li>Conversely, if you have a dedicated team that manages infrastructure (e.g., platform team), a separate repository for infrastructure is more convenient because it’s a standalone project in that case.</li><li>When infrastructure code is stored with the application, sometimes you have to deal with additional rules for the pipeline to separate triggers for these code parts. But sometimes (e.g., serverless apps) changes to either part (app/infra) should trigger the deployment.</li></ul><p>There is <strong>no right or wrong approach</strong>, but whichever you choose, remember to follow the <em>Don’t Repeat Yourself (DRY)</em> principle: make the infrastructure code modular by logically grouping resources into higher abstractions and reusing these modules.</p><h3>2. Prepare Terraform Execution Environment</h3><p>Running Terraform locally generally means that all dependencies are already in-place: you have the Terraform installed and present in the user’s PATH and providers already stored in the .terraform directory.</p><p>But, when you shift Terraform runs from your local machine to stateless pipelines, this is not the case. However, you can still have a pre-built environment — this will speed up the pipeline execution and provide control over the process.</p><p>Docker image with a Terraform binary is one of the popular solutions that address this. Once created, you can execute Terraform within a container context with configuration files mounted as a <a href="https://spacelift.io/blog/docker-volumes">Docker volume</a>.</p><p>You can use the official <a href="https://hub.docker.com/r/hashicorp/terraform/">image from Hashicorp</a>, but sometimes, it’s wise to maintain your own Docker images with additional tools you may need. For instance, you can bake the <a href="https://spacelift.io/blog/what-is-tfsec">tfsec </a>tool into the image to use for security inspection and have it ready inside the Docker container without the need to install it every single time.</p><p>Here is an <a href="https://spacelift.io/blog/dockerfile">example of a Dockerfile</a> that builds an image with a custom Terraform version (you can override it as a build argument) and a tfsec tool.</p><p>This example also shows how to verify the <a href="https://spacelift.io/blog/how-to-install-terraform">installed Terraform</a> binary to make sure it’s signed by HashiCorp before we run it.</p><pre>dockerfile<br>FROM alpine:3.14<br>ARG TERRAFORM_VERSION=1.0.11<br>ARG TFSEC_VERSION=0.59.0<br>RUN apk add --no-cache --virtual .sig-check gnupg<br>RUN wget -O /usr/bin/tfsec https://github.com/aquasecurity/tfsec/releases/download/v${TFSEC_VERSION}/tfsec-linux-amd64 \<br>    &amp;amp;&amp;amp; chmod +x /usr/bin/tfsec<br>RUN cd /tmp \<br>    &amp;amp;&amp;amp; wget &quot;https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip&quot; \<br>    &amp;amp;&amp;amp; wget https://keybase.io/hashicorp/pgp_keys.asc \<br>    &amp;amp;&amp;amp; gpg --import pgp_keys.asc \<br>    &amp;amp;&amp;amp; gpg --fingerprint --list-signatures &quot;HashiCorp Security&quot; | grep -q &quot;C874 011F 0AB4 0511 0D02  1055 3436 5D94 72D7 468F&quot; || exit 1 \<br>    &amp;amp;&amp;amp; gpg --fingerprint --list-signatures &quot;HashiCorp Security&quot; | grep -q &quot;34365D9472D7468F&quot; || exit 1 \<br>    &amp;amp;&amp;amp; wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_SHA256SUMS \<br>    &amp;amp;&amp;amp; wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig \<br>    &amp;amp;&amp;amp; gpg --verify terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig terraform_${TERRAFORM_VERSION}_SHA256SUMS || exit 1 \<br>    &amp;amp;&amp;amp; sha256sum -c terraform_${TERRAFORM_VERSION}_SHA256SUMS 2&amp;gt;&amp;amp;1 | grep -q &quot;terraform_${TERRAFORM_VERSION}_linux_amd64.zip: OK&quot; || exit 1 \<br>    &amp;amp;&amp;amp; unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /bin \<br>    &amp;amp;&amp;amp; rm -rf /tmp/* &amp;amp;&amp;amp; apk del .sig-check</pre><p>But the main functionality of <a href="https://spacelift.io/blog/terraform-providers">Terraform is delivered by provider plugins</a>. It takes time to download the provider: for example, the AWS provider is about 250MB, and in a large scale, with hundreds of Terraform runs per day, this makes a difference.</p><p>There are <strong>two common ways to deal with it</strong>: either use a shared cache available to your pipeline workloads or bake provider binaries into the runtime environment (i.e., Docker image).</p><p>The critical element of both approaches is the configuration of the plugin cache directory path. By default, Terraform looks for plugins and downloads them in the .terraform directory, which is local to the main project directory. But you can override this by leveraging the TF_PLUGIN_CACHE_DIR environment variable to do that.</p><p>If supported by your CI/CD tool, the shared cache can significantly reduce the operational burden because all your pipeline runtime environments can use it to get the needed provider versions.</p><p>That way, all you have to do is to maintain the provider versions in the shared cache and instruct Terraform to use it:</p><ul><li>Mount the cache directory to the pipeline runtime (i.e., docker container) and specify its internal path.</li><li>Set the value of the TF_PLUGIN_CACHE_DIR environment variable accordingly.</li></ul><p>On the other hand, you can bake the provider binaries into the Docker image and inject the value for the TF_PLUGIN_CACHE_DIR environment variable right into the Dockerfile. This approach takes more operational effort but makes the <a href="https://spacelift.io/blog/terraform-environments">Terraform environment</a> self-sufficient and stateless. It also allows you to set strict boundaries around permitted provider versions as a security measure.</p><p>💡 You might also like:</p><ul><li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale">5 Ways to Manage Terraform at Scale</a></li><li><a href="https://spacelift.io/blog/devops-best-practices">16 DevOps Best Practices to Follow</a></li><li><a href="https://spacelift.io/blog/atlantis-terraform-tutorial">Using Atlantis for Terraform Pull Request Automation</a></li></ul><h3>3. Plan and Apply Changes</h3><p>Now, let’s review the ways to automate planning and applying of changes. Although terraform apply can do both, it’s sometimes useful to separate these actions.</p><h3>Initialization</h3><p>CI/CD pipelines generally run in stateless environments. Thus, every subsequent run of Terraform looks like a fresh start, so the project needs to be initialized before other actions can be performed.</p><p>The usage of the init command in CI/CD slightly differs from its common local usage:</p><pre>&amp;gt; terraform init -input=false</pre><p>The -input=false option prevents Terraform CLI from asking for user actions (it will throw an error if the input was required).</p><p>Also, there is the -no-color option that prevents the usage of color codes in a shell, so the output will look much cleaner if your CI/CD logging system cannot render the terminal formatting.</p><p>Another aspect of the <a href="https://spacelift.io/blog/terraform-init">init command</a> that is useful in CI is -backend-config. This option allows you to <strong>override the backend configuration</strong> in your code or define it if you prefer to use partial configuration, thus creating more uniform pipelines.</p><p>For example, here’s how you can use the same code with different roles in different environments on AWS:</p><pre>&amp;gt; terraform init -input=false \<br>-backend-config=&quot;role_arn=arn:aws:iam::012345678901:role/QADeploymentAutomation&quot;</pre><p>However, I suggest checking in .terraform.lock.hcl as suggested by HashiCorp (<a href="https://www.terraform.io/docs/language/dependency-lock.html">Dependency Lock File</a>): this way you will be able to control dependencies more thoroughly, without worrying about transferring this file between build stages.</p><h3>Plan</h3><p>The terraform plan command helps you validate the changes manually. Still, there are ways to use it in automation as well.</p><p>By default, Terraform prints the plan output in a human-friendly format, but also supports machine-readable JSON. With additional command line options, you can extend your CI experience. For example, you can use validation conditions to decide whether to apply the changes automatically; or, you can parse the plan details and integrate the summary into a Pull Request description. Let’s review a simple example that illustrates this.</p><p>First, save the plan output to the file:</p><pre>&amp;gt; terraform plan -input=false -compact-warnings -out=plan.file</pre><p>The main point here is the -out option — it tells Terraform to save its output into a binary plan file. We will also talk about it later in this section.</p><p>The -compact-warnings option suppresses the warning-level messages produced by Terraform</p><p>Also, the plan command has -detailed-exitcode option that returns detailed exit codes when the command exits. For example, you can leverage this in a script that wraps Terraform and adds more conditional logic to its execution, because CIs will generally fail the pipeline on a command’s non-zero exit code. However, that may add complexity to the pipeline logic. So if you need to get detailed info about the plan, I suggest parsing the plan output.</p><p>Once you have a plan file, you can read it in JSON format and parse it. Here is a code snippet that illustrates this:</p><pre>&amp;gt; terraform show -json plan.file| jq -r &#39;([.resource_changes[]?.change.actions?]|flatten)|{&quot;create&quot;:(map(select(.==&quot;create&quot;))|length),&quot;update&quot;:(map(select(.==&quot;update&quot;))|length),&quot;delete&quot;:(map(select(.==&quot;delete&quot;))|length)}&#39;<br>{<br>  &quot;create&quot;: 1,<br>  &quot;update&quot;: 0,<br>  &quot;delete&quot;: 0<br>}</pre><p>Another way to see the information about changes, is to <a href="https://spacelift.io/blog/terraform-plan">run the </a><a href="https://spacelift.io/blog/terraform-plan">plan command</a> with -json option and parse its output to stdout (available starting from Terraform 1.0.5):</p><pre>&amp;gt; terraform plan -json|jq &#39;select( .type == &quot;change_summary&quot;)|.&quot;@message&quot;&#39;<br>&quot;Plan: 1 to add, 0 to change, 0 to destroy.&quot;</pre><p>This technique can make your Pull Request messages more informative and improve your collaboration with teammates. (Btw. <a href="https://docs.spacelift.io/concepts/policy/push-policy/#push-and-pull-request-events">Spacelift</a> provides this feature out of the box).</p><p>You can write a custom script/function that sends a Pull Request comment to VCS using its API. Or, you can try the existing features of your VCS: with <a href="https://spacelift.io/blog/github-actions-terraform">GitHub Actions</a>, you can use the <a href="https://github.com/marketplace/actions/terraform-pr-commenter">Terraform PR Commenter</a> or similar action to achieve this; for GitLab, there is a built-in functionality that integrates plan results into the Merge Request. See more here: <a href="https://docs.gitlab.com/ee/user/infrastructure/iac/mr_integration.html">Terraform Integration in Merge Requests</a></p><p>And you can find more information about the specification of the JSON output here: <a href="https://www.terraform.io/docs/internals/json-format.html">Terraform JSON Output Format</a>.</p><h3>Apply</h3><p>Once the plan file is ready, and the proposed changes are expected and approved, it’s time to apply them.</p><p>Here is how the apply command may look like in automation:</p><pre>terraform apply -input=false -compact-warnings plan.file</pre><p>The plan.file is the file we got from the previous plan step.</p><p>Alternatively, you might want to omit the planning phase altogether. In that case, the following command will apply the configuration immediately, without the need for a plan:</p><pre>terraform apply -input=false -compact-warnings -auto-approve</pre><p>Here, the -auto-approve option tells Terraform to create the plan implicitly and skip the interactive approval of that plan before applying.</p><p>Whichever way you choose, keep in mind the destructive nature of the apply command. Hence, the fully automated application of configuration generally works well with environments that tolerate unexpected downtimes, such as development or testing. On the other hand, plan review is recommended for production-grade environments, in which case the `apply` job is configured for a manual trigger.</p><h3>4. Deal with Stateless Environments</h3><p>If you run init, plan, and apply commands in different environments, you need to take care of some artifacts produced by Terraform:</p><ul><li>The .terraform directory with information about <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work">Terraform modules</a>, providers, and the state file (even in the case of remote state).</li><li>The <a href="https://spacelift.io/blog/terraform-lock-hcl-file">.terraform.lock.hcl file</a> — the dependency lock file which Terraform uses to check the integrity of provider versions used for the project. If your VCS does not track it, you’ll need to pass that file to the plan and apply commands to make them work after init.</li><li>The output file of the plan command is essential for the apply command, so treat it as a vital artifact. This file includes a full copy of the project configuration, the state, and variables passed to the plan command (if any). Therefore, mind the security precautions because sensitive information may be present there.</li></ul><p>Make sure to pass them successively between operations: init -&gt; plan -&gt; apply.</p><p>There is one shortcut, though. You can execute the init and plan commands within the same step/stage and transfer the artifacts only once — to the apply execution.</p><h3>5. Use the Command Line and Environments Variables</h3><p>Last but not least, a few words on ways to maximize the advantage of <a href="https://spacelift.io/blog/how-to-use-terraform-variables">variables when running Terraform</a> in CI.</p><p>There are two common ways of passing values for the variables used in the configuration:</p><p><strong>1.</strong> Using a -var-file option with the variable definitions file — a filename ending in .tfvars or .tfvars.json. For example:</p><pre>terraform apply -var-file=development.tfvars -input=false -no-color -compact-warnings -auto-approve</pre><p>Also, Terraform can automatically load the variables from files named exactly terraform.tfvars or terraform.tfvars.json: with that approach, you don’t need to specify the tfvar file as a command option explicitly.</p><p><strong>2.</strong> Using environment variables with the prefix TF_VAR_. Implicitly, Terraform always looks for the environment variables (within its process context) with that prefix, so the same “instance_type” variables from the example above can be passed as follows:</p><pre>export TF_VAR_instance_type=t3.nano<br>terraform -input=false -no-color -compact-warnings -auto-approve</pre><p>The latter method is widely used in CI because modern CI/CD tools support the management of the environment variables for automation jobs.</p><p>Please refer to the following official documentation if you want to know more about variables: <a href="https://www.terraform.io/docs/language/values/variables.html">Terraform Input Variables</a>.</p><p>Additionally, Terraform supports several configuration parameters in the form of environment variables. These parameters are optional; however, they can simplify the automation management and streamline its code.</p><ul><li>TF_INPUT — when set to “false” or “0”, this tells Terraform to behave the same way as with the -input=false flag;</li><li>TF_CLI_ARGS — can contain a set of command-line options that will be passed to one or another Terraform command. Therefore, the following notation can simplify the execution of apply and plan commands by unifying their options for CI:</li></ul><pre>export TF_CLI_ARGS=&quot;-input=false -no-color -compact-warnings&quot;<br>terraform plan ...<br>terraform apply ...</pre><p>You can take even more advantage of this when using this variable as the environment configuration of stages or jobs in a CI/CD tool.</p><ul><li>TF_IN_AUTOMATION — when set to any non-empty value (e.g., “true”), Terraform stops suggesting commands run after the one you execute, hence producing less output.</li></ul><h3>Automation Principles</h3><p>Although this article provides a comprehensive overview of possible automation approaches, there is always room for some particular and unusual scenarios. Whichever use case and conditions you have, keep the following goals in mind when using Terraform in CI:</p><ul><li>Ease of code management</li><li>A secure and controlled execution environment</li><li>Coherent runs of init, plan, apply phases</li><li>Leveraging of built-in Terraform capabilities</li></ul><p>Also, there are <a href="https://spacelift.io/">CI/CD automation tools like Spacelift that solve all the issues out of the box</a>. They’re worth looking at as an alternative to a homegrown solution on top of a vanilla CI. Spacelift helps overcome common state management issues and adds several must-have features for infrastructure management. You can check it out for free <a href="https://spacelift.io/free-trial">by going here and creating a trial account</a> or <a href="https://spacelift.io/schedule-demo">booking a demo with one of our engineers</a>.</p><p><strong>Note:</strong> New versions of Terraform are placed under the BUSL license, but everything created before version 1.5.x stays open-source. <a href="https://opentofu.org/">OpenTofu</a> is an open-source version of Terraform that expands on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fedc994db1d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CI/CD Pipeline : Everything You Need To Know]]></title>
            <link>https://medium.com/@navin205/ci-cd-pipeline-everything-you-need-to-know-95bf141fa446?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/95bf141fa446</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 12:50:25 GMT</pubDate>
            <atom:updated>2024-10-16T12:50:25.193Z</atom:updated>
            <content:encoded><![CDATA[<h3>CI/CD Pipeline : Everything You Need To Know</h3><p>In this post we will delve into the basics of CI/CD pipelines, outlining the benefits of using them. We will then look at each stage in the pipeline, and what makes up a good pipeline, along with some examples relating to Terraform.</p><h3>What is a CI/CD Pipeline?</h3><p>A CI/CD pipeline is used to automate software or <a href="https://spacelift.io/blog/infrastructure-as-code">infrastructure-as-code</a> delivery, from source code to production. It can be thought of as a series of steps that needs to be taken for code to be released.</p><p>CI stands for Continuous Integration, and CD stands for Continuous Delivery or Deployment. ‘Pipeline’ refers to the automation of the delivery workflow, consisting of build, test, delivery, and deployment stages. Although each of these steps can be executed manually, automating and orchestrating the stages maximizes the benefits of using CI/CD pipelines by minimizing human error and bringing consistency to each release.</p><p>CI/CD pipelines are commonly used by application delivery teams and operations teams (<a href="https://spacelift.io/blog/who-is-devops-engineer">DevOps</a>). They are the backbone of adopting a DevOps methodology. Read more: <a href="https://spacelift.io/blog/devops-pipeline">Building the DevOps Pipeline</a>.</p><p>CI/CD pipelines are themselves commonly configured in code, sometimes referred to as ‘pipelines-as-code’.</p><p>Typically a build server (or build agent) is used to enable CI/CD runs. Typically these take the form of self-hosted virtual machines in the cloud which can be fully configured but also have to be maintained, or virtual machines provided as part of the platform you are using, which are typically less flexible in terms of adding software and plugins to them.</p><p>Containers can also be used to enable consistent build environments, further removing the reliance on having to maintain a build server. Each step of the CI/CD pipeline can be run in its own container, allowing each step to be run inside a fully customized container. This also enables pipelines to make full use of all the benefits containerization orchestration affords, such as resilience and scaling where required.</p><h3>Continuous Integration (CI)</h3><p>CI covers the build and test stages of the pipeline. Each change in code should trigger an automated build and test, allowing the developer of the code to get quick feedback.</p><h3>Continuous Delivery / Deployment (CD)</h3><p>The CD part of a CI/CD pipeline refers to Delivery and Deployment (CI/CDD anyone?!). CD takes place after the code successfully passes the testing stage of the pipeline. Continuous delivery refers to the automatic release to a repository after the CI stage. Continuous deployment refers to the automatic deployment of the artifact that has been delivered.</p><h3>Stages of a CI/CD Pipeline</h3><p>Starting with writing the source code, and ending up in production, these phases make up the development workflow and form the lifecycle of the CI/CD pipelines. Pipeline runs are usually triggered automatically by a change in the code, but can also be run on a schedule, run manually by a user, or triggered after another pipeline has run.</p><p>The <strong>four parts</strong> of the CI/CD pipeline are:</p><ul><li>Build stage</li><li>Test stage</li><li>Deliver stage</li><li>Deploy stage</li></ul><h3>1) Build</h3><p>The build stage involves the code being written. This is typically done by multiple people in a team, and for larger projects, multiple teams. Code is held in a version control system (VCS), and a Git-based workflow is commonly used to control code being added to the repository (also referred to as GitOps). When used in a pipeline, tools to regulate developer environments and standardize them are particularly useful in order to eliminate any differences between different authors’ code. This will usually take the form of a docker container when using cloud-native software.</p><h3>2) Test</h3><p>As greater resiliency is required and more varied infrastructure is introduced, testing code leads to greater confidence that the code will perform as expected. Testing of your code can be automated. In general, this is normally a repetitive, complex and sometimes tedious process to perform manually. A mistake that a lot of teams make is to skip the test stage or underuse it, as the benefits of properly testing code before delivery and deployment can be huge, contributing to a high-quality product.</p><p>Testing can be broken up into multiple types, using a combination of these with a combination of different tools will give the highest code coverage and result in a higher quality product. Where lots of different tests are used, these can be parallelized to reduce the pipeline run time.</p><p><strong>Smoke testing</strong> — Perform quick sanity checks on the code.</p><p><strong>Integration Testing </strong>— Integration tests are intended to test all the code, or system as a whole. They validate that newly introduced code does not break the existing code.</p><p><strong>Unit Testing </strong>— Unit tests are intended to test a particular function, several functions together, or part of the code. Smaller parts of the infrastructure can be isolated, and tests can be run in parallel to shorten the feedback cycle.</p><p><strong>Compliance Testing </strong>— Compliance testing is used to ensure the configuration follows the policies you’ve defined for the project.</p><p><strong>End-to-end Testing (E2E)</strong> — E2E tests validate everything works together before deploying to production. It is the complete test of the whole process.</p><h3>3) Deliver</h3><p>After the code has been tested, the code is packaged up as an artifact and committed to a repository.</p><h3>4) Deploy</h3><p>The deploy stage allows the orchestration of the artifact release. Usually, teams will deploy to multiple environments, including environments for internal use such as development and staging, and Production for end-user consumption. Using this model, teams can automatically deploy to a staging environment when a pipeline is triggered. Once the staging environment has been reviewed and approved, the code can be merged into the main branch, and then automatically deployed to Production.</p><p>Below you can find a diagram of the CI/CD pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*vApZQh9bIWqBfmbIt7QtXA.png" /></figure><h3>CI/CD Pipeline Benefits</h3><p>Adopting the use of CI/CD pipelines brings many <strong>benefits</strong>, including:</p><ul><li><strong>Reduced costs</strong> — Reducing the time it takes the app or infrastructure to get coded and deployed means less human resources are taken up.</li><li><strong>Reduced time to deployment </strong>— Time to deploy is reduced through automation. The entire process, from coding to deployment, is streamlined, reducing the total process length and making it more efficient. Deployments can be made more often. The rate of release can give companies a significant competitive business advantage over competitors. Deployments can also be rolled back easily, should there be a problem. Developers of the code can remain focused.</li><li><strong>Enabling continuous feedback — </strong>Enables teams to improve their code and workflow based on the feedback received. The testing stage highlights problems or faults and immediately gives feedback to allow code to be improved. Detecting errors earlier in the coding process means that these are easier and quicker to fix. Notifications can be set up to alert team members to failures or successes at each stage.</li><li><strong>Improving team collaboration</strong> — The team has visibility into problems, can view feedback, and make changes where appropriate.</li><li><strong>Audit trails</strong> — Each stage of the pipeline generates logs and can enable accountability and traceability.</li></ul><p>Check out <a href="https://spacelift.io/blog/scaling-ci-cd">7 steps to optimize your CI/CD pipelines for scaling</a>.</p><p>💡 You might also like:</p><ul><li><a href="https://spacelift.io/blog/ci-cd-tools">Top Most Useful CI/CD Tools for DevOps</a></li><li><a href="https://spacelift.io/blog/terraform-in-ci-cd">How to Deploy your Infrastructure in CI/CD using Terraform</a></li><li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac">11 Ways to Improve Your Infrastructure Orchestration</a></li></ul><h3>What Makes a Good CI/CD Pipeline?</h3><p>A good pipeline should be reliable, accurate, and fast.</p><h3>Reliability</h3><p>A pipeline should run reliably each time, without any errors, or unexpected intermittent errors. Broken pipelines can cause frustration and wasted time. When a pipeline is broken or throwing errors, it is recommended that the owner fix it as soon as possible before continuing work on the product. This approach should save other users from the same issues and aid the overall team. It is for this reason that many organizations have a specified ‘DevOps’ team, that will own the pipelines and monitor their success.</p><p>A reliable pipeline will produce the same result, given the same input.</p><p>Self-hosted build agents can often be the cause of unreliable pipeline runs, due to the required maintenance required on these (underlying infrastructure, patching, package management, security, etc). However, they are sometimes necessary as they offer greater flexibility when compared to platform-provided build agents.</p><h3>Accuracy</h3><p>Ironing out errors in repetitive tasks is where CI/CD pipelines can really shine to improve the overall quality of a product.</p><h3>Speed</h3><p>Optimizing a pipeline to run as quickly as possible ensures that the developer of the code can get feedback on the success or failure of the pipeline run quickly, making them more efficient, reducing the chance of distractions and ‘context switching’ (switching to another task/meeting/browsing the web).</p><p>Fast pipeline runs can also enable deployments more often. For example, if a pipeline takes 1 hour to run, and the working day is 8 hours long, then an absolute maximum of 8 deployments can be made per day. Reduce the pipeline run time down to 30 minutes, now 16 deployments can be made.</p><p>A common problem with pipeline runs is that they end up in a queued state, waiting for a build agent to process a previous run before the next can start. For this reason, multiple agents should be provisioned so different pipelines can be run in parallel. A serverless model or <a href="https://spacelift.io/blog/container-orchestration-tools">container orchestration solution</a> is particularly useful here in order to scale the build agent capacity dynamically when demand is high.</p><p>Good pipelines are also quick to create. They should be written in code and kept in a VCS alongside the product code where appropriate. Base templates can be written by providing a common structure, which other templates can reference, speeding up the time required to create a new pipeline.</p><p>See our <a href="https://spacelift.io/blog/ci-cd-tools">list of the most useful CI/CD tools</a> that define the state of today’s software automation.</p><p>And take a look at how <a href="https://spacelift.io/ci-cd-for-infrastructure">Spacelift</a> provides a platform to manage IaC in an automated, easy, controlled, and secure manner. It goes above and beyond the support that is offered by the plain backend system. Spacelift is a sophisticated CI/CD platform for IaC that enables Gitops on infrastructure provisioning workflows. Check it out for free and create a <a href="https://spacelift.io/free-trial">trial account</a> or <a href="https://spacelift.io/schedule-demo">book a demo</a> with one of our engineers.</p><h3>Download The Practitioner’s Guide to Scaling Infrastructure as Code</h3><h3>Example Pipeline Workflow Using Terraform</h3><h3>1) Build</h3><p>Write and store the Terraform Code in VCS.</p><h3>2) Test</h3><p>Testing tools used with Terraform CLI-driven pipeline runs include <a href="https://spacelift.io/blog/what-is-tfsec">Tfsec</a> and <a href="https://spacelift.io/blog/what-is-checkov">Checkov</a>. Many infrastructure-as-code-specific CI/CD platforms enable policy as code such as <a href="https://spacelift.io/blog/what-is-terraform-cloud">Terraform Cloud</a>, or better still, Spacelift, which uses the open-source <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works">Open Policy Agent (OPA)</a>, meaning this testing language could be used across multiple projects using different tooling.</p><p>Learn <a href="https://spacelift.io/blog/terraform-test">how to test your Terraform code</a>.</p><h3>3) Deliver</h3><p>Prepare Terraform Execution Environment (could include things like getting secrets from a key vault, or setting environment variables).</p><p><a href="https://spacelift.io/blog/terraform-init">Terraform init</a></p><p>Terraform plan (copy output file to a repository, ready to be used by the apply command in the deploy stage.</p><h3>4) Deploy</h3><p><a href="https://spacelift.io/blog/terraform-apply">Terraform apply</a></p><p>See also best practices and techniques for managing <a href="https://spacelift.io/blog/kubernetes-ci-cd">CI/CD pipelines with Kubernetes</a>.</p><h3>Example Pipeline Workflow Using Kubernetes</h3><h3>1) Build</h3><p>Developers write Kubernetes configuration files and push them to a Git repository in a VCS. YAML files could be added to define other components of the pipeline, such as ConfigMaps for configuration data or Secrets for sensitive information.</p><h3>2) Test</h3><p>The code is built, and tests are run on it. <em>Kube-bench</em> is a security tool that checks Kubernetes configurations against best practices and industry standards, including CIS benchmarks. <em>Helm unit testing</em> is a testing tool for Helm charts.</p><h3>3) Deliver</h3><p>Once the tests pass, the CI server creates a Docker container image of the application and pushes it to a container registry such as Docker Hub or Amazon ECR.</p><h3>4) Deploy</h3><p>Once the container image is available in the registry, Kubernetes can be used to deploy the application. The docker image can be tagged as appropriate.</p><p>For more details, check out — <a href="https://spacelift.io/blog/kubernetes-ci-cd">Kubernetes CI/CD Pipelines — 7 Best Practices and Tools</a>.</p><h3>Example Pipeline Workflow for a .NET Application</h3><h3>1) Build</h3><p>Developers write .NET code and push it to a Git repository in a VCS. The code is built when it is pushed, triggered by CI using a build tool such as MSBuild or dotnet CLI.</p><h3>2) Test</h3><p>The code is built, and tests are run on it. Unit testing can be done using a framework such as <em>NUnit </em>or <em>xUnit</em>.</p><h3>3) Deliver</h3><p>Once the tests pass, the CI server generates an artifact, such as a NuGet package or a self-contained executable that can be deployed.</p><h3>4) Deploy</h3><p>The artifact is deployed.</p><h3>Key Points</h3><p>CI/CD pipelines are an important concept for the modern engineer to understand fully, especially in the current cloud-native world. CI/CD pipelines enhance the software delivery process by automating key stages such as building, testing, delivery and deployment. Adopting a pipeline-based workflow helps you ship more quickly by passing all code through a consistent set of steps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95bf141fa446" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform with Azure DevOps CI/CD Pipelines — Tutorial]]></title>
            <link>https://medium.com/@navin205/terraform-with-azure-devops-ci-cd-pipelines-tutorial-31e084956576?source=rss-f68b27754c86------2</link>
            <guid isPermaLink="false">https://medium.com/p/31e084956576</guid>
            <dc:creator><![CDATA[Naveen kumar]]></dc:creator>
            <pubDate>Wed, 16 Oct 2024 07:28:58 GMT</pubDate>
            <atom:updated>2024-10-16T12:41:33.093Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform with Azure DevOps CI/CD Pipelines — Tutorial</h3><p>In this article, we will look at how to run Terraform in an Azure DevOps pipeline step by step. We will go from the start of the process, showing how to create an Azure DevOps instance and project, how to set up Terraform in Azure DevOps, and how to create Terraform configuration files for the infrastructure and pipelines using YAML, sharing some examples and best practices along the way.</p><ol><li>Prerequisites</li><li>How to run Terraform in an Azure DevOps pipeline</li><li>Best practices for Terraform in Azure DevOps pipelines</li></ol><h3>Prerequisites</h3><p>Before we get started, make sure you have the following in place:</p><ol><li><strong>Azure Subscription</strong>: To host your resources provisioned by Terraform. If you don’t have one, you can sign up for a free trial.</li><li><strong>Azure DevOps Account</strong>: To create <a href="https://spacelift.io/blog/ci-cd-pipeline">CI/CD pipelines</a>. If you don’t have one, you can <a href="https://azure.microsoft.com/en-us/services/devops/">sign up here</a>.</li><li><strong>Service Principal (SP)</strong>: Setup a Service Principal in Azure Entra ID (Formally known as Azure Active Directory (AAD) with Contributor access to the Azure subscription or Resource Group where you want to deploy resources. Terraform will use the SP to authenticate and interact with Azure. Instructions on how to do that can be found on the Microsoft docs pages <a href="https://learn.microsoft.com/en-us/azure/azure-sql/database/authentication-aad-service-principal-tutorial?view=azuresql#create-a-service-principal-a-microsoft-entra-application-in-microsoft-entra-id">here</a>.</li></ol><h3>How to run Terraform in an Azure DevOps pipeline</h3><p>Follow these steps to run Terraform in an Azure DevOps pipeline:</p><ol><li>Set up an Azure DevOps project</li><li>Create Terraform configuration files</li><li>Define CI/CD steps in YAML</li><li>Configure the Azure DevOps pipeline</li><li>Run the pipeline</li></ol><h3>1. Set up an Azure DevOps project</h3><p>With your account set up, you next need to sign into <a href="https://spacelift.io/blog/what-is-azure-devops">Azure DevOps</a> and create a project where you’ll be setting up your CI/CD pipelines.</p><ol><li>Go to the <a href="https://dev.azure.com/">Azure DevOps website</a> and sign in with your Microsoft account or organizational account.</li><li>Navigate to Organization: If you’re part of multiple organizations, select the organization where you want to create the project from the top right corner of the page.</li><li>Click on the “New Project” button.</li><li>In the “Create a new project” form, fill in the Project Name and visibility (whether the project should be public or private) and optionally, provide a description for your project, then click on the “Create” button.</li><li>Once the project is created, you’ll be redirected to the project’s dashboard. Depending on your requirements, you may want to customize your project further by adding team members, configuring permissions, setting up repositories, boards, and pipelines, and integrating with other services.</li></ol><h3>2. Create Terraform configuration files</h3><p>Create your Terraform configuration files and store them in a repository. This repo could be in your Azure DevOps project, or hosted elsewhere. In this example, we define a configuration file to provision a resource group containing a VNET and subnet.</p><p><em>network.tf</em></p><pre>provider &quot;azurerm&quot; {<br>  features {}<br>}</pre><pre>resource &quot;azurerm_resource_group&quot; &quot;example&quot; {<br>  name     = &quot;example-resources&quot;<br>  location = &quot;UK South&quot;<br>}</pre><pre>resource &quot;azurerm_virtual_network&quot; &quot;example&quot; {<br>  name                = &quot;example-vnet&quot;<br>  resource_group_name = azurerm_resource_group.example.name<br>  location            = azurerm_resource_group.example.location<br>  address_space       = [&quot;172.16.0.0/16&quot;]<br>}</pre><pre>resource &quot;azurerm_subnet&quot; &quot;example&quot; {<br>  name                 = &quot;example-subnet&quot;<br>  resource_group_name  = azurerm_resource_group.example.name<br>  virtual_network_name = azurerm_virtual_network.example.name<br>  address_prefixes     = [&quot;172.16.1.0/24&quot;]<br>}</pre><h3>3. Define CI/CD steps in YAML</h3><p>As a best practice, your pipelines should be created in code. Create the pipeline in code in <a href="https://spacelift.io/blog/yaml">YAML format</a> and store it in the same repository:</p><p><em>azure-pipelines.yml</em></p><pre>trigger:<br>- main</pre><pre>pool:<br>  vmImage: &#39;ubuntu-latest&#39;</pre><pre>stages:<br>- stage: Plan<br>  displayName: &#39;Terraform Plan&#39;<br>  jobs:<br>  - job: TerraformPlan<br>    displayName: &#39;Terraform Plan&#39;<br>    steps:<br>    - task: TerraformInstaller@0<br>      inputs:<br>        terraformVersion: &#39;latest&#39;</pre><pre>    - script: |<br>        terraform --version<br>        terraform init -backend-config=&quot;storage_account_name=$(storageAccountName)&quot; -backend-config=&quot;container_name=$(containerName)&quot; -backend-config=&quot;key=$(key)&quot; -backend-config=&quot;access_key=$(accessKey)&quot;<br>        terraform plan -out=tfplan<br>      displayName: &#39;Terraform Init and Plan&#39;</pre><pre>- stage: Deploy<br>  displayName: &#39;Terraform Apply&#39;<br>  jobs:<br>  - job: TerraformApply<br>    displayName: &#39;Terraform Apply&#39;<br>    steps:<br>    - task: TerraformInstaller@0<br>      inputs:<br>        terraformVersion: &#39;latest&#39;</pre><pre>    - script: |<br>        terraform --version<br>        terraform init -backend-config=&quot;storage_account_name=$(storageAccountName)&quot; -backend-config=&quot;container_name=$(containerName)&quot; -backend-config=&quot;key=$(key)&quot; -backend-config=&quot;access_key=$(accessKey)&quot;<br>        terraform apply -auto-approve tfplan<br>      displayName: &#39;Terraform Init and Apply&#39;</pre><ul><li>The pipeline is triggered whenever changes are pushed to the main branch.</li><li>The pipeline runs on an Ubuntu agent (ubuntu-latest).</li><li><strong>TerraformInstaller:</strong> Installs the latest version of Terraform on the build agent.</li><li><strong>Script:</strong> Executes <a href="https://spacelift.io/blog/terraform-commands-cheat-sheet">Terraform commands</a> to initialize the Terraform working directory, generate a plan, and apply changes to Azure. It initializes Terraform with a backend configuration to store the state in an Azure Storage Account (replace &lt;your-azure-service-connection&gt; with your actual Azure service connection details).</li></ul><p><strong>TerraformTaskV1:</strong> Uses the Terraform task to apply changes to the infrastructure. It first specifies the plan command in the first stage, and then an apply stage, utilizing the Azure service connection for authentication (environmentServiceNameAzureRM and backendServiceArm), and the working directory where the Terraform files are located. The commandOptions: &#39;-auto-approve&#39; automatically approves the changes without prompting for confirmation.</p><h3>4. Configure the Azure DevOps pipeline</h3><p>To use the pipeline:</p><ol><li>In Azure DevOps, go to “Pipelines” &gt; “Pipelines” and click on “New Pipeline”.</li><li>Point your pipeline to the Git repository containing the azure-pipelines.yml file.</li><li>Azure DevOps will automatically detect the YAML configuration file. Review and confirm the configuration, then save.</li></ol><h3>5. Run the pipeline</h3><p>To run your new pipeline, first, navigate to Pipelines in the left sidebar. Choose the pipeline that you want to run from the list of available pipelines. If you have multiple pipelines, you’ll see them listed here.</p><p>Once you’ve selected the pipeline, you’ll see a “Run pipeline” button at the top right corner. Click on this button to initiate a manual run of the pipeline. If your pipeline is set up to trigger on specific branches, you may be prompted to select the branch you want to run the pipeline for. If not, you can skip this step.</p><p>Review the configuration and settings for the pipeline run, and then click on the “Run” button to confirm and start the pipeline execution.</p><p>Once the pipeline run is started, you’ll be taken to the pipeline run page, where you can monitor the progress of the pipeline execution. You’ll see the various stages and tasks being executed in real-time. As the pipeline runs, you can view detailed logs for each task to see the output and any error messages. You can also access any artifacts generated by the pipeline, such as build outputs or deployment packages.</p><p>After the pipeline run completes, you can review the results to see if the pipeline executed successfully or if there were any failures. You can view the overall status, individual task status, and any associated warnings or errors.</p><h3>Best practices for Terraform in Azure DevOps pipelines</h3><p>Here are 10 best practices you’ll want to think about when writing your Terraform pipelines in Azure DevOps:</p><ol><li><strong>Define your Azure DevOps pipelines using YAML as code</strong> (e.g., azure-pipelines.yml) alongside your Terraform configurations. This practice enables versioning, review, and audit of pipeline changes. Store your Terraform configurations in version control (e.g., Git) to track changes, collaborate with team members, and maintain a history of infrastructure modifications.</li><li>C<strong>onfigure pipeline triggers to automatically execute pipelines upon code changes</strong> (e.g., commits to specific branches or pull requests, or on an overnight schedule to run a plan). This ensures timely validation and deployment of infrastructure changes.</li><li><strong>Optimize pipeline performance by parallelizing tasks whenever possible</strong>. For example, run Terraform plan and apply stages concurrently for different environments or resource groups.</li><li><strong>Maintain separate pipelines and environments</strong> (e.g., Dev, QA, Prod) to isolate infrastructure changes and avoid unintended impacts across environments. These can be set up in the Environments section of Azure DevOps.</li><li><strong>Integrate automated testing into your pipelines for Terraform configurations</strong> using tools like <a href="https://spacelift.io/blog/what-is-terratest">Terratest</a> to validate infrastructure changes before deployment.</li><li><strong>Implement robust error handling and logging mechanisms within pipelines</strong> to capture and report errors effectively. Utilize Azure DevOps pipeline features like logging commands and error-handling tasks.</li><li><strong>Include validation steps in your pipelines</strong> to verify Terraform configurations against best practices, policies, and compliance requirements using tools like <a href="https://spacelift.io/blog/terraform-validate">terraform validate</a> to confirm your syntax is correct, <a href="https://spacelift.io/blog/terraform-fmt">terraform fmt</a> to validate your formatting or call static analysis tools.</li><li><strong>Monitor pipeline execution, performance metrics, and resource usage</strong> to identify bottlenecks, optimize workflows, and ensure the reliability of infrastructure deployments.</li><li><strong>Apply </strong><a href="https://spacelift.io/blog/ci-cd-security"><strong>security best practices to your pipelines</strong></a>, including access control, role-based permissions, and regular security reviews to mitigate risks associated with pipeline configurations and execution.</li><li><strong>Document pipeline configurations, deployment processes, and infrastructure designs</strong> to facilitate collaboration, knowledge sharing, and onboarding of team members.</li></ol><h3>Key points</h3><p>Terraform tasks in the Terraform workflow including deployments can be automated with Azure DevOps CI/CD Pipelines. Azure DevOps is a good choice to seamlessly integrate with Azure and integrate with the Azure ecosystem in particular. Using pipelines means you can repeat the workflow tasks defined in your pipeline in a repeatable and controlled manner, giving you consistency and removing manual effort.</p><p>You should also consider the using a Continuous Integration/Continuous Delivery (CI/CD) platform. The most flexible platforms, such as Spacelift, support multiple tools, including Terraform, OpenTofu Ansible, Pulumi, and AWS CloudFormation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31e084956576" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>