I was running a routine mkdocs serve on a project I have been maintaining for a while. MkDocs 1.6, Material theme, a handful of plugins, nothing fancy. Then this showed up:

│ ⚠ Warning from the Material for MkDocs team           │
│                                                         │
│ MkDocs 2.0, the underlying framework of Material for   │
│ MkDocs, will introduce backward-incompatible changes,  │
│ including:                                              │
│                                                         │
│ × All plugins will stop working                        │
│ × All theme overrides will break                       │
│ × No migration path exists                             │
│ × Closed contribution model                            │
│ × Currently unlicensed                                 │

I read it twice. Then I went and read the full analysis the Material for MkDocs team published.

That was the beginning of a longer journey than I expected.

What Is Actually Happening with MkDocs 2.0

Before I get into my migration story, let me give you the full picture because it is a bit of a mess and easy to get confused about.

MkDocs and Material for MkDocs are two separate projects. MkDocs is the underlying framework. Material for MkDocs is the theme and plugin layer that sits on top of it, maintained by a completely different team. Most people use both together, so they often get lumped in as one thing. They are not.

MkDocs 1.x has been effectively unmaintained for well over 18 months now. Issues and pull requests are piling up. Long-standing bugs — including live-reload problems that I personally ran into — have gone without fixes. Nobody is certain whether security issues will even be addressed going forward.

Then came MkDocs 2.0 — a ground-up rewrite with no backward compatibility. The plugin system was removed entirely. The theming system was rewritten from scratch. Your existing mkdocs.yml will not work with it. There is no migration guide. The project also shifted to a closed contribution model, meaning community members cannot even report bugs. And at the time of writing, it ships without a proper open source license, which makes it unsuitable for production use in most organizations.

The Material for MkDocs team dug into the pre-release thoroughly and published their findings. Their conclusion was clear — MkDocs 2.0 is not a drop-in replacement for anything. There is no path forward from 1.x to 2.0 without rebuilding your documentation setup from scratch.

So you have a situation where MkDocs 1.x is dead in everything but name, and MkDocs 2.0 breaks everything that made MkDocs 1.x worth using in the first place. Florian Maas documented the entire timeline well in The Slow Collapse of MkDocs — worth a read if you want the full backstory.

That is the context.

My Setup Before the Migration

I was running MkDocs 1.6 with the Material theme on a documentation project that had grown over a couple of years. Here is roughly what my mkdocs.yml looked like:

site_name: My Project Docs
theme:
  name: material
  palette:
    primary: indigo
  features:
    - navigation.tabs
    - navigation.sections
    - content.code.copy

plugins:
  - search
  - tags
  - minify:
      minify_html: true

markdown_extensions:
  - admonition
  - pymdownx.highlight
  - pymdownx.superfences
  - pymdownx.tabbed:
      alternate_style: true
  - attr_list
  - def_list

It worked. Mostly. But live-reload broke on me regularly. The search plugin occasionally produced stale results during development. And I had one plugin conflict between tags and a third-party extension that I never fully resolved — I just learned to live with it.

I am the kind of person who likes tools that stay out of the way. I wrote about this when covering must-have Terraform tools — the best tools are the ones you barely notice because they just work. MkDocs 1.6 had stopped feeling that way.

When the MkDocs 2.0 warning appeared, I knew it was time to stop patching around issues and actually make a decision.

Finding Zensical

The Material for MkDocs team has been building Zensical as their path forward. It is not just a theme — it is a complete documentation toolchain built from scratch after nearly a decade of lessons from maintaining Material for MkDocs. Martin Donath, the creator of Material for MkDocs, spoke about the backstory of Zensical on Talk Python To Me #542 — that conversation gives a lot of context on why a clean break was necessary rather than patching on top of what existed.

So I decided to try it.

First Attempt — Python 3.9.6

I went to install Zensical on my Mac.

pip install zensical

Error.

ERROR: Could not find a version that satisfies the requirement zensical
ERROR: No matching distribution found for zensical

My Mac was still on Python 3.9.6, which is what ships by default on older macOS setups. I had never updated it because everything I needed was working fine with it. This kind of Python version mismatch is something I have run into before in other contexts — I actually documented a similar situation with Ansible in my article on No python interpreters found for host, where the fix also came down to getting the right Python version in place.

Zensical requires Python 3.10 or higher. My setup was below the minimum and the install simply refused to continue. No helpful error pointing to the version requirement either — just a blunt “no matching distribution” message that sent me looking in the wrong direction for a while.

Fixing Python First

I manage Python versions with pyenv, which I strongly recommend if you are doing any serious Python work on a Mac. It keeps your system Python untouched and lets you run different versions per project. If you are not already using it, installing it is straightforward with Homebrew — and if you are still getting your macOS terminal setup sorted, my recent article on The Ultimate Terminal Stack in 2026 covers the full environment setup that makes this kind of tooling much easier to manage.

pyenv install 3.11.9
pyenv global 3.11.9
python --version
# Python 3.11.9

If you do not have pyenv yet:

brew install pyenv

Then add this to your .zshrc or .bash_profile:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Restart your terminal, install 3.11.9, set it as global, and you are good to go.

Installing Zensical — Second Try

With Python 3.11.9 in place, I created a fresh virtual environment for the project and tried again.

python -m venv zensical-env
source zensical-env/bin/activate
pip install zensical

Clean install. No errors this time.

The Migration from MkDocs 1.6

This is the part I want to be honest about. The migration is not a one-command operation. But it is also not as painful as I feared.

Step 1 — Initialize a new Zensical project

zensical init my-docs
cd my-docs

Step 2 — Move your Markdown files

Your existing docs/ folder structure carries over almost entirely. Zensical reads Markdown the same way. I copied my files over and the content was intact.

cp -r old-project/docs/* my-docs/docs/

Step 3 — Rewrite the config

This is the real work. Zensical uses TOML instead of YAML. Here is what my config looked like after the migration:

[site]
name = "My Project Docs"
description = "Documentation for my project"
[theme]
style = "material"
palette = "indigo"
[features]
navigation_tabs = true
navigation_sections = true
code_copy = true
[nav]
auto = true
[search]
enabled = true

Some things I had in MkDocs did not have a direct equivalent yet. The tags plugin, for instance, required checking the Zensical docs to see how tagging works in their model. The minify plugin is not needed — Zensical handles output optimization internally.

The pymdownx Markdown extensions largely work without explicit configuration because Zensical bundles sensible defaults. Admonitions, code highlighting, tabbed content — these all rendered correctly without me listing them one by one in the config.

Step 4 — Verify and fix

zensical serve

I went through every page in the browser and checked rendering. Most things looked right immediately. A few pages that used very specific Material-specific HTML overrides needed small fixes, but nothing that took more than a few minutes per page.

What Changed for the Better

Live-reload works. I mean actually works. I had gotten so used to stopping and restarting mkdocs serve multiple times during a writing session that I forgot live-reload was supposed to just work. With Zensical, I make a change, save, and the browser updates. Every time.

Config is readable. My old mkdocs.yml had grown into something I needed to scroll through carefully before making any change. The Zensical TOML config is short and obvious. I can read the whole thing in thirty seconds.

Auto navigation. Zensical builds the nav tree from your folder structure. No more maintaining a hand-written nav definition every time I add a new page.

No plugin conflicts. I have not run into a single compatibility issue. The things I need are either built in or have a clean extension path.

What I Had to Give Up (For Now)

Honesty matters here. The Zensical plugin ecosystem is still young. If you are heavily using specific MkDocs plugins, check whether Zensical has a replacement before you commit to migrating. The community and third-party extension library that MkDocs built over years does not exist yet for Zensical.

Also, if your team has a lot of custom theme overrides — custom CSS hooks, template blocks, JavaScript injections — those will need to be rethought. Zensical’s theming model is different enough that you cannot just copy your overrides across.

Should You Migrate?

If you are starting fresh, start with Zensical. There is no reason to build on MkDocs 1.x when it is no longer maintained, and MkDocs 2.0 is not a safe bet for production documentation.

If you are on an existing MkDocs 1.6 project, start planning the migration now rather than waiting for something to break. The Material for MkDocs team has committed to security patches for at least 12 months, so you have a window. Use it to migrate on your own terms rather than scrambling when something stops working.

And if you are still on Python 3.9 like I was — fix that first. Everything else can wait five minutes while you sort out your Python version.

From MkDocs 1.6 to Zensical — Here’s Why I Finally Made the Move was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

I love exploring new tools and publish about these awesome tools. Like my recent article on Developer tools for 2026. For years, my terminal setup was… adequate. Like many of you, I clung to iTerm2, sprinkled a few plugins on top, and called it a day. It worked, mostly. But as my coding demands grew, so did my frustration with the subtle lags, the lost command history, and the endless re-setting up of my workspace after a reboot.

The world of terminal tools has evolved rapidly. New projects written in languages like Zig and Rust are changing what’s possible. I decided it was time for a complete overhaul. My goal: a terminal that was lightning-fast, remembered everything, and helped me navigate my projects with zero friction.

Here’s how I built what I consider the ultimate macOS terminal environment for 2026.

The Foundation: Ghostty — The Speed Demon

The first step was ditching my old terminal emulator. Ghostty is the star here. It’s written in Zig and taps directly into Apple’s Metal API, making it incredibly performant. This isn’t just marketing hype; it means a buttery-smooth experience. Scrolling through huge log files is instant, and every character you type appears on screen without a hint of delay.

It also comes with native macOS features like tabs and split panes, and its configuration is a simple, human-readable file at ~/.config/ghostty/config. This makes it easy to back up and version control your settings.

The Aesthetic: Anka/Coder — My New Favorite Font

Font choice might seem minor, but it’s crucial when you stare at code all day. I moved to Anka/Coder. This font stands out because it’s a condensed monospace font. It allows you to see more code on your screen horizontally without sacrificing readability. It has a clean, almost retro-tech feel that perfectly complements a high-performance setup.

The Brain: Atuin & Zoxide — Never Forget, Always Find

This is where true productivity gains happen. Atuin completely replaces your shell’s default history. Instead of a basic ~/.zsh_history file, Atuin stores every command you've ever run in a searchable SQLite database. Hit your Up-Arrow key, and you get a beautiful, full-screen fuzzy search interface. You can filter by directory, exit code, or even sync it across multiple machines. That complex Docker command from last month? Found in seconds.

Then there’s Zoxide, a smarter cd command written in Rust. It learns your habits. If you frequently go to ~/Documents/Dev/projectX/backend/src, you can just type z back and Zoxide will take you there. It's like your terminal intuitively knows where you want to go.

The Safety Net: Tmux — Immortality for Your Sessions

Have you ever had your terminal crash, or had to reboot your Mac, only to lose your entire workspace layout? All your open tabs, split panes, and running processes gone? Tmux (Terminal Multiplexer) solves this.

Tmux lets you create persistent sessions that run in the background. Even if your terminal window closes, your Tmux session (and everything inside it) keeps running. With plugins like tmux-resurrect and tmux-continuum, your entire Tmux environment—including all your panes, windows, and their contents—is automatically saved and restored when you launch Tmux again, even after a system reboot. It's like a save game for your terminal.

The “One-Click” Setup: Your Complete Automation Script

I believe in automation. Setting up a new development environment should be as simple as running one script. This script will:

• Install Homebrew (if you don’t have it).
• Install Ghostty, Anka/Coder font, Tmux, Atuin, Zoxide, Starship, and other essential tools.
• Configure your Zsh shell with autosuggestions and syntax highlighting.
• Set up Ghostty with the correct font and persistence settings.
• Configure Tmux with mouse support, intuitive split shortcuts, and session persistence plugins (TPM, tmux-resurrect, tmux-continuum).

Just copy this entire block, save it as setup.sh on your Mac, and run bash setup.sh in your current terminal. Remember to chmod to give appropriate permissions.

#!/bin/bash

echo "🚀 Starting the Ultimate macOS Terminal Setup Script..."

# --- 1. Install Homebrew if not present ---
if ! command -v brew &> /dev/null; then
    echo "🍺 Homebrew not found. Installing Homebrew..."
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    eval "$(/opt/homebrew/bin/brew shellenv)"
else
    echo "✅ Homebrew already installed."
fi

# --- 2. Install Core Tools & Anka/Coder Font ---
echo "📦 Installing Ghostty, Tmux, Atuin, Starship, Zoxide, fzf, eza, bat..."
echo "Also installing the Anka/Coder font (via Homebrew Cask)."

brew install --cask ghostty
brew install --cask font-anka-coder # The "Anker Coder" font

brew install tmux atuin zoxide fzf eza bat starship

# --- 3. Setup Zsh Plugins (Autosuggestions & Syntax Highlighting) ---
echo "🔌 Setting up Zsh plugins: autosuggestions and syntax highlighting..."
mkdir -p ~/.zsh

# Zsh-autosuggestions
if [ ! -d "$HOME/.zsh/zsh-autosuggestions" ]; then
    git clone https://github.com/zsh-users/zsh-autosuggestions ~/.zsh/zsh-autosuggestions
else
    echo "   zsh-autosuggestions already cloned. Updating..."
    (cd ~/.zsh/zsh-autosuggestions && git pull)
fi

# Zsh-syntax-highlighting
if [ ! -d "$HOME/.zsh/zsh-syntax-highlighting" ]; then
    git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ~/.zsh/zsh-syntax-highlighting
else
    echo "   zsh-syntax-highlighting already cloned. Updating..."
    (cd ~/.zsh/zsh-syntax-highlighting && git pull)
fi

# --- 4. Write ~/.zshrc (Shell Configuration) ---
echo "📝 Creating/Updating ~/.zshrc configuration file..."
cat << 'EOF' > ~/.zshrc
# --- Environment Variables ---
export LANG=en_US.UTF-8
export EDITOR='nvim' # Or 'nano', 'code --wait', etc.

# --- Zsh Plugin Loading ---
source ~/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh
source ~/.zsh/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

# --- Tool Initializations ---
eval "$(starship init zsh)" # Minimalist, fast prompt
eval "$(zoxide init zsh)"   # Smart 'cd' command
eval "$(atuin init zsh)"    # Searchable shell history

# --- FZF Keybindings & Completion ---
# This line sources fzf's keybindings and completion setup
[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh

# --- Modern CLI Aliases ---
# Replace old commands with their modern, feature-rich alternatives
alias ls='eza --icons --group-directories-first' # 'ls' with icons and better grouping
alias ll='eza -lh --icons --grid'                # 'ls -l' with icons and grid view
alias cat='bat'                                  # 'cat' with syntax highlighting and Git integration
alias cd='z'                                     # Make 'cd' use zoxide's smart jumping
alias g='git'                                    # Shorthand for Git

# --- Custom Key Bindings ---
# Use Up-Arrow for Atuin's full-screen history search
bindkey '^[[A' atuin-up-search
EOF

# --- 5. Write Ghostty Configuration ---
echo "👻 Creating ~/.config/ghostty/config file..."
mkdir -p ~/.config/ghostty
cat << EOF > ~/.config/ghostty/config
# Theme and Appearance
theme = dark-plus
font-family = "Anka/Coder" # Using the installed Anka/Coder font
font-size = 14
background-opacity = 0.9
background-blur = true

# macOS Integration
macos-titlebar-style = hidden
window-save-state = "always" # Ensure sessions persist across Ghostty restarts
EOF

# --- 6. Write Tmux Configuration & Install TPM (Tmux Plugin Manager) ---
echo "🪟 Creating ~/.tmux.conf file and installing Tmux Plugin Manager (TPM)..."
mkdir -p ~/.tmux/plugins

# Clone TPM if it doesn't exist
if [ ! -d "$HOME/.tmux/plugins/tpm" ]; then
    git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm
else
    echo "   TPM already cloned. Updating..."
    (cd ~/.tmux/plugins/tpm && git pull)
fi

cat << 'EOF' > ~/.tmux.conf
# --- General Tmux Settings ---
set -g default-terminal "screen-256color" # Enable 256 colors
set -g mouse on                           # Enable mouse support for resizing panes, scrolling
set -s escape-time 0                      # Reduce escape key delay (important for Vim users)
set -g history-limit 10000                # Increase scrollback buffer

# --- Custom Key Bindings for Panes ---
# Split panes horizontally and vertically using intuitive | and - keys
bind | split-window -h -c "#{pane_current_path}" # Split vertical
bind - split-window -v -c "#{pane_current_path}" # Split horizontal
unbind '"'                                       # Unbind default split keys
unbind %

# --- Pane Navigation with Alt + Arrow Keys ---
# This allows quick pane switching without the Tmux prefix key (Ctrl+b)
bind -n M-Left select-pane -L
bind -n M-Right select-pane -R
bind -n M-Up select-pane -U
bind -n M-Down select-pane -D

# --- Status Bar Styling (Optional: For a cleaner look) ---
set -g status-style bg=default
set -g status-left-length 40
set -g status-left "#[fg=green]Session: #S #[fg=yellow]• " # Session name
set -g status-right "#[fg=cyan]%H:%M #[fg=magenta]%d-%b-%y" # Time and date

# --- Tmux Plugin Manager (TPM) & Plugins ---
# List of plugins to be installed by TPM
set -g @plugin 'tmux-plugins/tpm'           # Tmux Plugin Manager itself
set -g @plugin 'tmux-plugins/tmux-resurrect' # Save and restore Tmux environment
set -g @plugin 'tmux-plugins/tmux-continuum' # Continuously save Tmux environment

# Configure tmux-continuum to automatically restore sessions on Tmux start
set -g @continuum-restore 'on'

# Initialize TPM. This line MUST be at the very end of .tmux.conf
run '~/.tmux/plugins/tpm/tpm'
EOF

# --- Automated Tmux Plugin Installation ---
echo "⚙️  Automatically installing Tmux plugins (no manual keypress needed)..."
~/.tmux/plugins/tpm/bin/install_plugins

echo "----------------------------------------------------"
echo "✅ SETUP COMPLETE!"
echo "----------------------------------------------------"
echo "Instructions for a perfect experience:"
echo "1. Open 'Ghostty' from your Applications folder. (It should pick up the config automatically)."
echo "2. In Ghostty, type 'source ~/.zshrc' and press Enter to load your new Zsh config."
echo "3. Type 'tmux' and press Enter to start your persistent Tmux session."
echo "   Your modern terminal is now fully operational!"
echo "   (Press Up-Arrow to try Atuin's history search, 'z' for smart CD, etc.)"

The Cross-Platform Reality Check

Before we dive into the installation scripts, we need to talk about compatibility. The dream is to install the exact same tools everywhere. The reality is that the backend tools are universal, but the frontends require OS-specific choices.

What works everywhere (macOS, Linux, Windows via WSL2):

• Zsh: Our shell of choice.
• Tmux: For session persistence and window management.
• Atuin: The SQLite database that replaces your shell history and syncs it across all your machines.
• Zoxide: The smart replacement for the cd command.
• Starship: The minimalist, fast shell prompt.
• Modern CLI Tools: eza (replaces ls), bat (replaces cat), and fzf (fuzzy finder).
• Anka/Coder: The condensed monospace font for high-density coding.

The Exception: The Terminal Emulator

• Ghostty is the undisputed king of performance on macOS and Linux due to its native Zig and GPU-accelerated architecture. However, Ghostty is not built natively for Windows.
• Windows Terminal is the mandatory alternative for Windows users. When paired with the Windows Subsystem for Linux (WSL2), it provides a near-identical experience to Ghostty on Unix systems.

Operating System Specific Frontends

Let us get the emulators and core package managers set up based on your current machine.

macOS (The Ghostty Native Setup)

Apple environments are the easiest to bootstrap thanks to Homebrew. Open your default Terminal.app and run:

# Install the core tools and Ghostty
brew install --cask ghostty font-anka-coder
brew install tmux atuin zoxide fzf eza bat starship

Linux (The Hybrid Setup)

On Linux, you want to use your native package manager (like apt or pacman) for the heavy system tools, but use Homebrew for Linux to get the latest versions of the Rust-based utilities.

Open your terminal and run:

# Get the base system tools
sudo apt update && sudo apt install zsh curl git gcc
# Install Homebrew for Linux
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
# Install the modern stack
brew install tmux atuin zoxide fzf eza bat starship font-anka-coder

(Note: Ghostty is available for Linux, but installation varies heavily depending on whether you use Wayland or X11. Check the official Ghostty repository for the pre-compiled binaries for your specific distribution).

Windows (The WSL2 Workaround)

To get this stack running on Windows, we bypass the native Windows command prompt entirely. We use Windows Terminal as the visual wrapper and Ubuntu (via WSL2) as the actual engine.

Open PowerShell as Administrator:

# Install Windows Terminal
winget install Microsoft.WindowsTerminal

# Install the Windows Subsystem for Linux
wsl --install -d Ubuntu

Restart your computer. Open Windows Terminal, select the Ubuntu profile, and then follow the Linux instructions above to install Zsh, Homebrew, and the modern CLI tools inside your WSL environment. You will also need to download the Anka/Coder font from the web and install it through the Windows Font Settings menu to make it available to Windows Terminal.

The Universal Brain Configuration

Now that the frontends are installed, we unify the backend. This configuration is identical whether you are on Mac, Linux, or Windows (WSL).

Configure Your Shell

Clone the necessary Zsh plugins to give yourself IDE-like autocompletion:

mkdir -p ~/.zsh
git clone https://github.com/zsh-users/zsh-autosuggestions ~/.zsh/zsh-autosuggestions
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ~/.zsh/zsh-syntax-highlighting

Next, open your ~/.zshrc file and paste the logic that binds all our tools together:

# Load plugins
source ~/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh
source ~/.zsh/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
# Initialize the modern tools
eval "$(starship init zsh)"
eval "$(zoxide init zsh)"
eval "$(atuin init zsh)"
# Setup modern aliases
alias ls='eza --icons --group-directories-first'
alias ll='eza -lh --icons --grid'
alias cat='bat'
alias cd='z'
# Make Atuin launch on Up-Arrow
bindkey '^[[A' atuin-up-search

Configure Tmux for Immortality

To ensure your sessions survive unexpected reboots, we configure Tmux. First, install the Tmux Plugin Manager:

git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm

Then, create a ~/.tmux.conf file and add this text:

set -g default-terminal "screen-256color"
set -g mouse on
set -s escape-time 0

# Split panes logically
bind | split-window -h -c "#{pane_current_path}"
bind - split-window -v -c "#{pane_current_path}"
# Enable persistence plugins
set -g @plugin 'tmux-plugins/tpm'
set -g @plugin 'tmux-plugins/tmux-resurrect'
set -g @plugin 'tmux-plugins/tmux-continuum'
set -g @continuum-restore 'on'
run '~/.tmux/plugins/tpm/tpm'

Launch Tmux by typing tmux, then press Ctrl+b followed by I (capital i) to install the plugins.

The Payoff

If you followed these steps on your various machines, you now have a superpower.

When you sit down at your MacBook, Ghostty launches instantly. When you move to your Windows desktop, Windows Terminal fires up WSL2. On both machines, typing z project takes you to the exact same relative folder. Pressing the up arrow brings up your Atuin database, allowing you to pull a command you ran on your Linux server straight into your Windows environment.

By accepting that frontends must be OS-specific while standardizing your backend tools, you eliminate platform friction entirely.

Conclusion

This isn’t just a collection of tools; it’s an ecosystem designed for peak developer performance on macOS. By embracing modern, high-performance utilities, you’ll spend less time battling your terminal and more time focusing on what truly matters: building great software. Give it a try — your future, more productive self will thank you.

The Ultimate Terminal Stack in 2026: A Cross-Platform Guide for macOS, Linux, and Windows was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

I love Terraform and as a curious explorer, I started looking for an alternative when Terraform introduced HCL2. An IaC alternative where I can use the programming language of my choice for declarative approach.

Infrastructure as Code (IaC) has revolutionized how organizations manage and provision cloud resources. By defining infrastructure through code rather than manual processes, teams can achieve consistency, version control, and automated deployment workflows. However, working with IaC tools can present unique challenges, particularly when integrating with cloud services like IBM Cloud Secrets Manager.

While creating a Pulumi project to provision Secrets Manager group instance, I came across an error

CreateSecretGroupWithContext failed: Post "https://0a21f2b7-e695-42e4-873f.us-south.secrets-manager.appdomain.cloud/api/v2/secret_groups": dial tcp: lookup 0a21f2b7-e695-42e4-873f.us-south.secrets-manager.appdomain.cloud: no such host

halting the infrastructure provisioning entirely. This article provides a comprehensive guide to understanding, diagnosing, and resolving these DNS resolution errors while working with IBM Cloud Secrets Manager through both Terraform and Pulumi.

Understanding Infrastructure as Code (IaC)

Infrastructure as Code allows developers and operations teams to manage infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools. This approach brings several benefits including repeatability, consistency, and the ability to version control infrastructure changes.

Terraform vs Pulumi: A Brief Overview

While Terraform uses HashiCorp Configuration Language (HCL) and has become the industry standard for IaC, Pulumi offers a modern alternative that allows infrastructure definition using familiar programming languages like Python, TypeScript, Go, .NET, and Java. Both tools support IBM Cloud through dedicated providers, but they differ significantly in their approach and capabilities.

Key differences include:

• Language Support: Terraform uses HCL exclusively, while Pulumi supports multiple general-purpose programming languages
• State Management: Terraform uses local or remote state files, while Pulumi uses its SaaS backend by default
• Testing Capabilities: Pulumi provides native support for unit and integration testing through programming language ecosystems

Introduction to Pulumi

What is Pulumi?

Pulumi is a modern Infrastructure as Code platform that leverages existing programming languages and their native ecosystems to interact with cloud resources through the Pulumi SDK. Unlike traditional IaC tools that require learning domain-specific languages, Pulumi allows developers to use languages they already know and love.

Key Advantages of Pulumi over Traditional IaC Tools

Enhanced Developer Experience: Pulumi provides full IDE support with code completion, error checking, and refactoring capabilities that come naturally with general-purpose programming languages.

Advanced Testing Capabilities: Since Pulumi uses real programming languages, teams can leverage existing testing frameworks and methodologies to validate their infrastructure code.

Stronger Abstraction and Modularity: Programming language features like functions, classes, and packages enable more sophisticated infrastructure abstractions than simple configuration languages allow.

Native Integration with Application Code: Teams can embed infrastructure management directly in their applications or use the same languages for both application and infrastructure development.

Pulumi’s Multi-Language Support

Pulumi supports a wide range of programming languages:

• TypeScript/JavaScript: Native Node.js support with full npm ecosystem
• Python: Full Python 3.x support with pip package management
• Go: Native Go modules and tooling
• .NET: Support for C#, F#, and VB.NET
• Java: Full JVM ecosystem support
• YAML: For simpler, declarative scenarios

Setting Up IBM Cloud with Pulumi

Installing the IBM Cloud Provider for Pulumi

To work with IBM Cloud resources in Pulumi, you need to install the IBM Cloud provider. The IBM Cloud provider for Pulumi must be installed as a local package:

pulumi package add terraform-provider ibm-cloud/ibm

Understanding the Service Endpoint Configuration Error

Though I am exploring Pulumi, the IBM Pulumi provider is based on Terraform provider. So, I took a step back to see if the Terraform provider works but I hit the same error. While digging deeper, I found this GitHub issue that clarifies many questions

Error Analysis from GitHub Issue

The specific issue documented in GitHub issue reveals a fundamental misconfiguration problem. When users specify service_endpoints = "public-and-private" in their Terraform configuration for IBM Secrets Manager, the resulting instance is provisioned with only private endpoints, despite the configuration suggesting otherwise.

Root Cause Identification:
The issue stems from IBM’s April 2025 change to make private endpoints the default for Secrets Manager instances. However, the critical distinction is that Secrets Manager uses a different parameter structure than other IBM Cloud services for endpoint configuration.

Configuration Inconsistency:

• Standard IBM Services: Use service_endpoints parameter
• Secrets Manager: Uses allowed_network within the parameters block

Analyzing the Incorrect Configuration

The problematic Terraform configuration from the GitHub issue:

resource "ibm_resource_instance" "secrets_manager" {
  count                = var.provision_sm_instance == true ? 1 : 0
  name                 = var.sm_instance_name
  service             = "secrets-manager"
  plan                = var.sm_service_plan
  location            = var.region
  resource_group_id   = data.ibm_resource_group.sm_instance.id
  tags                = var.tags
  service_endpoints   = "public-and-private"  # ← This is INCORRECT for Secrets Manager
  
  timeouts {
    create = "20m"
  }
}

Pulumi implementation

secrets_manager = ibm.ResourceInstance(
    "vpn-secrets-manager",
    name="vpn-secrets-manager",
    service="secrets-manager",
    plan="standard",
    location=region,
    resource_group_id=resource_group_id,
    service_endpoints="public-and-private", 
    tags=["vpn", "certificates", "pulumi"],
    opts=secrets_manager_opts
)

Result: Despite specifying service_endpoints = "public-and-private", the instance is provisioned with only private endpoints, leading to DNS resolution failures when attempting to create secret groups.

The Correct Configuration Approach

Based on IBM’s documentation and the GitHub issue resolution, the correct approach uses the parameters block:

resource "ibm_resource_instance" "secrets_manager" {
  count             = var.provision_sm_instance == true ? 1 : 0
  name              = var.sm_instance_name  
  service           = "secrets-manager"
  plan              = var.sm_service_plan
  location          = var.region
  resource_group_id = data.ibm_resource_group.sm_instance.id
  tags              = var.tags
  
  parameters = {
    "allowed_network" = "public-and-private"  # ← This is CORRECT for Secrets Manager
  }
  
  timeouts {
    create = "20m"
  }
}

Pulumi code fix:

pulumi/__main__.py at 57c6600dae9a3fc7048a59da3779200138a22170 · VidyasagarMSC/pulumi

secrets_manager = ibm.ResourceInstance(
    "vpn-secrets-manager",
    name="vpn-secrets-manager",
    service="secrets-manager",
    plan="standard",
    location=region,
    resource_group_id=resource_group_id,
    #service_endpoints=secrets_manager_service_endpoints,
    parameters={
        "allowed_network" : secrets_manager_service_endpoints,    
    },
    tags=["vpn", "certificates", "pulumi"],
    opts=secrets_manager_opts
)

For complete Pulumi code examples, refer to my GitHub repository

Conclusion

The IBM Secrets Manager DNS resolution error stems from a critical configuration difference where Secrets Manager requires parameters = { "allowed_network" = "..." } instead of the standard service_endpoints parameter.

Key Solutions:

• Always use allowed_network within the parameters block for Secrets Manager
• Implement validation to catch configuration errors early
• Use infrastructure modules to encapsulate correct patterns
• Test endpoint accessibility after provisioning

Pulumi Advantages: Offers superior error handling, testing capabilities, and IDE support compared to traditional HCL-based approaches, making it easier to implement robust validation and error handling patterns.

By following these patterns and understanding the service-specific requirements, teams can avoid DNS resolution errors and build reliable infrastructure automation workflows for IBM Secrets Manager.

Related Documentation

• IBM Cloud Secrets Manager
• IBM Cloud VPC VPN
• Pulumi IBM Cloud Provider
• Certificate Management Best Practices

Resolving Secrets Manager DNS Resolution Errors in Terraform and Pulumi IaC was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

This comprehensive guide introduces a powerful yet lightweight Bash script that delivers cross-platform process monitoring with enterprise-grade capabilities. We’ll explore:

• The diverse landscape of Linux distributions and their monitoring needs
• How this script solves real-world administration challenges
• Detailed usage examples for various scenarios
• Advanced integration techniques for professional environments

Understanding the Linux Distribution Landscape

1. Enterprise-Grade Workhorses

Red Hat Enterprise Linux (RHEL), Ubuntu LTS, and SUSE Linux Enterprise dominate mission-critical environments. These distributions prioritize:

• Stability over cutting-edge features
• Long-term support (5+ years typically)
• Certified compatibility with enterprise hardware/software

Monitoring Consideration: These systems often run business-critical services where unexpected process termination can cost thousands per minute of downtime.

2. Rolling Release Distributions

Arch Linux, openSUSE Tumbleweed, and Gentoo appeal to developers and enthusiasts with:

• Continuous updates to the latest software versions
• Minimal default installations
• Customization-focused philosophies

Monitoring Consideration: Frequent updates increase the chance of service disruptions, making automated monitoring particularly valuable.

3. Specialized and Lightweight Systems

From Alpine Linux’s container-optimized 5MB footprint to Kali’s security toolkit, these distributions serve niche purposes:

• Embedded systems (Raspberry Pi OS)
• Security auditing (Kali, Parrot OS)
• Edge computing (Fedora IoT)

Monitoring Consideration: Resource constraints demand ultra-efficient monitoring tools that don’t contribute to system load.

Introducing the Universal Process Monitor

Our solution addresses these diverse needs through intelligent design:

Key Features

1. Dual-Platform Support

• Automatically detects and adapts to Linux or macOS environments
• Handles differences in command syntax between systems

2. Flexible Output Options

• Color-coded terminal output (default)
• JSON format for programmatic use (--json flag)
• Log file support (--log-file option)

3. Intelligent Process Detection

• Uses pgrep for reliable process matching
• Excludes the monitoring script itself from results
• Normalizes output across different platforms

4. Lightweight Operation

• Minimal dependencies (only basic Unix tools)
• Fast execution with low resource usage

Usage Examples

• Basic Process Check

./process_monitor.sh nginx mysql docker

Sample Output:

PROCESS CHECK - Ubuntu 22.04 LTS
TIME: 2023-07-15 14:30:45

nginx                     RUNNING       1234 www-data /usr/sbin/nginx -g daemon off;
mysql                     RUNNING       5678 mysql /usr/sbin/mysqld
docker                    NOT RUNNING   No matching process found

On macOS,

• JSON Output Mode

./process_monitor.sh --json nginx docker

Sample Output:

{
    "hostname": "web-server",
    "timestamp": "2023-07-15 14:30:45",
    "os": "Ubuntu 22.04 LTS",
    "processes": [
        {
            "process": "nginx",
            "status": "RUNNING",
            "pid": "1234",
            "user": "www-data",
            "command": "/usr/sbin/nginx -g daemon off;"
        },
        {
            "process": "docker",
            "status": "NOT RUNNING",
            "pid": null,
            "user": null,
            "command": null
        }
    ]
}

• Logging to File

./process_monitor.sh --log-file "/var/log/process_monitor.log" redis postgres

Integration Possibilities

1. System Monitoring

• Schedule regular checks with cron
• Parse JSON output with monitoring tools

2. CI/CD Pipelines

• Verify service status before deployments
• Check for conflicting processes

3. Troubleshooting

• Quick verification of running services
• Check process dependencies

Performance Characteristics

• Typically completes in under 100ms for basic checks
• Minimal CPU and memory usage
• Efficient process detection without spawning unnecessary subshells
• The script is production-ready and can be immediately deployed in any Linux or macOS environment. Future enhancements could include Windows Subsystem for Linux (WSL) support or additional filtering options.

Installation:

chmod +x process_monitor.sh 
./process_monitor.sh --help

Real-World Implementation Scenarios

1. Proactive Service Health Monitoring

Problem: Web servers occasionally crash without alerting

Solution:

# Cron job checking every 2 minutes
*/2 * * * * /usr/local/bin/monitor.sh --silent nginx || systemctl restart nginx

Outcome: Automatic recovery with failure timestamps in logs

2. Security Baseline Enforcement

Problem: Need to detect unauthorized processes

Solution:

# Daily diff against approved list
diff <(./monitor.sh --json | jq .) <(jq . /etc/approved-processes.json)

Outcome: Immediate visibility into any unexpected processes

3. Containerized Environment Monitoring

Problem: Process visibility inside Kubernetes pods

Solution:

# Kubectl integration example
kubectl exec -it pod-name -- /monitor.sh --json

Outcome: Consistent monitoring across container boundaries

Advanced Integration Patterns

1. Prometheus/Grafana Dashboarding

# Metrics endpoint creation
echo "# HELP process_up Process status" > /var/lib/node_exporter/process.prom
echo "# TYPE process_up gauge" >> /var/lib/node_exporter/process.prom
./monitor.sh --json | jq -r '.processes[] | "process_up{process=\"\(.process)\"} \(.status == "RUNNING" ? 1 : 0)"' >> /var/lib/node_exporter/process.prom

Result: Beautiful dashboards showing service status across your entire fleet

2. Centralized Log Analysis

# Syslog integration example
./monitor.sh | logger -p local0.notice -t process-monitor

Result: Correlation of process events with other system logs in SIEM tools

3. CI/CD Pipeline Gates

# GitLab CI example
verify_services:
  script:
    - if ! ./monitor.sh --json postgresql | grep -q "RUNNING"; then exit 1; fi

Result: Failed deployments when dependent services aren’t available

Conclusion: Building a Monitoring Culture

Implementing this script represents more than just technical deployment — it’s about cultivating observability best practices:

1. Start Small Begin with critical services before expanding coverage
2. Establish Baselines

./monitor.sh --json > /etc/process-baseline-$(date +%F).json

3. Iterate and Expand Add checks gradually as you identify new requirements

4. Document Expectations Maintain a living document of what should be running

This approach gives you a professional-grade monitoring foundation that scales from single servers to enterprise environments while remaining lightweight and adaptable.

The Essential Role of Process Monitoring Scripts in Linux Environments was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is s3cmd?

S3cmd is a powerful open-source command-line tool designed for managing Amazon S3 and S3-compatible object storage services like IBM Cloud Object Storage. It provides a user-friendly interface for performing various operations on S3 buckets and objects, including creating and removing buckets, uploading, downloading, and deleting files.

Key features of s3cmd include:

1. Bucket management: Create, list, and delete buckets.
2. Object operations: Upload, download, and delete files within buckets.
3. Synchronization: Sync local directories with S3 buckets.
4. Access control: Modify bucket policies and access control lists.
5. Website hosting: Create and manage static websites hosted on S3.
6. Multipart uploads: Handle large file uploads efficiently.
7. Encryption: Support for server-side encryption.

S3cmd is compatible with Python 2.6 or newer, including Python 3.x, and runs on Linux and macOS systems. Its versatility and ease of use make it an excellent choice for both beginners and advanced users who need to interact with S3-compatible storage services.

This tool simplifies the process of managing cloud storage, allowing users to automate tasks, integrate S3 operations into scripts, and efficiently handle large amounts of data. Whether you’re a developer, system administrator, or data analyst, s3cmd can streamline your workflow when working with S3 and similar object storage platforms.

s3cmd installation and configuration

1. Install S3cmd following the instructions here

2. To configure, you need to have an S3 cloud instance or any S3 compatible instance with HMAC credentials — Accesskey and Secrets key and endpoint that starts with or similar to s3://…

To understand what are HMAC credentials, refer to my article here

3. Once you have the HMAC credentials and endpoint, you can run the following command for an interactive configuration.

s3cmd --configure

Key commands

Here are some commonly used s3cmd commands:

1. Create a bucket:

s3cmd mb s3://bucket-name

2. List all buckets:

s3cmd ls

3. Upload a file to a bucket:

s3cmd put file.txt s3://bucket-name/

4. Download a file from a bucket:

s3cmd get s3://bucket-name/file.txt

5. Delete a file from a bucket:

s3cmd rm s3://bucket-name/file.txt

6. List objects in a specific bucket:

s3cmd ls s3://bucket-name/

7. Synchronize a local directory with a bucket:

s3cmd sync local_directory/ s3://bucket-name/

8. Set public access for a file:

s3cmd setacl s3://bucket-name/file.txt --acl-public

9. Delete a bucket:

s3cmd rb s3://bucket-name

These commands allow you to perform basic operations like creating and deleting buckets, uploading and downloading files, and managing access permissions. S3cmd offers many more advanced features for managing your S3-compatible storage, making it a powerful tool for both simple and complex storage management tasks.

403 error resolution

The most common error is 403 — Access denied. Which you may think is because of the AccessKeyID or the Secrets key you provided. But actually this is a generic error.

> ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

One of the many reasons for this error is this configuration parameter that you either forgot to set or wrongly configured

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

The right way to configure this parameter is

%(bucket)s.<END_POINT>

For example:
%(bucket)s.s3.eu-de.cloud-object-storage.appdomain.cloud

Once the right of credentials, endpoint & the DNS-style is setup, you won’t see the 403 (InvalidAccesskeyId) error.

If this article is helpful. please clap 👏 and ♻️ share with your network.

S3cmd: CLI for Object Storage was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

SymPy aims to become a full-featured computer algebra system (CAS) while keeping the code simple, extensible, and free of external dependencies. With SymPy, you can perform algebraic manipulations, calculus operations, linear algebra, equation solving, discrete mathematics, and much more, all symbolically, rather than numerically.

In this article, we will explore the SymPy library, its capabilities, and some practical examples that demonstrate how it can be used effectively for symbolic computation.

What is SymPy?

At its core, SymPy provides functions for performing symbolic calculations, which means it works with mathematical expressions in their exact form, rather than approximating values as floating-point numbers. SymPy leverages Python’s object-oriented nature to represent mathematical objects, such as variables, functions, and matrices, as Python objects.

SymPy is built in pure Python and does not require any external libraries or software packages. This makes it easy to install and use, and it’s suitable for applications in scientific computing, education, engineering, and mathematics.

Key Features of SymPy

• Symbolic computation: Perform algebraic operations like simplification, expansion, and factorization on symbolic expressions.
• Calculus: Compute derivatives, integrals, limits, series expansions, and solve differential equations symbolically.
• Linear algebra: Manipulate matrices, solve linear systems, and compute determinants, eigenvalues, and eigenvectors.
• Equation solving: Solve algebraic equations (linear, polynomial, non-linear) symbolically.
• Plotting: Generate plots of functions and equations, both symbolic and numerical.

Installing SymPy

To install SymPy, you can use pip:

pip install sympy

After installation, you can import the library into your Python script:

import sympy as sp

Basic Usage of SymPy

Let’s start by using SymPy to define symbolic variables and perform some basic algebraic manipulations.

Defining Symbolic Variables

You can define symbolic variables using the symbols function. For example:

import sympy as sp

# Define symbols
x, y, z = sp.symbols('x y z')

# Display the symbols
print(x, y, z)

Simple Algebraic Operations

Once you’ve defined symbolic variables, you can perform various algebraic operations such as addition, multiplication, and exponentiation.

# Define an expression
expr = x**2 + 2*x + 1

# Simplify the expression
simplified_expr = sp.simplify(expr)
print("Simplified expression:", simplified_expr)

# Expand a binomial
expanded_expr = sp.expand((x + 1)**2)
print("Expanded expression:", expanded_expr)

# Factor an expression
factored_expr = sp.factor(x**2 + 2*x + 1)
print("Factored expression:", factored_expr)

Simplification and Expansion

SymPy has built-in functions for simplifying and expanding expressions:

# Simplify the expression x**2 - 2*x + 1 - (x - 1)**2
expr_to_simplify = x**2 - 2*x + 1 - (x - 1)**2
simplified = sp.simplify(expr_to_simplify)
print("Simplified expression:", simplified)

Solving Equations

SymPy makes it easy to solve equations symbolically. Here’s how to solve a linear equation:

# Solve x + 3 = 7
solution = sp.solve(x + 3 - 7, x)
print("Solution:", solution)

For solving quadratic equations, SymPy will return all possible roots:

# Solve a quadratic equation
quadratic_solution = sp.solve(x**2 - 5*x + 6, x)
print("Quadratic solution:", quadratic_solution)

Calculus Operations

SymPy excels in symbolic calculus. Let’s look at some common operations, such as differentiation and integration.

Differentiation

To compute the derivative of an expression:

# Derivative of x**3 + 2*x**2 + x with respect to x
derivative = sp.diff(x**3 + 2*x**2 + x, x)
print("Derivative:", derivative)

Integration

To compute the integral of an expression:

# Integral of x**2 with respect to x
integral = sp.integrate(x**2, x)
print("Integral:", integral)

Limits

You can also compute limits of expressions:

# Limit of (sin(x)/x) as x approaches 0
limit = sp.limit(sp.sin(x)/x, x, 0)
print("Limit:", limit)

Series Expansion

SymPy allows you to expand an expression as a series around a given point:

# Series expansion of exp(x) around x = 0
series_expansion = sp.series(sp.exp(x), x, 0, 5)
print("Series expansion:", series_expansion)

Solving Differential Equations

SymPy also provides tools for solving ordinary differential equations (ODEs) symbolically.

# Solve the differential equation y'' - 2*y' + y = 0
y = sp.Function('y')
ode_solution = sp.dsolve(y(x).diff(x, x) - 2*y(x).diff(x) + y(x), y(x))
print("Differential equation solution:", ode_solution)

Linear Algebra with SymPy

SymPy has robust support for linear algebra operations. Let’s see how to perform matrix operations and solve linear systems.

Matrix Operations

# Define matrices
A = sp.Matrix([[1, 2], [3, 4]])
B = sp.Matrix([[5, 6], [7, 8]])

# Matrix addition
C = A + B
print("Matrix addition:\n", C)

# Matrix multiplication
D = A * B
print("Matrix multiplication:\n", D)

# Determinant of a matrix
det_A = A.det()
print("Determinant of A:", det_A)

# Eigenvalues and eigenvectors
eigen = A.eigenvals(), A.eigenvects()
print("Eigenvalues and eigenvectors:", eigen)

Solving a Linear System

You can solve systems of linear equations symbolically using the linsolve function.

# Solve the system of equations: x + y = 2, x - y = 0
system = [x + y - 2, x - y]
solution = sp.linsolve(system, x, y)
print("Solution to the system:", solution)

Plotting with SymPy

SymPy includes basic plotting capabilities for visualizing symbolic expressions. You can use the plot function for this purpose:

from sympy.plotting import plot

# Plot the function sin(x)
plot(sp.sin(x))

Advanced Features of SymPy

SymPy has many more advanced features such as:

1. Mathematical functions: Special functions like gamma, Bessel, and error functions.
2. Solving Diophantine equations: SymPy can handle integer solutions for certain equations.
3. SymPy physics module: For symbolic calculations in physics, including mechanics, optics, and quantum mechanics.
4. Symbolic Fourier transforms: Useful for signal processing and other applications.
5. Code generation: Generate Python, C, Fortran, or other language code for symbolic expressions.

Applications in Various Fields

SymPy finds applications in numerous domains:

1. Physics: Quantum mechanics calculations and equation derivations
2. Engineering: Solving complex mathematical models
3. Computer Science: Algorithm analysis and optimization
4. Finance: Option pricing and risk analysis

Code Generation

One of SymPy’s powerful features is its ability to generate code in various programming languages:

from sympy.utilities.codegen import codegen 
expr = x**2 + sin(x) [(c_name, c_code), (h_name, c_header)] = codegen(('f', expr), 'C', 'file', header=False, empty=False) 
print(c_code)

This feature allows users to convert symbolic expressions into efficient, executable code for numerical computation.

The Jupyter Notebook with the outputs is available on the GitHub repository. You can run the Jupyter Notebook on Colab following the instructions on the GitHub repository.

Conclusion

SymPy is a powerful tool for symbolic mathematics in Python. It can simplify complex expressions, solve equations, perform calculus operations, and handle linear algebra tasks symbolically. Whether you are a researcher, educator, engineer, or student, SymPy provides an easy-to-use yet powerful framework for performing a wide variety of symbolic computations.

With its simple installation process, ease of use, and extensive capabilities, SymPy has become a popular choice in the Python ecosystem for symbolic mathematics. By using the examples in this article as a starting point, you can begin to explore the full range of features SymPy has to offer.

Originally published at https://vmacwrites.substack.com.

An Introduction to SymPy: A Python Library for Symbolic Mathematics was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

Created by Wes McKinney in 2008, it provides data structures and functions for working with structured data efficiently. Pandas allows users to analyze big data, clean messy datasets, and derive meaningful insights.

Key Features of Pandas

1. Data Structures: Pandas introduces two primary data structures:

• Series: A one-dimensional labeled array
• DataFrame: A two-dimensional labeled data structure with columns of potentially different types

2. Data Manipulation: Pandas offers functions for analyzing, cleaning, exploring, and manipulating data.

3. Data Analysis: It enables users to perform complex operations like correlation analysis, grouping, and statistical calculations.

4. Data Visualization: Pandas integrates well with other libraries to create insightful visualizations.

Practical Examples

Indexing with loc

The loc function enables label-based indexing in DataFrames, allowing precise data selection:

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}, index=['x', 'y', 'z'])

# Select rows with label 'y' and 'z', and columns 'A' and 'C'
result = df.loc[['y', 'z'], ['A', 'C']]
print(result)

The iloc function provides integer-based indexing for DataFrame selection:

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})

# Select rows 0 and 2, and columns 1 and 2
result = df.iloc[[0, 2], [1, 2]]
print(result)

Date Conversion with to_datetime

The to_datetimefunction transforms various date formats into standardized datetime objects:

import pandas as pd

# Convert string to datetime
date_string = "2023-09-17 14:30:00"
dt_object = pd.to_datetime(date_string)
print(dt_object)

# Convert multiple date strings
date_series = pd.Series(['20200101', '20200201', '20200301'])
dt_series = pd.to_datetime(date_series, format='%Y%m%d')
print(dt_series)

Output:

2023-09-17 14:30:00
0   2020-01-01
1   2020-02-01
2   2020-03-01
dtype: datetime64[ns]

Pandas simplifies data manipulation tasks, making it an essential tool for data scientists and analysts. Its versatile functions like loc, iloc, and to_datetime provide powerful ways to interact with and transform data, enabling efficient data processing and analysis in Python.

Something to consider while using loc or iloc

Let’s convert the object column date to datetime using loc

import pandas as pd

df = pd.DataFrame({'date': ['2023-01-01', '2023-02-15', '2023-03-31']})

df.loc[:, 'date'] = pd.to_datetime(df.loc[:, 'date'])

print(df)
print(df.dtypes)

Output: 

   date
0  2023-01-01 00:00:00
1  2023-02-15 00:00:00
2  2023-03-31 00:00:00
date    object
dtype: object

If you observe, the dtype is object not datetime64[ns]. If you try to extract the date using df['date'].dt.date

You will see an error as the conversion was not successful.

Traceback (most recent call last):
  File "/HelloWorld.py", line 11, in <module>
    print(df.dt.date)
          ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/generic.py", line 6299, in __getattr__
    return object.__getattribute__(self, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'dt'. Did you mean: 'at'?

The reason lies in the changes made in version 2.x.x of Pandas.

From What’s new in 2.0.0 (April 3, 2023):

Changed behavior in setting values with df.loc[:, foo] = bar or df.iloc[:, foo] = bar, these now always attempt to set values inplace before falling back to casting (GH 45333)

How to overcome:

The best way to address this issue is to either avoid using loc or iloc or as suggested on the Pandas documentation use DataFrame.__setitem__()

df = pd.DataFrame({'date': ['2023-01-01', '2023-02-15', '2023-03-31']})

df['date'] = pd.to_datetime(df.loc[:, 'date'])

print(df)
print(df.dtypes)

print(df['date'].dt.date)

Additional read:

https://stackoverflow.com/questions/76766136/pandas-pd-to-datetime-assigns-object-dtype-instead-of-datetime64ns

Pandas: Conversion using loc and iloc was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

NVIDIA MIG (Multi-Instance GPU) is a technology introduced by NVIDIA to partition their GPUs (particularly the A100 and A30 models) into smaller, isolated instances. This allows multiple workloads or containers to run on a single physical GPU, enabling more efficient resource utilization and optimizing GPU usage, especially in multi-tenant environments.

With Kubernetes, the integration of MIG provides dynamic and flexible GPU resource allocation, optimizing GPU resources at a granular level. Kubernetes supports resource management through the nvidia.com/gpu resource plugin, which is extended to handle the MIG-enabled GPUs using specific operators and controllers.

GPU Optimization Techniques

Single Strategy

In this approach, all MIG devices on a GPU are created with the same size. For example, on a P4d.24XL instance, you could create 56 slices of 1g.5gb or 24 slices of 2g.10gb.

Mixed Strategy

This technique allows for creating MIG devices of different sizes on the same GPU. For instance, you could partition a GPU into two 1g.10gb units, one 2g.20gb unit, and one 3g.40gb unit.

Kubernetes Operators for MIG

Two main operators facilitate MIG management in Kubernetes:

1. NVIDIA GPU Operator: This operator automates the creation of MIG partitions based on node labels. It uses a ConfigMap to define MIG profiles and applies them to labeled nodes.
2. k8s-mig-operator: This is a Python-based operator that provides more granular control over MIG configurations. It allows for custom MIG profiles and supports toggling MIG on/off for individual GPUs.

Key Concepts in NVIDIA MIG for Kubernetes

GPU Partitioning:

MIG allows a single GPU to be split into several smaller, independent instances (each with its own memory, compute cores, and other resources).
Each instance of MIG can be treated as a separate, discrete resource, enabling better isolation and multi-tenant support, similar to how virtual machines or containers work.

Dynamic Resource Allocation

With dynamic resource allocation, Kubernetes can allocate and deallocate GPU resources as needed. This provides flexibility in managing GPU workloads and ensures that resources are efficiently distributed among workloads based on real-time demand.

The MIG feature enables a more fine-grained allocation of GPU resources, allowing each container or pod to use a portion of the GPU instead of the whole GPU. This is especially beneficial in environments where GPU resources are a limited and valuable commodity.

Kubernetes can dynamically allocate MIG devices to pods using the resources section in pod specifications. For example:

resources:
  limits:
    nvidia.com/mig-1g.5gb: 1

This requests one 1g.5gb MIG device for the pod.

Benefits of MIG in Kubernetes

1. Improved GPU Utilization: MIG allows multiple pods to share a single GPU, increasing overall utilization.
2. Isolation: Each MIG instance is hardware-isolated, ensuring performance predictability.
3. Flexibility: Administrators can create various MIG profiles to match different workload requirements.
4. Cost Efficiency: By running more pods per GPU, organizations can optimize their hardware investments.

MIG technology, combined with Kubernetes operators, provides a powerful toolset for maximizing GPU resources in containerized environments, especially for AI and high-performance computing workloads.

Key Operators for Managing NVIDIA MIG in Kubernetes

To enable and manage NVIDIA MIG within Kubernetes clusters, several operators and tools are used. These operators automate the deployment, monitoring, and scaling of GPU resources (including MIG partitions).

1. NVIDIA GPU Operator:

• The NVIDIA GPU Operator is a Kubernetes operator that simplifies the management of NVIDIA GPUs in Kubernetes clusters. It automates the deployment of NVIDIA drivers, runtime, monitoring tools, and device plugins.
• The operator supports the configuration of MIG-enabled GPUs, dynamically allocating GPU instances to Kubernetes workloads.
• It ensures that all necessary components (like device plugins) are installed and configured properly to expose MIG partitions for use by workloads.

2. NVIDIA Device Plugin for Kubernetes:

• The NVIDIA Device Plugin is a Kubernetes device plugin that allows pods to request GPUs. When MIG is enabled, this plugin can be extended to manage MIG instances.
• It facilitates the allocation of MIG instances to workloads and ensures that workloads are scheduled based on available GPU instances.

3. NVIDIA DCGM Exporter:

• The NVIDIA Data Center GPU Manager (DCGM) Exporter is used to expose GPU metrics for monitoring within Kubernetes. It helps track GPU health, performance, and utilization, including MIG instances, which is crucial for dynamically adjusting resource allocation.

Conclusion

NVIDIA MIG, combined with Kubernetes, offers a powerful mechanism to optimize GPU usage in multi-tenant, resource-constrained environments. The ability to partition GPUs into smaller, isolated instances allows for better resource utilization, fault isolation, and flexibility in GPU scheduling. By leveraging dynamic resource allocation, Kubernetes can adjust to workloads in real-time, ensuring optimal performance and efficient scaling of GPU resources. The NVIDIA GPU Operator, along with the NVIDIA Device Plugin, helps automate and manage this complex ecosystem, making it easier for users to utilize MIG-enabled GPUs effectively in Kubernetes clusters.

Resources:

• https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
• https://docs.nvidia.com/launchpad/ai/fleet-command/latest/fc-step-09.html
• https://github.com/riskfuel/k8s-mig-operator
• https://towardsdatascience.com/how-to-increase-gpu-utilization-in-kubernetes-with-nvidia-mps-e680d20c3181?gi=901c97cfd4c1
• https://www.leadergpu.com/articles/554-what-is-nvidia-mig
• https://aws.amazon.com/blogs/containers/maximizing-gpu-utilization-with-nvidias-multi-instance-gpu-mig-on-amazon-eks-running-more-pods-per-gpu-for-enhanced-performance/
• https://blog.realvarez.com/get-more-out-of-gpus-with-nvidia-multi-instance-gpu/
• https://blogs.nvidia.com/blog/multi-instance-gpus/
• https://catalog.ngc.nvidia.com/orgs/nvidia/teams/cloud-native/containers/k8s-mig-manager
• https://www.infracloud.io/blogs/guide-to-nvidia-gpu-operator-in-kubernetes/
• https://docs.nvidia.com/datacenter/cloud-native/kubernetes/latest/index.html

Nvidia MIG with GPU Optimization in Kubernetes was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anthropic has introduced the Model Context Protocol (MCP), an open-source standard designed to simplify and standardize connections between AI models and external data sources. This protocol aims to solve the challenge of integrating AI systems with various tools and data repositories, which previously required custom implementations for each new data source.

The announcement, available at https://www.anthropic.com/news/model-context-protocol, provides detailed information about this new protocol.

Key Features and Benefits

1. Universal Connectivity: MCP enables AI systems to connect with diverse data sources, including content repositories, business tools, and development environments.

2. Standardized Integration: Instead of creating custom connectors for each data source, developers can build against a single, universal protocol.

3. Two-Way Connections: MCP facilitates secure, bidirectional communication between AI-powered tools and data sources.

4. Open-Source Approach: The protocol is open-source, encouraging community-driven adoption and innovation.

5. Improved AI Performance: By providing AI models with better access to relevant data, MCP aims to enhance the quality and relevance of AI-generated responses.

Technical Architecture

MCP utilizes a client-server architecture:

1. MCP Servers: Expose data from various sources
2. MCP Clients: AI applications that connect to these servers

The protocol consists of three main components:

1. Protocol Layer: Handles message framing, request/response linking, and high-level communication patterns
2. Transport Layer: Manages communication between client and server
3. SDKs: Currently available for TypeScript and Python

Security and Privacy

MCP prioritizes security through a local-first approach:

1. Local Connections: Initially, MCP only supports connections to servers running on the local machine
2. Explicit Permissions: The protocol requires specific permissions for each tool and interaction
3. Server-Controlled Resources: Servers maintain control over their resources, eliminating the need to share API keys with LLM providers

Current Status and Adoption

1. Early Adopters: Companies have begun integrating MCP into their systems
2. Development Tools: Various firms are adding MCP support to their platforms
3. Pre-built Servers: Servers are available for popular systems like Google Drive, Slack, GitHub, and SQL databases

Future Developments

1. Remote Connections: Work is in progress to allow for remote servers with enterprise-grade authentication
2. Expanded Toolkit: Plans to provide developer toolkits for deploying remote production MCP servers

Potential Impact

MCP has the potential to become a foundational tool for AI model integration, similar to how ODBC standardized database connectivity in the 1990s. By simplifying AI integration and improving data accessibility, MCP could accelerate the adoption and effectiveness of AI systems across various industries and applications.

Key points from the announcement include:

1. Open-source nature: MCP is released as an open-source project to encourage community contributions and adoption.

2. Purpose: MCP aims to solve the challenge of integrating AI systems with various data sources, replacing fragmented integrations with a single, universal protocol.

3. Architecture: MCP uses a client-server model, where MCP servers expose data from various sources, and MCP clients (AI applications) connect to these servers.

4. Components: The announcement introduces three major components:

• The Model Context Protocol specification and SDKs
• Local MCP server support in Claude Desktop apps
• An open-source repository of MCP servers

5. Pre-built servers: Anthropic has shared pre-built MCP servers for popular systems like Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer.

6. Early adoption: Companies like Block and Apollo have integrated MCP, while development tool firms such as Zed, Replit, Codeium, and Sourcegraph are adding MCP support.

7. Getting started: The announcement provides instructions for developers to start building and testing MCP connectors, including installing pre-built servers through the Claude Desktop app and following a quickstart guide.

8. Future developments: Anthropic plans to provide developer toolkits for deploying remote production MCP servers that can serve entire Claude for Work organizations.

9. Community involvement: Anthropic emphasizes their commitment to building MCP as a collaborative, open-source project and invites developers, enterprises, and early adopters to contribute to the ecosystem.

Conclusion

The announcement highlights MCP’s potential to significantly improve AI assistants’ ability to access and utilize relevant data, potentially leading to more accurate and contextually appropriate responses across various applications and industries.

Model Context Protocol Overview was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.

The chain of trust is a fundamental concept in digital security that underpins the reliability and authenticity of SSL/TLS certificates used to secure online communications. It establishes a hierarchical structure of trust, starting from a trusted root Certificate Authority (CA) and extending down to the end-entity certificate used by a website or service.

This chain is important for several reasons. First of all, it allows for the scalable distribution of trust across the internet. Instead of requiring every device to trust millions of individual certificates, they only need to trust a small number of root CAs. Second, it provides a mechanism for verifying the authenticity of certificates. When a browser encounters a certificate, it can trace the chain back to a trusted root, ensuring that each link in the chain is valid and trustworthy. This process helps prevent man-in-the-middle attacks and other forms of certificate fraud. Additionally, the chain of trust enables the revocation of compromised certificates without undermining the entire system. If an intermediate CA is compromised, only the certificates issued by that CA need to be revoked, rather than all certificates issued by the root CA. This flexibility and security make the chain of trust an essential component of modern internet security.

Understanding the Chain of Trust

The chain of trust is a hierarchical structure of digital certificates that validates the authenticity of a certificate. This chain typically consists of three main components:

1. Root Certificate Authority (Root CA)
2. Intermediate Certificate Authority (Intermediate CA)
3. End-entity Certificate (Server or Client Certificate)

Root Certificate Authority (Root CA)

The Root CA sits at the top of the trust hierarchy. It is a self-signed certificate, meaning it vouches for its own authenticity. Root CAs are implicitly trusted by web browsers and operating systems. They are responsible for issuing certificates to Intermediate CAs and, in some cases, directly to end-entities.

Key characteristics of Root CAs:

• Self-signed certificates
• Stored securely offline to prevent compromise
• Long validity periods (typically 20–30 years)
• Highly trusted and recognized globally

Intermediate Certificate Authority (Intermediate CA)

Intermediate CAs bridge the gap between Root CAs and end-entity certificates. They are issued by Root CAs and, in turn, issue certificates to end-entities or other Intermediate CAs. The use of Intermediate CAs enhances security by allowing the Root CA to remain offline while the Intermediate CA handles day-to-day certificate issuance.

Key characteristics of Intermediate CAs:

• Issued by Root CAs
• Can issue certificates to end-entities or other Intermediate CAs
• Shorter validity periods than Root CAs (typically 10–15 years)
• Enhance security and flexibility in certificate management

End-entity Certificate

This is the certificate issued to a specific entity, such as a website or email server. It’s the certificate that end-users interact with when visiting a secure website or sending encrypted email.

Key characteristics of End-entity Certificates:

• Issued by Intermediate CAs or, occasionally, Root CAs
• Shortest validity period (typically 1–2 years)
• Contains information about the entity it represents (e.g., domain name for websites)

Role in Security

The chain of trust plays a crucial role in ensuring the security and integrity of online communications:

1. Authentication: It verifies the identity of the certificate holder, ensuring that you’re communicating with the intended party.
2. Integrity: It ensures that the certificate hasn’t been tampered with or altered in transit.
3. Encryption: While not directly responsible for encryption, a valid certificate chain is necessary for establishing encrypted connections.
4. Revocation: If a certificate in the chain is compromised, it can be revoked, invalidating all certificates below it in the chain.

Tools to Check Chain of Trust and Certificate Validity

Several tools are available to verify the chain of trust and validate certificates:

OpenSSL

OpenSSL is a versatile command-line tool that can perform various SSL/TLS-related tasks.

To check a certificate chain:

openssl s_client -connect example.com:443 -showcerts

This command will display the entire certificate chain and highlight any issues.

Certificate Checker Tools

Many online services provide comprehensive certificate checking capabilities:

1. SSL Labs Server Test: Offers in-depth analysis of SSL/TLS configurations.
2. DigiCert Certificate Inspector: Scans for certificate issues across your entire network.
3. Qualys SSL Server Test: Provides detailed information about SSL/TLS configuration and potential vulnerabilities.

Browser Developer Tools

Modern web browsers include developer tools that can display certificate information:

1. Click on the padlock icon in the address bar.
2. Select “Certificate” or a similar option.
3. Review the certificate details, including the chain of trust.

Certificate Validation Process

When a client connects to a server, it performs several checks to validate the certificate:

1. Signature Verification: The client verifies each certificate’s digital signature in the chain, starting from the end-entity certificate up to the Root CA.
2. Trust Anchor Check: The client checks if the Root CA is in its list of trusted root certificates.
3. Validity Period: The client ensures all certificates in the chain are within their validity periods.
4. Revocation Check: The client may check if any certificates in the chain have been revoked using Certificate Revocation Lists (CRLs) or Online Certificate Status Protocol (OCSP).
5. Name Matching: For the end-entity certificate, the client verifies that the certificate’s Subject Alternative Name (SAN) or Common Name (CN) matches the domain being accessed.

Common Issues in Certificate Chain Validation

1. Incomplete Chain: The server doesn’t provide all necessary intermediate certificates.
2. Expired Certificates: One or more certificates in the chain have expired.
3. Revoked Certificates: A certificate in the chain has been revoked.
4. Name Mismatch: The server’s certificate doesn’t match the domain name being accessed.
5. Untrusted Root: The Root CA isn’t in the client’s list of trusted roots.

The testssl.sh tool is a powerful, open-source command-line utility for testing SSL/TLS enabled servers. It provides comprehensive information about a server’s SSL/TLS configuration, including supported protocols, ciphers, and certificate details. This tool is particularly useful for administrators and security professionals looking to assess the security of their SSL/TLS implementations.

./testssl.sh --jsonfile-pretty SSL_test.json https://example.com:443/

While running ./testssl.sh, if you see an error Chain of trust — NOT ok , you need to check whether intermediate certificates are supplied and are part of the chain of trust.

If you are supplying all the certificates including the intermediate certificate but still seeing the error, supply the certificates to the testssh using the add-ca flag pointing to the folder with all the certificates in the chain of trust

./testssl.sh --add-ca ~/Downloads/amc-private-certificate --jsonfile-pretty SSL_test.json https://example.com:443/

The result should now be

{
    "id": "cert_chain_of_trust",
    "severity": "OK",
    "finding": "passed."
}

Conclusion

Understanding the chain of trust and regularly validating certificates is crucial for maintaining a secure online environment. By utilizing the tools and knowledge discussed, you can ensure that your digital certificates are properly configured and trusted, thereby safeguarding your online communications and transactions.

Further recommended reading

1. Troubleshooting SSL: Why Your SSL Certificate Isn’t Working on Windows

2. Advanced SSL Certificate Troubleshooting for Windows: Chain of Trust, Debugging, and Best Practices

Chain of Trust: Decoding SSL Certificate Security Architecture was originally published in vmacwrites on Medium, where people are continuing the conversation by highlighting and responding to this story.