Skip to content

Failed to remove secrets(driver=pass) when inspected storages no longer exist. #27666

@Zeor-Xain

Description

@Zeor-Xain

Issue Description

Podman can't remove a secret that was created using --driver=pass if the gnupg and password-store storages corresponding to podman secret inspect no longer exist.

Is there no way to force removal (--force ...?)

Steps to reproduce the issue

This proof demonstrates the issue using nested podman containers (podman-in-podman):

The test proceeds as follows within a container:

  1. A secret ('secret-1') is created and successfully removed while its driver directories exist.
  2. A second secret ('secret-2') is created.
  3. The 'pass' driver's storage and GPG directories are manually deleted.
  4. podman secret rm secret-2 is executed and is expected to fail,
    demonstrating the issue.

Caution

This example was developed using nested Podman containers to safely demonstrate the issue, without
interacting with real gnupg or password-store storages.
However, only use it if you know what you are doing.

Click too see the code
podman run -it --rm --privileged \
  registry.fedoraproject.org/fedora:43 \
  /bin/bash -c '
    set -e
    
    echo; echo "------ Create test script ------"; echo
    
    cat << \EODRIVERSH > /tmp/proof.sh
#!/usr/bin/env bash
set -e 

pushd $(pwd) > /dev/null

trap "popd > /dev/null" EXIT

# Podman secret name
PODMAN_SECRET_NAME="${1:-driver_pass_test}"

# GPG key ID used by `pass`
GPG_KEY_EMAIL="[email protected]"
## Name for the new key if the ID does not exist
GPG_KEY_NAME="Podman Secret"

# Configure `gpg`: GPG key directory (will be created if it does not exist)
GNUPGDIR="$(pwd)/.gnupg-podman"

# Configure `pass`:
## `pass` storage directory (sub-dir "podman"). (will be created if it does not exist)
PASSWORD_STORE_DIR="$(pwd)/.password-store-podman/podman"
## `gpg` options used by `pass` when getting keys
PASSWORD_STORE_GPG_OPTS="--homedir $GNUPGDIR"

# Export for `pass` usage
export PASSWORD_STORE_DIR
export PASSWORD_STORE_GPG_OPTS

# gpg wrapper with --homedir
gpg() {
  command gpg --homedir "$GNUPGDIR" "$@"
}

# Create example credentials file
cat << \EOF > "$(pwd)/cred-example.json"
{
  "id": 1,
  "pass": "secret_pass"
}
EOF

# Ensure the GPG key exists (create if necessary)
if ! gpg --list-keys "$GPG_KEY_EMAIL" &>/dev/null; then
  mkdir -m 700 "$GNUPGDIR"
  # These options set a blank password and inhibit the prompt: --batch --yes --pinentry-mode loopback --passphrase ""
  if gpg --batch --yes --pinentry-mode loopback --passphrase "" --quick-gen-key "$GPG_KEY_NAME <$GPG_KEY_EMAIL>"; then
    echo "Key successfully created in $GNUPGDIR"
  fi
else
  echo "Key already exists for $GPG_KEY_EMAIL in $GNUPGDIR"
fi

# Ensure `pass` storage exists
if ! pass ls &> /dev/null; then
  # The PASSWORD_STORE_DIR directory will be created if it does not exist
  # Initialize the `pass` storage, assigning a "gpg-id" to a specific "podman" sub-storage
  pass init "$GPG_KEY_EMAIL" 2> /dev/null
fi

# Create a podman secret protected by `pass` (--driver=pass) from the credentials file.
# The `pass` driver considers the GNUPGDIR and PASSWORD_STORE_DIR settings for the GPG_KEY_EMAIL key
podman secret create --replace --driver=pass \
  "--driver-opts=root=$PASSWORD_STORE_DIR" \
  "--driver-opts=gpghomedir=$GNUPGDIR" \
  "--driver-opts=key=$GPG_KEY_EMAIL" \
  "$PODMAN_SECRET_NAME" /tmp/cred-example.json

# End of script

EODRIVERSH
    [[ $? -eq 0 ]] && echo "Script created: /tmp/proof.sh"
    chmod +x /tmp/proof.sh && echo "Script executable: /tmp/proof.sh"
    
    
    
    #  --- LOGS ---
    
    GREEN="\033[0;32m"
    RED="\033[0;31m"
    NC="\033[0m" # No Color  
    log_success() { echo -e "${GREEN}[PASS]${NC} $1"; }
    log_error() { echo -e "${RED}[FAIL]${NC} $1";  }
    
    echo; echo "----- Container Installs and environment config-----"; echo
    echo "Installs: podman pass gnupg2 pinentry"
    
    dnf install -y podman pass gnupg2 pinentry
    
    # Use "vfs" for improved compatibility with podman-in-podman.
    sudo tee /etc/containers/storage.conf > /dev/null << \EOVFS
[storage]
driver = "vfs"
EOVFS
    
    podman system reset -f
    
    echo; echo "------ Prepare Tests ------"; echo
    
    cd /tmp
    
    # create secret with driver=pass from non default gpg and pass root directories
    ./proof.sh "secret-1" > /dev/null
    
    # create secret with driver=pass from non default gpg and pass root directories
    ./proof.sh "secret-2" > /dev/null
    
    
    echo; echo ".......... Tests ..........."; echo
    
    podman secret exists "secret-1"           && log_success "Secret exists: secret-1" || log_error "Secret not exists: secret-1 "
    podman secret rm "secret-1" &> /dev/null  && log_success "Can remove secret: secret-1" || log_error "Can not remove secret: secret-1"
    ! podman secret exists "secret-1"         && log_success "Secret no more exists: secret-1" || log_error "Secret still exists: secret-1"
    
    podman secret exists "secret-2"             && log_success "Secret exists: secret-2 " ||  log_error "Secret not exists: secret-2 "
    rm -rf /tmp/.gnu* /tmp/.pass* > /dev/null   && log_success "Success to remove: /tmp/.gnu* /tmp/.pass*" || log_error "Fail to remove: /tmp/.gnu* /tmp/.pass*"
    podman secret rm "secret-2" &> /dev/null    && log_success "Can remove secret: secret-2" || log_error "Can not remove secret: secret-2"
    ! podman secret exists "secret-2"           && log_success "Secret no more exists: secret-2 " || log_error "Secret still exists: secret-2 "

    # Interactive 
    /bin/bash
'

Describe the results you received

.......... Tests ...........

[PASS] Secret exists: secret-1
[PASS] Can remove secret: secret-1
[PASS] Secret no more exists: secret-1
[PASS] Secret exists: secret-2 
[PASS] Success to remove: /tmp/.gnu* /tmp/.pass*
[FAIL] Can not remove secret: secret-2
[FAIL] Secret still exists: secret-2 

Describe the results you expected

.......... Tests ...........

[PASS] Secret exists: secret-1
[PASS] Can remove secret: secret-1
[PASS] Secret no more exists: secret-1
[PASS] Secret exists: secret-2 
[PASS] Success to remove: /tmp/.gnu* /tmp/.pass*
[PASS] Can remove secret: secret-2
[PASS] Secret no more exists: secret-2

podman info output

host:
  arch: amd64
  buildahVersion: 1.42.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-2.fc43.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 96.49
    systemPercent: 0.9
    userPercent: 2.62
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "43"
  eventLogger: file
  freeLocks: 2048
  hostname: dd113fd30f13
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.14.0-36-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 966803456
  memTotal: 6211403776
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.17.0-1.fc43.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: netavark-1.17.0-1.fc43.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.0
  ociRuntime:
    name: crun
    package: crun-1.25.1-1.fc43.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.25.1
      commit: 156ae065d4a322d149c7307034f98d9637aa92a2
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250919.g623dbf6-1.fc43.x86_64
    version: |
      pasta 0^20250919.g623dbf6-1.fc43.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16123498496
  swapTotal: 17179860992
  uptime: 10h 55m 27.00s (Approximately 0.42 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 80107012096
  graphRootUsed: 11603017728
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.7.0
  BuildOrigin: Fedora Project
  Built: 1762819200
  BuiltTime: Tue Nov 11 00:00:00 2025
  GitCommit: 0370128fc8dcae93533334324ef838db8f8da8cb
  GoVersion: go1.25.4 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.0

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions