Recover a failed upgrade in Google Distributed Cloud

Occasionally, Google Distributed Cloud runs into conditions where you need to start an upgrade over again, or fix an error condition to complete the upgrade successfully. The following conditions may occur when upgrading clusters. Try to complete the cluster upgrade as noted with the condition.

Recover admin, hybrid, and standalone cluster upgrades

Admin, hybrid, and standalone clusters are all upgraded with the bmctl upgrade command.

If your upgrade doesn't complete successfully, check the following conditions and try the upgrade again.

  • bmctl fails to parse the config file due to a mistake. Fix the config file and rerun the command.
  • bmctl fails to bootstrap the temporary cluster. Retry the command.
  • the log files indicate that the upgrade failed a preflight check. Correct the preflight condition, or force the upgrade.
  • the upgrade took too long (more than 30 minutes) or timed out. Retry the upgrade command.
  • the upgrade stops during the initialization phase:
    • For version 1.35 and later, if you use RHEL 8 or Rocky Linux, check the node logs for Ansible execution errors indicating an unsupported Python version. Ansible requires Python 3.9 or newer on the target cluster. Verify the installation of Python 3.9 or higher on the failing node and retry the operation.
    • For version 1.35 and later, check that cgroupsv2 is enabled. For more information on cgroupsv2, see migrating to cgroup v2.

Recover user cluster upgrades

User clusters are upgraded from an associated admin cluster with the kubectl apply command.

If your upgrade doesn't complete successfully, check the following conditions and try the upgrade again.

  • Preflight check failed. Check the preflight check log and fix the corresponding error.
  • You may need to construct a new preflight custom resource following these instructions and apply it to the cluster. Check that the triggered preflight passes, and that the cluster reconciler picks up the latest passing state.

What's next

If you need additional assistance, reach out to Cloud Customer Care. You can also see Getting support for more information about support resources, including the following:

  • Requirements for opening a support case.
  • Tools to help you troubleshoot, such as your environment configuration, logs, and metrics.
  • Supported components.