[k8s] kubectl返回certificate has expired or is not yet valid错误

最新推荐文章于 2024-10-16 18:22:54 发布

原创最新推荐文章于 2024-10-16 18:22:54 发布 · 5k 阅读

1 ·

本内容遵循CC 4.0 BY-SA版权协议

k8s 专栏收录该内容

2 篇文章

订阅专栏

当在k8s集群中遇到kubectl返回'certificate has expired or is not yet valid'错误时，原因可能是默认客户端证书有效期已过。解决方案包括升级集群或手动更新证书。升级集群可按照官方文档操作，手动更新则需在master节点上执行特定命令，包括备份、更新证书和重启control plane Pods。

前两天在k8s master上执行kubectl get pod时，返回x509 certificate has expired or is not yet valid错误。

原因

使用kubeadm安装k8s时，默认生成的client certificate的有效期是1年。
可以使用kubeadm certs check-expiration或kubeadm alpha certs check-expiration(较早的kubeadm版本中，该命令还是在alpha阶段)查看当前证书的信息。
后续命令为了避免重复，均使用不带alpha的格式，请根据实际情况进行调整。

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 30, 2020 23:36 UTC   364d                                    no
apiserver                  Dec 30, 2020 23:36 UTC   364d            ca                      no
apiserver-etcd-client      Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Dec 30, 2020 23:36 UTC   364d            ca                      no
controller-manager.conf    Dec 30, 2020 23:36 UTC   364d                                    no
etcd-healthcheck-client    Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
etcd-peer                  Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
etcd-server                Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
front-proxy-client         Dec 30, 2020 23:36 UTC   364d            front-proxy-ca          no
scheduler.conf             Dec 30, 2020 23:36 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 28, 2029 23:36 UTC   9y              no
etcd-ca                 Dec 28, 2029 23:36 UTC   9y              no
front-proxy-ca          Dec 28, 2029 23:36 UTC   9y              no

解决方法

以下两种方案仅适用于使用kubeadm安装的并使用默认内置CA的k8s集群

方案1-升级集群

升级集群的过程中，kubeadm会更新所有的证书。该方案相对比较简单，按照官方文档 Upgrading kubeadm clusters操作即可

方案2-手动更新证书

以下命令请在所有master节点上执行

备份/etc/kubernetes目录，可以使用sudo tar czf etc-kubernetes.tar.gz /etc/kubernetes/
创建一个kubeadmconfig.yaml文件，格式如下。具体内容请按实际替换。

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: "x.x.x.x/x"
  podSubnet: "y.y.y.y/y"
controlPlaneEndpoint: "z.z.z.z:z"

执行sudo kubeadm certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

重启control plane Pods，来完成pod证书更新

mkdir $HOME/manifests
sudo mv /etc/kubernetes/manifests/*.yaml $HOME/manifests/ #等待至少20秒，让kubelet去terminate相关pod
sudo mv $HOME/manifests/*.yaml /etc/kubernetes/manifests/ #等待至少1分钟，让kubelet重新创建相关pod

更新/etc/kubernetes/kubelet.conf

sudo tar czf kubelet-pki.tar.gz /var/lib/kubelet/pki/ #备份kubelet pki文件
#删除/var/lib/kubelet/pki/kubelet-client*文件
sudo kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE --config kubeadmconfig.yaml > kubelet.conf #$NODE需要替换为k8s节点名称
#修改生成的kubelet.conf中cluster name和server endpoint，于备份文件中的相同
sudo cp kubelet.conf /etc/kubernetes/kubelet.conf
sudo systemctl daemon-reload && sudo systemctl restart kubelet
#等待/var/lib/kubelet/pki/kubelet-client*文件重新生成
sudo kubeadm init phase kubelet-finalize all

检查集群状态
kubectl get node
kubectl get pod

引用链接

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert