K8S pod、Docker 容器镜像 无法跟宿主机外网ip通讯（宿主机也可能是多外网网卡），pod无法连接K8S Api Server，CoreDNS notReady问题

最近学习gitlab cicd，想要弄金丝雀发布，装istio。

gitlab蓝绿发布ok了。

装istio。发现k8s的 CoreDNS Ready 为 0/1。

不多说，先说解决方法：

 

[root@centos xxxuseer]#  kubectl get pod -o wide -A
NAMESPACE      NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
default        test-pod-new                        1/1     Running   0          15h   10.244.0.93      centos   <none>           <none>
kube-flannel   kube-flannel-ds-46s9b               1/1     Running   0          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    coredns-66f779496c-xv5jv            0/1     Running   0          15h   10.244.0.91      centos   <none>           <none>
kube-system    coredns-66f779496c-z2ftl            0/1     Running   0          15h   10.244.0.92      centos   <none>           <none>
kube-system    etcd-centos                      1/1     Running   2          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-apiserver-centos            1/1     Running   2          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-controller-manager-centos   1/1     Running   11         15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-proxy-b74jv                    1/1     Running   0          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-scheduler-centos            1/1     Running   12         15h   宿主机真实外网IP   centos   <none>           <none>
[root@centos xxxuseer]# ip rule list
0:	from all lookup local 
32764:	from 157.122.224.14 lookup e1 
32765:	from 宿主机真实外网IP lookup e0 
32766:	from all lookup main 
32767:	from all lookup default 
[root@centos xxxuseer]# ip route show e0
Error: any valid prefix is expected rather than "e0".
[root@centos xxxuseer]# ip route show table e0
default via 157.122.224.9 dev enp134s0f0 src 宿主机真实外网IP 
127.0.0.0/8 dev lo scope link 
[root@centos xxxuseer]# ip route add 10.244.0.0/16 dev cni0 src 宿主机真实外网IP table e0
[root@centos xxxuseer]# kubectl exec test-pod-new -- ping -c 1  -t 1 宿主机真实外网IP
PING 宿主机真实外网IP (宿主机真实外网IP): 56 data bytes
64 bytes from 宿主机真实外网IP: seq=0 ttl=64 time=0.206 ms

--- 宿主机真实外网IP ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.206/0.206/0.206 ms
[root@centos xxxuseer]# kubectl exec test-pod-new -- wget -O /tmp/v1.txt --timeout=1 --no-check-certificate https://10.96.0.1/version
Connecting to 10.96.0.1 (10.96.0.1:443)
saving to '/tmp/v1.txt'
v1.txt               100% |********************************|   263  0:00:00 ETA
'/tmp/v1.txt' saved
[root@centos xxxuseer]# 

[root@centos xxxuseer]# kubectl get pod -o wide -A
NAMESPACE      NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
default        test-pod-new                        1/1     Running   0          15h   10.244.0.93      centos   <none>           <none>
kube-flannel   kube-flannel-ds-46s9b               1/1     Running   0          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    coredns-66f779496c-xv5jv            1/1     Running   0          15h   10.244.0.91      centos   <none>           <none>
kube-system    coredns-66f779496c-z2ftl            1/1     Running   0          15h   10.244.0.92      centos   <none>           <none>
kube-system    etcd-centos                      1/1     Running   2          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-apiserver-centos            1/1     Running   2          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-controller-manager-centos   1/1     Running   11         15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-proxy-b74jv                    1/1     Running   0          15h   宿主机真实外网IP   centos   <none>           <none>
kube-system    kube-scheduler-centos            1/1     Running   12         15h   宿主机真实外网IP   centos   <none>           <none>
[root@centos xxxuseer]# 

原因：

推荐下腾讯的CodeBuddy，免费。能力很强。

虽然一开始它走了弯路，走了好多天弯路。但当我高清firewall，iptables，之后从容器内部出发找到关键问题，给了找问题角度之后，还是它找到问题的原因和解决方法。

补充问题现象：

宿主机外网ip xxxx， 在宿主机上开了防火墙，不能访问docker私有镜像 xxxx:33443，但是能访问127.0.0.1。
k8s，用宿主机外网ip初始化，装了flannel，coredns也不能ready。

当然其实推荐用虚拟网卡装k8s和docker！我只是不喜欢这种方式。然后又不是多机有内网ip。