[摘要] kubernetes 可能会产生垃圾或者僵尸pod，在删除rc的时候，相应的pod没有被删除，手动删除pod后会自动重新创建，这时一般需要先删除掉相关联的resources，实际中还要具体情况具体分析。
在使用Helm部署gitlab的时候发现有垃圾的pod存在，一直删除不掉
# kubectl get pods -n kube-systemNAME ???????????????????????????READY ???STATUS ???RESTARTS ??AGEgitlab-migrations.1-2km-4rwnm ?????1/1 ??Running ??????158 ???21hgitlab-migrations.1-30d-gx9nz ?????1/1 ??Running ??????164 ???21hgitlab-migrations.1-dsw-m5qbc ?????1/1 ??Running ??????167 ???21h
一般来说是要删除pod关联的rc，rs以及deployment等等，可以执行如下命令查询
kubectl get ClusterRole -n kube-systemkubectl get ClusterRoleBinding -n kube-systemkubectl get Deployments -n kube-systemkubectl get Pods -n kube-systemkubectl get ServiceAccounts -n kube-systemkubectl get CustomResourceDefinition -n kube-systemkubectl get replicasets -n kube-systemkubectl get replicationcontrollers -n kube-systemkubectl get services -n kube-system
找到对应的标签进行删除，对应的pod一般就会被删除掉，如果pod还在的话再手动删除pod。
实际中我这三个pod还是无法删除掉，删除之后仍然会重新启动，所以仔细看了下pod的具体描述内容。
# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nzName: ??????????gitlab-migrations.1-30d-gx9nzNamespace: ?????kube-systemNode: ??????????172.30.0.169/172.30.0.169Start Time: ????Mon, 26 Mar 2018 11:53:45 +0800Labels: ????????app=migrations ???????????????controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209 ???????????????job-name=gitlab-migrations.1-30d ???????????????release=gitlabAnnotations: ???<none>Status: ????????RunningIP: ????????????172.100.14.7Controlled By: ?Job/gitlab-migrations.1-30dInit Containers: ?configure: ???Container ID: ?docker://a3051d24e65267489aa97ae368bcbe5819e4ae4f3b5ddc4d277b0b25ee585716 ???Image: ????????busybox ???Image ID: ?????docker-pullable://docker.io/busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21 ???Port: ?????????<none> ???Command: ?????sh ?????/config/configure ???State: ?????????Terminated ?????Reason: ??????Completed ?????Exit Code: ???0 ?????Started: ?????Mon, 26 Mar 2018 11:54:24 +0800 ?????Finished: ????Mon, 26 Mar 2018 11:54:24 +0800 ???Ready: ?????????True ???Restart Count: ?0 ???Environment: ???<none> ???Mounts: ?????/config from migrations-config (ro) ?????/init-config from init-migrations-secrets (ro) ?????/init-secrets from migrations-secrets (rw) ?????/var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)Containers: ?migrations: ???Container ID: ?docker://c33e809de76ba44da330063d9e3c6f58631b30e3c805ea561dc68af38ffe3926 ???Image: ????????registry.gitlab.com/gitlab-org/build/cng/gitlab-rails:latest ???Image ID: ?????docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-rails@sha256:7dca5eb3b686342a9cff798f8349ad07459a1ef1e9d47a23b787b284d874cc17 ???Port: ?????????<none> ???Args: ?????/scripts/wait-for-deps ?????/scripts/db-migrate ???State: ?????????Running ?????Started: ?????Tue, 27 Mar 2018 09:50:09 +0800 ???Last State: ????Terminated ?????Reason: ??????Error ?????Exit Code: ???1 ?????Started: ?????Tue, 27 Mar 2018 09:41:52 +0800 ?????Finished: ????Tue, 27 Mar 2018 09:45:02 +0800 ???Ready: ?????????True ???Restart Count: ?164 ???Environment: ?????CONFIG_TEMPLATE_DIRECTORY: ?/var/opt/gitlab/templates ?????CONFIG_DIRECTORY: ??????????/var/opt/gitlab/config/gitlab/ ?????GITLAB_ROOT_PASSWORD: ??????Vi4ree2R ???Mounts: ?????/etc/gitlab from migrations-secrets (ro) ?????/home/git/gitlab/config/secrets.yml from migrations-secrets (rw) ?????/var/opt/gitlab/templates from migrations-config (rw) ?????/var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)Conditions: ?Type ??????????Status ?Initialized ???True ?Ready ?????????True ?PodScheduled ??TrueVolumes: ?migrations-config: ???Type: ?????ConfigMap (a volume populated by a ConfigMap) ???Name: ?????gitlab-migrations ???Optional: ?false ?init-migrations-secrets: ?<unknown> ?migrations-secrets: ???Type: ???EmptyDir (a temporary directory that shares a pod‘s lifetime) ???Medium: ?Memory ?default-token-6rdp2: ???Type: ???????Secret (a volume populated by a Secret) ???SecretName: ?default-token-6rdp2 ???Optional: ???falseQoS Class: ??????BestEffortNode-Selectors: ?<none>Tolerations: ????<none>Events: ?Type ????Reason ??????Age ?????????????????From ??????????????????Message ?---- ????------ ??????---- ????????????????---- ??????????????????------- ?Warning ?FailedMount ?6m (x655 over 21h) ??kubelet, 172.30.0.169 ?MountVolume.SetUp failed for volume "migrations-config" : configmaps "gitlab-migrations" not found ?Warning ?BackOff ?????3m (x3516 over 21h) ?kubelet, 172.30.0.169 ?Back-off restarting failed container
需要关注的地方是Labels标签
Labels: ????????app=migrations ???????????????controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209 ???????????????job-name=gitlab-migrations.1-30d ???????????????release=gitlab
发现有个job-name=gitlab-migrations.xxx
于是继续查看kubernetes是否有job存在
# kubectl get jobs -n kube-systemNAME ?????????????????????DESIRED ??SUCCESSFUL ??AGEgitlab-migrations.1-2km ??1 ????????0 ???????????23hgitlab-migrations.1-30d ??1 ????????0 ???????????23hgitlab-migrations.1-dsw ??1 ????????0 ???????????23h
先执行删除job看下：
# kubectl delete job gitlab-migrations.1-2km -n kube-systemjob "gitlab-migrations.1-2km" deleted# kubectl delete job gitlab-migrations.1-30d -n kube-systemjob "gitlab-migrations.1-30d" deleted# kubectl delete job gitlab-migrations.1-dsw -n kube-systemjob "gitlab-migrations.1-dsw" deleted
再看pod情况：
# kubectl get pods -n kube-systemNAME ??????????????????????????????READY STATUS ????RESTARTS ?AGEgitlab-migrations.1-30d-gx9nz ?????1/1 ??Terminating ??173 ???23hgitlab-migrations.1-dsw-m5qbc ?????0/1 ??Terminating ??176 ???23h
附图为kubernets 的一些对象资源：
再看pod已经被删除了，还有处于Terminating状态的，一会就会消失掉。至此pod删除成功，所以删除与pods相关联的依赖资源是解决问题的最终方法！
kubernetes 无法删除 pod 问题的解决
原文地址：https://www.cnblogs.com/weifeng1463/p/9970895.html