Deleting a Hung Namespace in Kubernetes
I've been working a bit with Kubeflow the past few weeks and getting it installed proved challenging. While working through the install I deleted various namespaces and pods while troubleshooting and ran across a situation where sometimes my namespaces would hang after deletion using kubectl delete namespace <namespace>
with a state of Terminating
. The rest of this post talks about how to move past that issue.
Note: this post originated from this great post by Craig Newton. I changed to reflect Kubeflow and posted here so I have a quick place to find it next time I need it.
First you have to get dump the namespace config into a json file:
kubectl get namespace <namespace> -o json > <namespace>.json
You should see something like:
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2020-01-14T21:05:16Z",
"labels": {
"control-plane": "kubeflow",
"katib-metricscollector-injection": "enabled"
},
"name": "kubeflow",
"resourceVersion": "884821",
"selfLink": "/api/v1/namespaces/kubeflow",
"uid": "28e3078b-ce62-425e-8145-a0606c028173"
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"phase": "Active"
}
}
You want to open the file for editing and remove "kubernetes"
from the finalizers
array:
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2020-01-14T21:05:16Z",
"labels": {
"control-plane": "kubeflow",
"katib-metricscollector-injection": "enabled"
},
"name": "kubeflow",
"resourceVersion": "884821",
"selfLink": "/api/v1/namespaces/kubeflow",
"uid": "28e3078b-ce62-425e-8145-a0606c028173"
},
"spec": {
"finalizers": [
]
},
"status": {
"phase": "Active"
}
}
Next you'll execute your cleanup command:
kubectl replace --raw "/api/v1/namespaces/<namespace>/finalize" -f ./<namespace>.json
Check to ensure that the hung namespace is now removed from your cluster:
kubectl get namespaces -all-namespaces