How a Simple ETL Job Turned Into a 5-Hour Kubernetes DNS Nightmare

"It was just supposed to be a simple ETL microservice. Fetch some data, transform it, and load it into Postgres. 30 minutes, tops."

— Famous last words before a 5-hour DNS debugging marathon

The Setup That Should Have Been Simple

It's a regular Tuesday afternoon, and I'm building what I thought would be a straightforward microservices setup in Kubernetes. Simple enough, right? A data loader microservice that fetches user data from an API, transforms it, and loads it into a PostgreSQL database. ConfigMaps for non-sensitive configs, Secrets for credentials, and textbook Kubernetes stuff.

Everything Was Going Fine... Until It Wasn't

The architecture looked clean on paper:

• Data Loader Microservice: Python app containerized and ready to roll

• PostgreSQL Database: Running in its own pod with persistent storage

• ConfigMaps: Handling API URLs, log levels, DB port, DB name

• Secrets: Safeguarding DB credentials (properly base64 encoded, thank you very much)

I built my Docker image, configured my deployments, and applied my ConfigMaps and Secrets. The PostgreSQL pod started up beautifully. Green lights everywhere.

Then I deployed the ETL microservice and... 💥

Me: "That's odd. Let me check the logs..."

ERROR:root:Error: could not translate host name 'postgres-service' to address: 
Temporary failure in name resolution

The DNS Mystery Deepens

Wait, what? The service is right there! I can see it in kubectl get svc. The name is correct. The port is correct. What's happening?

I tried the classic debugging move, checking if it's an external network issue:

kubectl logs etl-deployment-7cd7bd469b-7dqfl
...
Failed to resolve 'random-data-api.com': Temporary failure in name resolution

Oh no. It's not just the internal service. DNS is completely broken.

Time to investigate CoreDNS:

kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS             RESTARTS
coredns-668d6bf9bc-8xg47   0/1     CrashLoopBackOff   142

There it was. My DNS server was having an existential crisis, trapped in an infinite CrashLoopBackOff.

Down the Troubleshooting Rabbit Hole

Attempt #1: "Just Check the Logs"

kubectl logs -n kube-system coredns-668d6bf9bc-8xg47
Listen: listen tcp :53: bind: permission denied

Ah! A permission error. CoreDNS can't bind to port 53. Easy fix; just add the `NET_BIND_SERVICE` capability!

I edited the CoreDNS deployment:

securityContext:
  capabilities:
    add:
      - NET_BIND_SERVICE

Surely this would work... right?

Narrator: It did not work.

Attempt #2: "Maybe it needs allowPrivilegeEscalation?"

Added allowPrivilegeEscalation: true to the security context. Restarted CoreDNS.

kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS             RESTARTS
coredns-668d6bf9bc-tsn9r   0/1     CrashLoopBackOff   12

Still crashing. Still the same error.

Me, increasingly frustrated: "But `NET_BIND_SERVICE` is literally right there in the deployment file!"

Attempt #3: "Let's Try a Different Port!"

Okay, if port 53 is causing problems because it's a privileged port (ports < 1024), let's just use a non-privileged port instead!

I edited the CoreDNS ConfigMap and changed the port to 1053:

data:
  Corefile: |
    .:1053 {
      log
      errors
      health {
        lameduck 5s
      }
      ready
      kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
      }
      prometheus :9153
      forward . /etc/resolv.conf {
        max_concurrent 1000
      }
    }

Restarted CoreDNS:

kubectl -n kube-system delete pod -l k8s-app=kube-dns

AND IT WORKED! CoreDNS was finally running!

kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS
coredns-698f4cc5c9-xflhm   1/1     Running   0

Me: "YES! We did it! DNS is working now!"

But then I tested my ETL pod...

kubectl logs etl-deployment-7cd7bd469b-7dqfl
ERROR:root:Error: could not translate host name 'postgres-service' to address: 
Temporary failure in name resolution

Me: "WHAT? How is it STILL broken?"

The problem? CoreDNS was now listening on port 1053, but all the pods were still configured to use port 53 for DNS lookups. I'd essentially moved the DNS server to a different address without telling anyone where it went.

The Nuclear Option: Full Minikube Reset

At this point, I was ready to throw my laptop out the window. Instead, I did the next best thing, nuked everything, and started fresh:

minikube delete
rm -rf ./minikube  # Clean slate, baby!
minikube start

Fresh install, clean DNS configuration, port 53 as intended. Everything should work now.

But still, the Same error. Same CrashLoopBackOff. Same "permission denied" message.

Me, on the verge of tears: "WHY?! It's a fresh install! What is happening?!"

The Eureka Moment (Finally!)

After hours of debugging, countless chats with ChatGPT, Claude Code, and Gemini, and more coffee than is probably healthy, I finally understood the root cause:

Linux containers run as non-root users by default for security.

Port 53 is a privileged port that requires root access to bind to. Even with the `NET_BIND_SERVICE` capability, even with `allowPrivilegeEscalation: true`, CoreDNS running as a non-root user simply couldn't bind to port 53.

The solution? Allow CoreDNS to run as root (in a controlled way):

kubectl patch deployment coredns -n kube-system \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext/runAsUser", "value": 0}]'

This sets `runAsUser: 0`, telling the container to run as root (UID 0).

I also enabled host networking to give CoreDNS direct access to the host's network namespace:

kubectl patch deployment coredns -n kube-system \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/hostNetwork", "value": true}]'

Restarted CoreDNS one more time:

kubectl -n kube-system delete pod -l k8s-app=kube-dns

And then... silence. No crash. No errors.

kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS
coredns-698f4cc5c9-xflhm   1/1     Running   0

I nervously checked the logs:

kubectl logs -n kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration SHA512 = b141a95c4ad45582b9d8d1be55241970...
CoreDNS-1.11.3
linux/amd64, go1.21.11, a6338e9
[INFO] 127.0.0.1:54374 - 17618 "HINFO IN 1085771236002924139.3914264802840083309...

It was actually working! Time to test DNS resolution:

kubectl run -it --rm dns-test --image=busybox:1.28 --restart=Never -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

✅ Internal DNS working!

kubectl run -it --rm dns-test --image=busybox:1.28 --restart=Never -- nslookup postgres-service
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      postgres-service
Address 1: 10.98.33.166 postgres-service.default.svc.cluster.local

✅ Service discovery working!

kubectl run -it --rm dns-test --image=busybox:1.28 --restart=Never -- nslookup random-data-api.com
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      random-data-api.com
Address 1: 67.205.161.199

✅ External DNS working!

Victory at Last

I redeployed my ETL microservice and watched the logs with bated breath:

THE ETL JOB COMPLETED SUCCESSFULLY!

Wait, But Why Didn't Port 1053 Work?

You might be wondering: "Port 1053 is non-privileged. CoreDNS was running. Why didn't it work?"

The answer: It's like changing your phone number without updating your contacts.

The DNS Resolution Chain

When a Kubernetes pod resolves a hostname:

1. Pod reads `/etc/resolv.conf` → points to 10.96.0.10:53

2. Query sent to kube-dns Service on port 53

3. Service forwards to CoreDNS on port 53 (defined in targetPort)

4. CoreDNS responds

The problem? When I changed CoreDNS to port 1053, only step 4 changed. Steps 1-3 still expected port 53!

What Actually Needed to Change

Result: CoreDNS was listening on the wrong port. It's like moving apartments without telling the post office.

The Real Insight

Port 1053 solved the permission problem (CoreDNS could bind without root), but created a communication problem (nothing could reach it there).

Port 53 has been the DNS standard since 1987. Changing it means updating every component in the cluster, nodes, services, pods, everything.

The simpler fix? Let CoreDNS run as root (`runAsUser: 0`) and keep port 53. One change instead of cluster-wide reconfiguration.

The Takeaway

Three lessons from 5 hours of debugging:

1. Privileged ports need root access - This is Linux security, not a bug

2. System defaults exist for a reason - Port 53 is everywhere; changing it cascades

3. Simple root cause fix beats complex workarounds - Sometimes "just run as root" is the right answer

Quick Debug Commands

If you hit CoreDNS issues:

# Check CoreDNS status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns

# Test DNS resolution
kubectl run -it --rm dns-test --image=busybox:1.28 --restart=Never -- nslookup kubernetes.default

Remember: Fix DNS first, everything else second.