When GKE Ran Out of Certificates: Debugging the 15-ManagedCertificate Ceiling

I was deploying changes to a GKE cluster that hosts several client-facing services behind a single ingress, when the sync just… stopped working! No obvious code change had broken anything. The ingress events told a different story than I expected:

At most 15 SSL certificate(s) (17 in the request)

That one line sent me down a debugging rabbit hole that taught me more about how GCP’s load balancers actually work than any doc page had up to that point. Here’s what I found.

The setup

The cluster in question is managed with a mix of Helm and Terraform, which is a fairly common pattern once infrastructure grows past “one team, one repo.” Multiple hostnames were routed through a single ingress (and, underneath it, a single target-https-proxy), and historically, every service got its own ManagedCertificate custom resource. One service, one cert. Simple to reason about, easy to template in a Helm chart.

Until it wasn’t.

The core problem: a limit Kubernetes doesn’t know about

Here’s the part that tripped me up initially: ManagedCertificate is a Kubernetes CRD, so you can create as many of them as you like in-cluster. Nothing in the Kubernetes API stops you. The limit isn’t Kubernetes-side at all — it’s enforced by GCP itself, on the load balancer’s target-https-proxy. GCP allows a hard maximum of 15 SSL certificates per proxy, full stop.

Think of it like a filing cabinet with exactly 15 slots. You can print as many folders as you want at your desk, but the cabinet itself won’t take a 16th. Kubernetes is happily printing folders; GCP is the one holding the cabinet, and it doesn’t warn you until you actually try to file something and there’s no room left.

That gap — between what your manifests describe and what the cloud provider will actually let you attach — is really the whole story of this incident.

Diagnosing it

A few things made this harder to untangle than “just delete some certs”:

Ghost references. Digging through the namespace, I found several stale ManagedCertificate objects left behind by earlier Helm rollbacks. They weren’t doing anything useful anymore, but GCP was still counting them toward the proxy’s cert request.

A proxy that had drifted from reality. At some point the target-https-proxy itself had been recreated. Comparing creation timestamps on the proxy against the timestamps on the individual certs made it clear: only a small fraction of the certs marked Active in the namespace were actually attached to the new proxy. Everything else was stale bookkeeping. This is a useful technique in general — when you’re missing direct visibility into an out-of-band infrastructure event, timestamps are often the closest thing you’ll get to a paper trail.

An annotation you can’t just patch. The ingress carries a pre-shared-cert annotation that’s fully owned and managed by the GKE ingress controller. My first instinct was to edit it by hand to force things back into sync — that doesn’t work, and trying will just get overwritten on the next reconcile loop.

A separate, contradictory config bug that showed up during mitigation. One host was set with allow-http: false alongside redirectToHttps: true. Individually, both settings sound reasonable. Together, they don’t make sense — instead of an HTTP→HTTPS redirect, connections were being refused outright. It’s a good reminder that two settings can each pass a mental “does this look right?” check and still combine into something broken.

Why it got worse before it got better

The first mitigation attempt was straightforward: remove a couple of ingress host rules to bring the count back under 15. It didn’t work — or rather, it did reduce the desired state, but the requested cert count on GCP’s side actually climbed, from 17 to 19, while the proxy’s attached-cert count lagged behind at 14.

This is the part I’d flag as the real lesson of the incident. The target-https-proxy sync is asynchronous and eventually consistent, not transactional. A clean, correct-looking change to your manifests doesn’t guarantee GCP’s actual state updates in lockstep. There’s a window where things can look worse than before you touched anything, purely because the visible state hasn’t caught up to the desired state yet. If you don’t know that going in, it’s easy to panic and start making a second and third change on top of the first, which usually makes the eventual reconciliation messier, not cleaner.

Getting out of the hole

A few directions were on the table for actually fixing this, rather than just patching around it:

• Consolidation — batching multiple hostnames onto fewer ManagedCertificate objects instead of the one-cert-per-service default.
• Wildcard certificates — covering multiple subdomains under a single cert, which structurally reduces the count regardless of how many services you add later.
• GCP Certificate Manager — a newer alternative to the classic per-ingress ManagedCertificate model, worth evaluating separately for how it handles the cap.
• Fixing it at the source — the shared Helm chart was defaulting to one ManagedCertificate per service. That default was the actual root cause; every other fix here is downstream of it. Changing the chart itself stops the count from climbing back toward the ceiling the next time someone adds a service.

The takeaway

The satisfying fix here isn’t any single command — it’s noticing that the problem lived at a layer Kubernetes doesn’t have visibility into. CRDs describe desired state; they don’t know anything about a cloud provider’s enforced limits, quotas, or caps. Nothing warns you until the sync fails. That gap between “what my manifests say” and “what the cloud will actually let me have” is worth watching for anytime you’re operating at the edge of a platform’s defaults — and it’s a good example of why understanding the underlying cloud resources, not just the Kubernetes abstractions sitting on top of them, still matters.

Progress over perfection — this one didn’t get solved in a single pass, and that’s fine.