Zero Trust Security in Practice: Implementing Keycloak with Microservices

"Zero trust" gets used as a marketing term often enough that it's worth being specific about what it actually means at the level of a request between two of your own services: never trust a caller because of where it's calling from (the internal network, a known IP range, a service mesh sidecar) — verify every call cryptographically, every time, regardless of origin. In practice, for a microservices architecture, that means every service-to-service call carries a token, and every service that receives a call verifies it — not just the ones facing the public internet.

Here's how we actually implement that with Keycloak, and where teams commonly get it wrong.

Client credentials: the workhorse grant type

User-facing auth uses the authorization code flow — a human logs in, gets redirected, ends up with a token. Service-to-service auth uses a different OAuth2 grant entirely: client credentials. Each service is registered in Keycloak as a confidential client with its own client ID and secret (or, better, mutual TLS client authentication instead of a shared secret — more on that below), and when Service A needs to call Service B, it first exchanges its client credentials for an access token scoped to that purpose, then attaches it as a bearer token on the call.

The mistake we see most often here is one shared "internal services" client used by everything, which defeats most of the point — if every service has the same credentials, you can't tell which service actually made a call from an audit log, and revoking one compromised service means rotating credentials for all of them. Give every service its own client registration. It's more Terraform, and it's worth it the first time you need to answer "which service made this call" during an incident.

Realm and role design

We default to one realm per environment (dev/staging/production), not one realm per service — a realm is a strong isolation boundary, and splitting realms per service means duplicating IdP federation config, themes, and policy N times for no real security benefit at that granularity. Within a realm, use client roles (roles scoped to a specific client, like orders-service:read-inventory) rather than realm roles (global roles) for service-to-service permissions — this keeps a compromised service's blast radius limited to exactly the permissions it was granted, not a broad realm-wide role that happens to also grant it things it never needed.

A pattern worth calling out explicitly: don't reuse the same roles for human users and services. A role like admin that both a human support engineer and an automated billing service hold makes token scope audits much harder to reason about — you can no longer look at a token's roles and know whether you're looking at a person or a process.

Token validation without a bottleneck

Every service that receives a token needs to validate it — signature, expiry, issuer, and audience (make sure a token minted for Service C can't be replayed against Service D just because it's structurally valid; check the aud claim). The naive approach — calling Keycloak's introspection endpoint on every request — turns your identity provider into a single point of failure and a latency bottleneck under load.

The fix is JWT signature validation done locally: fetch Keycloak's JWKS (JSON Web Key Set) endpoint once, cache the public keys, and validate token signatures in-process using a standard JWT library — no network call to Keycloak per request. This is the entire reason Keycloak issues signed JWTs rather than opaque tokens for this use case. Cache the JWKS with a sane TTL (we typically use 10–15 minutes) and handle key rotation gracefully — Keycloak supports multiple active signing keys specifically so you can rotate without a coordinated flag-day across every service.

A token that requires a network call to validate isn't zero trust — it's a single point of trust with extra steps.

mTLS as the layer underneath

Token-based auth answers "is this caller authorized to do this," but it doesn't by itself answer "is this actually the service it claims to be" at the transport layer — a token can be stolen and replayed by anything that can reach the network. That's what mutual TLS is for, and the two are complementary, not competing: mTLS (commonly via a service mesh sidecar like Istio or Linkerd) establishes that the calling pod is who it claims to be at the network layer, and the Keycloak-issued token establishes what that specific service is authorized to do. Running both means a stolen token alone isn't enough to make a valid call, and a compromised network position alone isn't enough either.

Token lifetime: the trade-off nobody wants to tune

Short-lived access tokens (we typically use 5–10 minutes for service-to-service tokens, shorter than the 15–30 minutes that's common for user-facing tokens) limit the damage window if one leaks, but they mean more frequent token refresh calls — which, at high request volume, adds real load to Keycloak's token endpoint. The mitigation is caching: a service caches its own client-credentials token and only re-requests when it's within a buffer window of expiry, rather than fetching a fresh token per outbound call. Get this wrong in either direction and you'll find out the hard way — either tokens live long enough that a leak matters, or your services are hammering Keycloak's token endpoint hard enough to become the actual bottleneck in the system.

Zero trust, done properly at the microservices layer, isn't a single product decision — it's this specific set of defaults (per-service client credentials, scoped client roles, local JWT validation, mTLS underneath, short-lived cached tokens) applied consistently across every service, including the ones that only ever talk to other internal services and never see a request from outside the cluster.