The plus in the secret

unauthorized_client.

That is the whole error. No field, no hint, no indication of which half of the exchange Keycloak was unhappy about. The client ID was right, it existed, it was enabled, and the secret I had pasted into the application was the secret Keycloak was showing me. I had checked. Twice.

What this was

A self-hosted Paperless instance behind Keycloak for OIDC. Nothing exotic. The kind of integration you do on a Sunday afternoon and expect to take twenty minutes.

I had rotated the client secret earlier that day as part of a general tidy-up, regenerated it in the Keycloak admin console, copied it into the application's config, restarted, and gone to log in.

unauthorized_client.

Ruling out the obvious

First instinct: I fat-fingered the paste. So I did the only check that actually settles that question, which is to hash both sides rather than look at them.

# in Keycloak's admin console, copied to a file, no trailing newline
printf '%s' "$secret_from_keycloak" | sha256sum | cut -c1-16
# and on the application host
printf '%s' "$secret_in_config" | sha256sum | cut -c1-16

Same sixteen characters. Same length. The secret was byte-identical on both sides of the connection.

That is the point where the interesting part starts, because it eliminates the entire category of explanation I had been carrying around. It was not a typo, not a trailing newline, not a shell-eaten character on the way into the config file. The two strings were the same string.

Ruling out the next obvious thing

So it must be the client configuration. I went through it.

Client authentication on. Standard flow on. Valid redirect URIs matching, including the trailing path. Client not set to public. Service accounts irrelevant here. Realm correct. Clock skew between the two hosts under a second.

I recreated the client from scratch with the same settings. Same error.

At this point I had spent about forty minutes on a twenty-minute job, and I had eliminated both the secret and the configuration, which between them are supposed to be the whole problem.

Asking a different question

When the value is right and the configuration is right, the thing left is the transport. So I stopped asking "is the secret correct" and started asking "what arrives at the other end".

There is a specific test for this, and it is the most useful debugging move I know for auth problems. Send the same token request three ways and compare what comes back:

# 1. secret in the body, url-encoded
curl -s -o /dev/null -w '%{http_code}\n' \
  -d grant_type=client_credentials \
  --data-urlencode "client_id=$ID" \
  --data-urlencode "client_secret=$SECRET" \
  "$TOKEN_URL"

# 2. secret in the body, raw
curl -s -o /dev/null -w '%{http_code}\n' \
  -d "grant_type=client_credentials&client_id=$ID&client_secret=$SECRET" \
  "$TOKEN_URL"

# 3. secret via HTTP Basic
curl -s -o /dev/null -w '%{http_code}\n' \
  -u "$ID:$SECRET" \
  -d grant_type=client_credentials \
  "$TOKEN_URL"

If all three agree, the secret is wrong. If they disagree, the secret is fine and something in the encoding is not.

I got 200, 401, 200.

Three requests, same secret, two different answers. The secret was not wrong. The middle one was.

The character

The difference between request 1 and request 2 is that --data-urlencode escapes the value and -d does not. So what in this secret needed escaping?

gK7+xQ2mLp...

A plus sign.

In an application/x-www-form-urlencoded body, + means space. It is not an escape, it is not a special case, it is the definition of the format: a literal plus has to be sent as %2B, and a bare + decodes to U+0020 on arrival. Keycloak was receiving a secret with a space where my plus was, comparing it against the stored hash, and correctly refusing.

The application was building its token request without encoding the secret. Byte-identical at rest on both sides, different by one character in flight.

The fix, and the better fix

The immediate fix took ten seconds: regenerate the secret until it has no + in it.

The real fix is to stop generating machine secrets from an alphabet that contains characters other layers care about. A secret travels through a form body, a URL, a shell, a YAML file, a systemd unit and a Go template before anything validates it, and each of those assigns meaning to a different set of characters. Length is free for a machine. Alphabet is not.

So machine secrets here are now 40 characters of A-Za-z0-9 and nothing else. That is 238 bits, which is far past anything that matters, and it cannot be misread by any layer it passes through.

mkpw -a          # 40 chars, A-Za-z0-9

The characters I now refuse, and where each one bites:

-_.~ are technically URL-safe. They are on the list anyway, because "no exceptions" is a rule you cannot misremember at eleven at night.

What I have not fixed

Four of the eight OIDC clients in that realm still have secrets from the old generator, because rotating a secret means a coordinated restart and I have not scheduled it. They work. They will keep working until one of them ends up in a code path that encodes differently, and then I will lose another hour to a one-word error message.

I am writing that down here partly so it stays visible.

The general shape of it

The lesson is not about OIDC and it is not really about Keycloak, which behaved correctly throughout and had no way to tell me anything more useful.

It is that when a credential is provably correct and authentication still fails, the fault is in the shape, not the value. Compare hashes to settle the value. Then send the same request three ways and watch where the answers diverge. Divergence is an encoding bug, and encoding bugs never name themselves in the error message.