JWT security
Eureka’s scheduler, login node, and compute nodes authenticate to the control service with a short-lived token. This is separate from browser sign-in. Browser sign-in is Sign-in.
Each node has its own shared secret. The node signs tokens with it. The control host keeps a matching copy and uses that copy to check the signature. A token from one node cannot be checked with another node’s secret.
What a token contains
Section titled “What a token contains”A token is an HS256 JWT:
| Claim | Meaning |
|---|---|
sub |
The Unix user who asked for the token. Root can ask for another user. Anyone else gets their own name. |
iat |
When it was created. |
exp |
Ten minutes after iat. |
aud |
eureka-afw. |
iss |
The node name in /etc/eureka-personality, normally the node’s fully qualified domain name. |
There is no fallback issuer named control. A missing or unusable personality file mints no token.
Tokens have no unique id, so a captured token works until it expires. Do not print tokens, put them in command arguments, paste them into tickets, or write them into logs. Clients request a new token for each call.
The control service allows about 120 seconds of clock skew so a VM that was paused and resumed can still authenticate while time sync catches up. A larger lag fails as expired until the clock is corrected.
Who a node may act as
Section titled “Who a node may act as”| Issuer | How it is recognized | Subjects it may use |
|---|---|---|
| Control | The issuer matches the control host’s own personality file | root and omnisched for any request, and a person |
| Login node | A class file on the control host says login |
Scheduler operations as root or omnisched, a person, and the sign-in records |
| Compute node | A class file says compute |
Scheduler operations only. A person token is refused. |
A missing class is refused. Scheduler operations are the reads and writes a node needs to start and report work. Writes to people, projects, permissions, launchers, flows, credentials, and providers are accepted from a scheduler subject only on the control host.
A compute node may update only the job records of its own instance group, and only the ready flag and the reason. Other job writes from a node are denied.
On a node, /var/eurekakey is the signing secret (root:root, mode 0600) and /etc/eureka-personality is the issuer name (root:root, mode 0644).
On the control host, /var/eurekakeys/ is mode 0700. One file per issuer, mode 0600, is the verifier. Class files live in /var/eurekakeys/.class/ and say login or compute. They are removed with the verifier when the node stops.
The secret is 32 random bytes written as 64 hexadecimal characters. The text is the key. It is not decoded to binary. These are shared secrets, not a public and private key pair. Anyone who can read one can sign tokens for that issuer. Anyone who can read the control host can read every verifier.
The guest secret is installed over SSH standard input, not in the remote command. The verifier is published on the control host only after the node has the secret. A node that never installs it is left out, and its verifier is removed so its tokens are not accepted. If no node in a group registers, the start fails and the cloud group is deleted. The machine also leaves Environment overview.
Stop, prune, and revoke
Section titled “Stop, prune, and revoke”A normal stop removes that node’s verifier, class, and group files on the control host, including when the node cannot be reached. The node may still have its signing secret, but the control host will no longer accept tokens signed with it.
A VM deleted by hand can leave a verifier behind. The control host then still accepts tokens from that secret. On the control host, as root:
sudo eureka keys prunesudo eureka keys prune --yesprune lists verifiers whose node no longer exists. It keeps the control host’s own issuer and every verifier that still matches a Node row. It refuses to run when the node list comes back empty, so a control outage is not treated as “every key is stale”. --yes removes the stale files. --json prints the same result as JSON. The command does not print secret contents.
To revoke one issuer immediately, remove /var/eurekakeys/<exact-issuer> on the control host. Check the name first. That rejects every token from that node. Use the normal stop and start to rotate a key. Restarting a process does not rotate it.
If a node secret or a bearer token may have leaked:
- Remove that issuer’s verifier on the control host.
- Stop or isolate the node.
- Read the control logs for that issuer. Do not copy bearer tokens into the incident notes.
- Start the node again so it gets a new secret.
If the control host is compromised, treat every verifier as compromised. Do not restore verifier files from that host.
Checks that do not print secrets
Section titled “Checks that do not print secrets”On a node:
sudo stat -c '%a %U:%G %n' /var/eurekakey /etc/eureka-personalitycat /etc/eureka-personalityhostname -fsudo test -s /var/eurekakeyThe personality should be the domain name the control host expects. test -s only checks that the secret file is non-empty.
On the control host:
sudo stat -c '%a %U:%G %n' /var/eurekakeyssudo find /var/eurekakeys -mindepth 1 -maxdepth 1 -type f -printf '%m %u:%g %f\n'Expect one verifier for the control host and one for each live node.
To confirm logs are not recording tokens, count matches and do not print the lines:
sudo journalctl -u afwfcgi --since today | grep -Ec 'jwtToken:|_payloadStr:|jwtData:|Bearer eyJ'A count of zero is what you want. grep exits 1 when there are no matches.
When a token is refused
Section titled “When a token is refused”- JWT has expired or JWT iat is in the future: compare
date -uon the node and on the control host. Fix time sync if they differ by more than 120 seconds. - scheduler identity is not allowed from this issuer or person tokens are accepted only from control or login issuers: the node class does not allow that subject. A person token from a compute node is refused on purpose.
- missing or invalid JWT issuer: the personality file is missing, empty, or not a single domain name. Do not repair a mismatch by copying the secret into an environment variable or a command line.
- Missing verifier or a signature failure: the control host has no
/var/eurekakeys/<exact-personality>, or the two copies are not from the same start. Stop and start the node so both copies are replaced together.