Using JWT Token in GKE

Background

There are a bunch of applications running in a GKE cluster, one of them (let’s call it Auth Service) is responsible for authentication between users and services (by integrating with the internal AD group). There are two main tasks for the Auth Service:

1. Generate tokens for users
2. Validate tokens and return some basic user information, e.g. user ID, email, roles, etc.

The token is in JSON web token (JWT) format and is signed using the HMAC algorithm by a secret maintained in the Auth Service.

So the existing supported workflow is: User A wants to talk to Service B, so the first thing user A needs to do is to get a token from the Auth Service, then send request to Service B with that token. After receiving the request, Service B needs to talk to Auth Service first to validate the token and get information about the user. Service B could be an automation service that is creating pull requests on behalf of the user automatically, or doing some clean up tasks for that user.

Requirements

Now the team is onboarding a new service, which is running in the background but will talk to other services. For some reasons (mostly security related), it’s not feasible to create a user account in AD for this service. The new service onboarding must satisfy the requirements below:

1. The new service is deployed in the same approach as other services.
2. No impact on other services, so token is still required for communication, but the Auth Service could be changed at minimum to support the use case.
3. No other external services should be involved except for GCP.

JWT Token Basics

Based on the above requirements, we know that if the new service wants to talk to other services, a JWT token should be provided, so that other service could present the token to Auth Service and swap for user information such as a git user name, an email address, etc.

Apparently, without real user account credentials, it’s not possible to get a token from the Auth Service. We have to make a fake token and make some changes in the Auth Service for token verification.

JWT token is easy to create, there is an open standard (RFC 7519) that defines clearly that JWT token is a compact and self-contained way for transmitting information between parties as a JSON object.

Based on the common use cases, JWT token shouldn’t contain any sensitive information such as credentials because the payload in the JWT token is considered public in plain text, although it’s possible to encrypt the token if the verification service is able to decrypt based on the agreed encryption algorithm which is not true for us.

JWT token is still a good way of securely transmitting information because it can be signed, which means you can be sure that the senders are who they say they are. Additionally, the structure of a JWT allows you to verify that the content hasn’t been tampered with. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA. When tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it.

There is a website jwt.io can help to get the details of a JWT token. Just remember to provide the correct secret or public keys if the token is signed, to make sure the token is signature verified.

Generate JWT Token in Kubernetes

Now we know the structure of JWT and how to perform the signature verification, where could we get a token or how could we create a token by ourselves? Especially, how we could find a trusted 3rd party entity so that the Auth Service could retrieve the secret/public key to verify the token signature?

If not considering other external public services, there are 2 entities in my mind that could satisfy the requirement: Kubernetes and Google Cloud.

Our applications are running inside GKE cluster, so Kubernetes API is definitely a trusted entity, Kubernetes even provides an API to allow a client to ask for an X.509 certificate to be issued. At the same time, when talking about JWT token, there is a concept in Kubernetes that sounds familiar to us, the service account (KSA). When creating a new service account (at least until v1.24, after v1.24 please talk to me if you want to know more), Kubernetes will automatically create a JWT token in a secret associated with the service account. Let’s take a look at the token payload for a service account coming from a GKE cluster:

Kubernetes already provides some information about the token, e.g. which pod the token is used for, which service account the token is associated with. The information here could help Kubernetes provide fine-grained RBAC capability.

However, currently there is no way we could customize the payload in the service account token, it’s totally generated by Kubernetes for us, but in order for our services to work, we need to provide some information such as git user name and email address.

So Kubernetes service account token is not an option for us.

Generate JWT Token in Google Cloud

When looking at Google Cloud, JWT token still sounds familiar.

Google Cloud provides the instruction in detail for how to generate JWT token for server to server application communication using the Google service account (GSA) key, which sounds like exactly a perfect solution for our use case.

However, when looking at the steps, I realized in order to generate the token, a GSA private key is necessary which is not possible for our new service. First, it is highly unlikely that the new service has permission to create a GSA. Second, it’s not recommended by Google Cloud to maintain the service account keys especially for an application. Google Cloud recommends using workload identity federation instead of service account keys to eliminate the maintenance and security burden associated with service account keys.

Luckily, workload identity federation has already been available for all the services in our GKE cluster, and it’s automatically configured for the associated Kubernetes service account (KSA) when the service is deployed. Our internal system has made one step further, the GSA of the KSA has been granted the role roles/iam.serviceAccountTokenCreator which is required in the following steps.

With workload identity federation in place, our new service could access Google Cloud API with the roles of the GSA. A short-lived JWT token for a service account could be obtained by calling GCP IAM’s projects.serviceAccounts.signJwt API.

Most importantly, we are able to customize the token payload for our particular use case. Inside our service pod, getting a JWT token is pretty straightforward:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${sa_email}:signJwt" \
    -d '
    {
        "payload": "{\"iss\": \"'$sa_email'\", \"sub\": \"'$sa_email'\", \"aud\": \"https://www.googleapis.com/auth/iam\", \"iat\": '$(date +%s)', \"exp\": '$(($(date +%s)+3600))', \"email\": \"test@example.com\", \"gituser\": \"testUser\"}"
    }
    '

The response is something like this:

{
  "keyId": "38db1200f0c58f9380b94c8b9f66988df6ab9e2e",
  "signedJwt": "xxxxxxx"
}

The signedJwt in the response is the JWT token we need! Putting the token to jwt.io, we could see all those customized fields in the payload.

However, some observant readers may notice that in the bottom left there is an error “Invalid Signature”. Because the JWT token is signed by the service account private key, to validate the token, we need to retrieve the public key.

Do you still remember the response of signJwt API? There is a field named “keyId”, which is exactly the ID of the key Google Cloud used to sign the JWT. The key used for signing will remain valid for at least 12 hours after the JWT is signed. To verify the signature, we could retrieve the public key in several formats from the following endpoints:

• RSA public key wrapped in an X.509 v3 certificate: https://www.googleapis.com/service_accounts/v1/metadata/x509/{ACCOUNT_EMAIL}
• Raw key in JSON format: https://www.googleapis.com/service_accounts/v1/metadata/raw/{ACCOUNT_EMAIL}
• JSON Web Key (JWK): https://www.googleapis.com/service_accounts/v1/metadata/jwk/{ACCOUNT_EMAIL}

Use JWK for an example:

There are multiple keys in the response, but only one of them could match the “keyId” of the signJwt API response. Copy the item (the second JSON item in the “keys” array) containing the keyId to the public key field in jwt.io:

Signature Verified!

Actually, we could even don’t rely on the signJwt API response for the keyId, as it could also be found in the token header. Yes, even the signature is not verified, we could still decode the token payload and explore the fields.

Now the service communication process should be, our new service gets a GSA JWT token from Google Cloud (instead of the Auth Service), then it calls other services with that token. The other services are still calling the Auth Service to validate token, it’s the Auth Service’s job to decode the token and interact with Google Cloud for token verification.

Show Me The Code!

Talk is cheap, show me the code.

In the steps above, we just use jwt.io to encode and decode the token, especially when verifying the signature, jwt.io is doing some “magic” under the hood. Actually, there are lots of libraries/packages out there could deal with token verification in various programming languages. Here I use Java to give you an example, I believe it should be similar for others.

The function below is simple, given a token, it’s getting the payload.

import io.jsonwebtoken.*;

public Claims getClaimsFromGoogleToken(String token) {
    SigningKeyResolver signingKeyResolver = new GoogleSigningKeyResolver();
    return Jwts.parser()
            .setSigningKeyResolver(signingKeyResolver)
            .parseClaimsJws(token)
            .getBody();
}

The GoogleSigningKeyResolver class is the place to verify the signature, the Java package I’m using provides a method could be overwritten in which the token header is available (we need the keyId to get the service account public key, right?).

import io.jsonwebtoken.*;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.math.BigInteger;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.*;
import java.security.spec.RSAPublicKeySpec;
import java.util.Base64;
import org.json.*;

public class GoogleSigningKeyResolver extends SigningKeyResolverAdapter {
    @Override
    public Key resolveSigningKey(JwsHeader jwsHeader, Claims claims) {
        String keyId = jwsHeader.getKeyId();
        String keyAlg = jwsHeader.getAlgorithm();
        String gsaEmail = claims.getSubject();

        Key key = lookupVerificationKey(keyId, keyAlg, gsaEmail);
        return key;
    }

    private Key lookupVerificationKey(String keyId, String keyAlg, String gsaEmail) {
        try {
            URL url = new URL("https://www.googleapis.com/service_accounts/v1/metadata/jwk/" + gsaEmail);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestProperty("Content-Type", "application/json");
            con.setRequestMethod("GET");

            int responseCode = con.getResponseCode();
            if (responseCode != 200 ) {
                throw new RuntimeException("Failed to get GSA JSON Web Key");
            }

            BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuffer response = new StringBuffer();
            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
            in.close();

            JSONObject keysResponse = new JSONObject(response.toString());
            JSONArray keys = keysResponse.getJSONArray("keys");
            SignatureAlgorithm alg = SignatureAlgorithm.forName(keyAlg);
            for (int i=0; i<keys.length(); i++) {
                JSONObject jObj = keys.getJSONObject(i);
                if (jObj.getString("kid").equals(keyId)) {
                    BigInteger modulus = new BigInteger(1, Base64.getUrlDecoder().decode(jObj.getString("n")));
                    BigInteger exponent = new BigInteger(1, Base64.getUrlDecoder().decode(jObj.getString("e")));
                    return KeyFactory.getInstance("RSA").generatePublic(new RSAPublicKeySpec(modulus, exponent));
                }
            }
        } catch (Exception ex) {
            throw new RuntimeException("Failed to look up the GSA JSON Web Key");
        }

        throw new RuntimeException("GSA JSON Web Key not found");
    }
}

The token verification code is not fun, I believe some other packages could provide much easier and more convenient way for that.

Summary

JWT token is used everywhere because of its small size and self contained payload, also designed with security in mind. It’s quite popular for authentication and authorization purposes. Maybe next time when you are making requests to an API with the header Authorization: Bearer <token>, paste the token to jwt.io and see what’s inside :-)