In this blog post, we will look at the first part of my ideal setup, which is to secure inbound communication via an authenticating reverse proxy (OAuth2_Proxy), and Keycloak.
This setup will use the follow technologies:
- Istio (ingress gateway)
- Certmanager (certificates) - not covered in this post
- OAuth2_Proxy (controls the OIDC flow)
- Redis (session storage)
- Keycloak (OIDC Provider)
Istio is a service mesh that allows you to define and secure services in your Kubernetes cluster. In my lab, I use it as the ingress gateway for my cluster, and I am planning on using it to secure service-to-service communication using mutual-tls.
Istio will require a valid certificate for the gateway, you can either set this up via cert-manager, or by importing a certificate into your cluster manually.
Here are some example config files for setting up basic services in Istio:
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: gateway namespace: istio-system spec: selector: istio: ingressgateway servers: - hosts: - '*' port: name: http number: 80 protocol: HTTP tls: httpsRedirect: true - hosts: - '*' port: name: https number: 443 protocol: HTTPS tls: credentialName: changeme httpsRedirect: false mode: SIMPLE privateKey: sds serverCertificate: sds
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: oauth-corp-justin-tech-com namespace: istio-system spec: host: oauth.corp.justin-tech.com trafficPolicy: loadBalancer: simple: ROUND_ROBIN portLevelSettings: - port: number: 443 tls: mode: SIMPLE sni: oauth.corp.justin-tech.com
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: oauthproxy-service namespace: default spec: gateways: - gateway.istio-system hosts: - oauth.corp.justin-tech.com - oauth.justin-tech.com http: - route: - destination: host: oauthproxy-service port: number: 4180 weight: 100
the virtual service for your application, I am using Rancher-Demo:
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: rancher-demo namespace: default spec: gateways: - gateway.istio-system hosts: - rancher-demo.corp.justin-tech.com http: - route: - destination: host: rancher-demo port: number: 80 weight: 100
The Envoy Filter, which is the part that starts the redirection for unauthenticated users. This is similar to the auth-url function of Nginx-Ingress. This acts on your Ingress-Gateway workload.
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: authn-filter spec: workloadLabels: istio: ingressgateway filters: - filterConfig: http_service: server_uri: uri: http://oauthproxy-service.default.svc.cluster.local cluster: outbound|4180||oauthproxy-service.default.svc.cluster.local timeout: 1.5s authorizationRequest: allowedHeaders: patterns: - exact: "cookie" - exact: "x-forwarded-access-token" - exact: "x-forwarded-user" - exact: "x-forwarded-email" - exact: "authorization" - exact: "x-forwarded-proto" - exact: "proxy-authorization" - exact: "user-agent" - exact: "x-forwarded-host" - exact: "from" - exact: "x-forwarded-for" - exact: "accept" - prefix: "x-forwarded" - prefix: "x-auth-request" authorizationResponse: allowedClientHeaders: patterns: - exact: "location" - exact: "proxy-authenticate" - exact: "set-cookie" - exact: "authorization" - exact: "www-authenticate" - prefix: "x-forwarded" - prefix: "x-auth-request" allowedUpstreamHeaders: patterns: - exact: "location" - exact: "proxy-authenticate" - exact: "set-cookie" - exact: "authorization" - exact: "www-authenticate" - prefix: "x-forwarded" - prefix: "x-auth-request" filterName: envoy.ext_authz filterType: HTTP listenerMatch: portNumber: 443 listenerType: GATEWAY
The important parts are to set the server_uri. The allowed lists of headers is probably more than what is needed, but it works for me.
This final part is optional, if you omit this part, you would be able to use the standard OAuth2_Proxy setup which is to send the cookies to the client directly, instead of using Redis as a session store. This configuration uses Istio’s JWT authentication validation to ensure that every request to your service is authenticated by your issuer. You could expand on this by requiring specific groups per service, and by doing client certificate validation (which you could also couple with Keycloak’s client certificate validation), for the best authentication of incoming requests.
This is set per service with the target definition.
apiVersion: authentication.istio.io/v1alpha1 kind: Policy metadata: name: ingress-auth-policy spec: targets: - name: rancher-demo origins: - jwt: issuer: "https://auth.justin-tech.com/auth/realms/Justin-Tech" jwksUri: "https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/certs" principalBinding: USE_ORIGIN
OAuth2_Proxy performs the OIDC flow for unauthenticated users. We will need to use the Redis session store, as having the ID Token sent to the browser results in cookies that are too long, and splitting the cookie into parts does not work with Istio (see https://pusher.github.io/oauth2_proxy/configuration#configuring-for-use-with-the-nginx-auth_request-directive for an example that spits the cookies with Nginx Ingress support).
Oauth2_Proxy Deployment example:
apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: oauth2-proxy name: oauth2-proxy namespace: default spec: replicas: 1 selector: matchLabels: k8s-app: oauth2-proxy template: metadata: labels: k8s-app: oauth2-proxy spec: containers: - args: - --provider=oidc - --provider-display-name="Keycloak" - --email-domain=corp.justin-tech.com - --email-domain=justin-tech.com - --pass-access-token=true - --pass-authorization-header=true - --set-authorization-header=true - --oidc-issuer-url=https://auth.justin-tech.com/auth/realms/Justin-Tech - --login-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/auth - --redeem-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/token - --validate-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/userinfo - --http-address=0.0.0.0:4180 - --cookie-expire=4h0m0s - --whitelist-domain=".corp.justin-tech.com" - --cookie-domain=.justin-tech.com - --standard-logging=true - --auth-logging=true - --request-logging=true - --skip-provider-button=true - --upstream=static:// - --redis-connection-url=redis://redis-master.default.svc.cluster.local:6379 - --session-store-type=redis env: - name: OAUTH2_PROXY_CLIENT_ID value: rancher-demo - name: OAUTH2_PROXY_CLIENT_SECRET value: # docker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));' - name: OAUTH2_PROXY_COOKIE_SECRET value: - name: OAUTH2_PROXY_COOKIE_DOMAIN value: .justin-tech.com image: quay.io/pusher/oauth2_proxy:v4.1.0 imagePullPolicy: Always name: oauth2-proxy ports: - containerPort: 4180 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: k8s-app: oauth2-proxy name: oauthproxy-service namespace: default spec: ports: - name: http port: 4180 protocol: TCP targetPort: 4180 selector: k8s-app: oauth2-proxy --- apiVersion: v1 kind: Service metadata: labels: k8s-app: oauth2-proxy name: oauthproxy-service namespace: default spec: ports: - name: http port: 4180 protocol: TCP targetPort: 4180 selector: k8s-app: oauth2-proxy
The above config is what I am using to deploy OAuth2_Proxy, some of the configuration is probably unnecessary. Some important parts are the URLs for your OIDC Provider (Keycloak in my case), and the cookie domain, if you have a domain and subdomains that are being used.
I am using the stable/redis helm chart, with minimal configuration explained below. Redis is needed in order to pass JWT tokens from Keycloak to Istio, otherwise the cookies are too large and get split (which is not supported easily in Istio). The downside is that currently OAuth2_Proxy does not support a password on the Redis connection.
The standard values.yaml from redis is fine to use, though you can change a few options:
- Enable persistence for the Redis master and slaves
- Enable metrics (if you want)
i will not cover Keycloak too much in this post, as I have other posts available which cover setting up Keycloak. One thing to note is that you need to set the Redirect URL to either a wildcard, or by setting up one OAuth2_Proxy service per service, or have your services available at different URIs instead of subdomains. This could be a security issue as any redirect url could be set, please consider this before implementing.
Altogether, this setup allows you to use Istio as your ingress-gateway, which provides a lot of features if you are using the rest of their service mesh, including mutual TLS inside the cluster, and integration with Kiali, for example. It allows you to use a service for starting/handling the OIDC flow (if you are using third party applications, or just want an authenticating proxy in front of your applications), and allows you to use any OIDC provider to do it.