Jekyll2023-11-20T10:07:17-05:00https://homelab.blog/feed.xmlHomelab.blogA blog about my Homelab experiences.Jax GauthierGetting Started with Cloudflare Workers2020-02-11T12:00:00-05:002020-02-11T12:00:00-05:00https://homelab.blog/blog/devops/cloudflare-workers<p>I have recently had the need to redirect a few subdomains to another domain, due to old links still being used by public individuals.</p>
<p>I wanted a solution that would not require running another web server, so I started looking into a few different options.</p>
<h2 id="cloudflare-page-rules">Cloudflare Page Rules</h2>
<p>While Page Rules do want I want, they were either not flexible enough (could only target a single subdomain), and would be relatively expensive for the number I would need.</p>
<h2 id="cloudflare-workers">Cloudflare Workers</h2>
<p>Cloudflare Workers are pretty cool. They allow you to run JS code at the Cloudflare edge. They are basically functions-as-a-service, which are super fast and customizable.</p>
<p>I was able to set up a function in less than 5 minutes that allows me to redirect any subdomain to another domain.</p>
<p>The function I am using looks like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="kd">function</span> <span class="nx">handleRequest</span><span class="p">(</span><span class="nx">request</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">Response</span><span class="p">.</span><span class="nx">redirect</span><span class="p">(</span><span class="nx">someURLToRedirectTo</span><span class="p">,</span> <span class="nx">code</span><span class="p">)</span>
<span class="p">}</span>
<span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">fetch</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="nx">event</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">event</span><span class="p">.</span><span class="nx">respondWith</span><span class="p">(</span><span class="nx">handleRequest</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">request</span><span class="p">))</span>
<span class="p">})</span>
<span class="cm">/**
* Example Input
* @param {Request} url where to redirect the response
* @param {number?=301|302} type permanent or temporary redirect
*/</span>
<span class="kd">const</span> <span class="nx">someURLToRedirectTo</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">https://homelab.blog</span><span class="dl">'</span>
<span class="kd">const</span> <span class="nx">code</span> <span class="o">=</span> <span class="mi">302</span> <span class="c1">// Temporary Redirect</span>
</code></pre></div></div>
<p>Basically, it handles requests by sending a 302 Temporary Redirect to my current blog page.</p>
<p>Additionally, the function is super fast. It seems significantly faster than if I were to run a webserver to handle the redirect myself.</p>
<p>It is also using the <a href="https://developers.cloudflare.com/workers/about/routes/">routes</a> function. You still have to define a DNS name for the URLs you want to redirect, and I currently just CNAME them to the workers.dev hostname of the worker.</p>
<p>This is just a start of what workers/functions are able to do, and I am excited to see where else I can use them to make my life easier.</p>Jax GauthierI have recently had the need to redirect a few subdomains to another domain, due to old links still being used by public individuals.Configuring Istio with OIDC authentication2020-02-03T12:00:00-05:002020-02-03T12:00:00-05:00https://homelab.blog/blog/devops/Istio-OIDC-Config<p>In this blog post, we will look at the first part of my ideal setup, which is to secure inbound communication via an authenticating reverse proxy (OAuth2_Proxy), and Keycloak.</p>
<p>This setup will use the follow technologies:</p>
<ol>
<li>Istio (ingress gateway)
<ol>
<li>Certmanager (certificates) - not covered in this post</li>
</ol>
</li>
<li>OAuth2_Proxy (controls the OIDC flow)
<ol>
<li>Redis (session storage)</li>
</ol>
</li>
<li>Keycloak (OIDC Provider)</li>
</ol>
<h2 id="istio">Istio</h2>
<p><a href="https://istio.io/">Istio</a> is a service mesh that allows you to define and secure services in your Kubernetes cluster. In my lab, I use it as the ingress gateway for my cluster, and I am planning on using it to secure service-to-service communication using mutual-tls.</p>
<p>Istio will require a valid certificate for the gateway, you can either set this up via cert-manager, or by importing a certificate into your cluster manually.</p>
<p>Here are some example config files for setting up basic services in Istio:</p>
<p>Istio Gateway:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.istio.io/v1alpha3</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Gateway</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">gateway</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">istio-system</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">istio</span><span class="pi">:</span> <span class="s">ingressgateway</span>
<span class="na">servers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">*'</span>
<span class="na">port</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
<span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">HTTP</span>
<span class="na">tls</span><span class="pi">:</span>
<span class="na">httpsRedirect</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">*'</span>
<span class="na">port</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">https</span>
<span class="na">number</span><span class="pi">:</span> <span class="m">443</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">HTTPS</span>
<span class="na">tls</span><span class="pi">:</span>
<span class="na">credentialName</span><span class="pi">:</span> <span class="s">changeme</span>
<span class="na">httpsRedirect</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">mode</span><span class="pi">:</span> <span class="s">SIMPLE</span>
<span class="na">privateKey</span><span class="pi">:</span> <span class="s">sds</span>
<span class="na">serverCertificate</span><span class="pi">:</span> <span class="s">sds</span>
</code></pre></div></div>
<p>Destination Rule:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.istio.io/v1alpha3</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">DestinationRule</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">oauth-corp-justin-tech-com</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">istio-system</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">oauth.corp.justin-tech.com</span>
<span class="na">trafficPolicy</span><span class="pi">:</span>
<span class="na">loadBalancer</span><span class="pi">:</span>
<span class="na">simple</span><span class="pi">:</span> <span class="s">ROUND_ROBIN</span>
<span class="na">portLevelSettings</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">port</span><span class="pi">:</span>
<span class="na">number</span><span class="pi">:</span> <span class="m">443</span>
<span class="na">tls</span><span class="pi">:</span>
<span class="na">mode</span><span class="pi">:</span> <span class="s">SIMPLE</span>
<span class="na">sni</span><span class="pi">:</span> <span class="s">oauth.corp.justin-tech.com</span>
</code></pre></div></div>
<p>Virtual Service:</p>
<p>oauthproxy-service:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.istio.io/v1alpha3</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">VirtualService</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">oauthproxy-service</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">gateways</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">gateway.istio-system</span>
<span class="na">hosts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">oauth.corp.justin-tech.com</span>
<span class="pi">-</span> <span class="s">oauth.justin-tech.com</span>
<span class="na">http</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">route</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">destination</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">oauthproxy-service</span>
<span class="na">port</span><span class="pi">:</span>
<span class="na">number</span><span class="pi">:</span> <span class="m">4180</span>
<span class="na">weight</span><span class="pi">:</span> <span class="m">100</span>
</code></pre></div></div>
<p>the virtual service for your application, I am using Rancher-Demo:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.istio.io/v1alpha3</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">VirtualService</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">rancher-demo</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">gateways</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">gateway.istio-system</span>
<span class="na">hosts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">rancher-demo.corp.justin-tech.com</span>
<span class="na">http</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">route</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">destination</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">rancher-demo</span>
<span class="na">port</span><span class="pi">:</span>
<span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
<span class="na">weight</span><span class="pi">:</span> <span class="m">100</span>
</code></pre></div></div>
<p>The Envoy Filter, which is the part that starts the redirection for unauthenticated users. This is similar to the auth-url function of Nginx-Ingress. This acts on your Ingress-Gateway workload.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.istio.io/v1alpha3</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">EnvoyFilter</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">authn-filter</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">workloadLabels</span><span class="pi">:</span>
<span class="na">istio</span><span class="pi">:</span> <span class="s">ingressgateway</span>
<span class="na">filters</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">filterConfig</span><span class="pi">:</span>
<span class="na">http_service</span><span class="pi">:</span>
<span class="na">server_uri</span><span class="pi">:</span>
<span class="na">uri</span><span class="pi">:</span> <span class="s">http://oauthproxy-service.default.svc.cluster.local</span>
<span class="na">cluster</span><span class="pi">:</span> <span class="s">outbound|4180||oauthproxy-service.default.svc.cluster.local</span>
<span class="na">timeout</span><span class="pi">:</span> <span class="s">1.5s</span>
<span class="na">authorizationRequest</span><span class="pi">:</span>
<span class="na">allowedHeaders</span><span class="pi">:</span>
<span class="na">patterns</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">cookie"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-access-token"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-user"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-email"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">authorization"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-proto"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">proxy-authorization"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">user-agent"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-host"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">from"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded-for"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">accept"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-auth-request"</span>
<span class="na">authorizationResponse</span><span class="pi">:</span>
<span class="na">allowedClientHeaders</span><span class="pi">:</span>
<span class="na">patterns</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">location"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">proxy-authenticate"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">set-cookie"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">authorization"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">www-authenticate"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-auth-request"</span>
<span class="na">allowedUpstreamHeaders</span><span class="pi">:</span>
<span class="na">patterns</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">location"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">proxy-authenticate"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">set-cookie"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">authorization"</span>
<span class="pi">-</span> <span class="na">exact</span><span class="pi">:</span> <span class="s2">"</span><span class="s">www-authenticate"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-forwarded"</span>
<span class="pi">-</span> <span class="na">prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">x-auth-request"</span>
<span class="na">filterName</span><span class="pi">:</span> <span class="s">envoy.ext_authz</span>
<span class="na">filterType</span><span class="pi">:</span> <span class="s">HTTP</span>
<span class="na">listenerMatch</span><span class="pi">:</span>
<span class="na">portNumber</span><span class="pi">:</span> <span class="m">443</span>
<span class="na">listenerType</span><span class="pi">:</span> <span class="s">GATEWAY</span>
</code></pre></div></div>
<p>The important parts are to set the server_uri. The allowed lists of headers is probably more than what is needed, but it works for me.</p>
<p>This final part is optional, if you omit this part, you would be able to use the standard OAuth2_Proxy setup which is to send the cookies to the client directly, instead of using Redis as a session store. This configuration uses Istio’s JWT authentication validation to ensure that every request to your service is authenticated by your issuer. You could expand on this by requiring specific groups per service, and by doing client certificate validation (which you could also couple with Keycloak’s client certificate validation), for the best authentication of incoming requests.</p>
<p>This is set per service with the target definition.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">authentication.istio.io/v1alpha1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Policy</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">ingress-auth-policy</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">targets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">rancher-demo</span>
<span class="na">origins</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">jwt</span><span class="pi">:</span>
<span class="na">issuer</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://auth.justin-tech.com/auth/realms/Justin-Tech"</span>
<span class="na">jwksUri</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/certs"</span>
<span class="na">principalBinding</span><span class="pi">:</span> <span class="s">USE_ORIGIN</span>
</code></pre></div></div>
<h2 id="oauth2_proxy">Oauth2_Proxy</h2>
<p><a href="https://pusher.github.io/oauth2_proxy">OAuth2_Proxy</a> performs the OIDC flow for unauthenticated users. We will need to use the Redis session store, as having the ID Token sent to the browser results in cookies that are too long, and splitting the cookie into parts does not work with Istio (see https://pusher.github.io/oauth2_proxy/configuration#configuring-for-use-with-the-nginx-auth_request-directive for an example that spits the cookies with Nginx Ingress support).</p>
<p>Oauth2_Proxy Deployment example:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">replicas</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">args</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">--provider=oidc</span>
<span class="pi">-</span> <span class="s">--provider-display-name="Keycloak"</span>
<span class="pi">-</span> <span class="s">--email-domain=corp.justin-tech.com</span>
<span class="pi">-</span> <span class="s">--email-domain=justin-tech.com</span>
<span class="pi">-</span> <span class="s">--pass-access-token=true</span>
<span class="pi">-</span> <span class="s">--pass-authorization-header=true</span>
<span class="pi">-</span> <span class="s">--set-authorization-header=true</span>
<span class="pi">-</span> <span class="s">--oidc-issuer-url=https://auth.justin-tech.com/auth/realms/Justin-Tech</span>
<span class="pi">-</span> <span class="s">--login-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/auth</span>
<span class="pi">-</span> <span class="s">--redeem-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/token</span>
<span class="pi">-</span> <span class="s">--validate-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/userinfo</span>
<span class="pi">-</span> <span class="s">--http-address=0.0.0.0:4180</span>
<span class="pi">-</span> <span class="s">--cookie-expire=4h0m0s</span>
<span class="pi">-</span> <span class="s">--whitelist-domain=".corp.justin-tech.com"</span>
<span class="pi">-</span> <span class="s">--cookie-domain=.justin-tech.com</span>
<span class="pi">-</span> <span class="s">--standard-logging=true</span>
<span class="pi">-</span> <span class="s">--auth-logging=true</span>
<span class="pi">-</span> <span class="s">--request-logging=true</span>
<span class="pi">-</span> <span class="s">--skip-provider-button=true</span>
<span class="pi">-</span> <span class="s">--upstream=static://</span>
<span class="pi">-</span> <span class="s">--redis-connection-url=redis://redis-master.default.svc.cluster.local:6379</span>
<span class="pi">-</span> <span class="s">--session-store-type=redis</span>
<span class="na">env</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">OAUTH2_PROXY_CLIENT_ID</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">rancher-demo</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">OAUTH2_PROXY_CLIENT_SECRET</span>
<span class="na">value</span><span class="pi">:</span>
<span class="c1"># docker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">OAUTH2_PROXY_COOKIE_SECRET</span>
<span class="na">value</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">OAUTH2_PROXY_COOKIE_DOMAIN</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">.justin-tech.com</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">quay.io/pusher/oauth2_proxy:v4.1.0</span>
<span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">Always</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">4180</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">oauthproxy-service</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">4180</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">targetPort</span><span class="pi">:</span> <span class="m">4180</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">oauth2-proxy</span>
</code></pre></div></div>
<p>The above config is what I am using to deploy OAuth2_Proxy, some of the configuration is probably unnecessary. Some important parts are the URLs for your OIDC Provider (Keycloak in my case), and the cookie domain, if you have a domain and subdomains that are being used.</p>
<h3 id="redis">Redis</h3>
<p>I am using the stable/redis helm chart, with minimal configuration explained below. Redis is needed in order to pass JWT tokens from Keycloak to Istio, otherwise the cookies are too large and get split (which is not supported easily in Istio). The downside is that currently OAuth2_Proxy does not support a password on the Redis connection.</p>
<p>The standard values.yaml from redis is fine to use, though you can change a few options:</p>
<ol>
<li>Enable persistence for the Redis master and slaves</li>
<li>Enable metrics (if you want)</li>
</ol>
<h2 id="keycloak">Keycloak</h2>
<p>i will not cover Keycloak too much in this post, as I have other posts available which cover setting up Keycloak. One thing to note is that you need to set the Redirect URL to either a wildcard, or by setting up one OAuth2_Proxy service per service, or have your services available at different URIs instead of subdomains. This could be a security issue as any redirect url could be set, please consider this before implementing.</p>
<p>Altogether, this setup allows you to use Istio as your ingress-gateway, which provides a lot of features if you are using the rest of their service mesh, including mutual TLS inside the cluster, and integration with Kiali, for example. It allows you to use a service for starting/handling the OIDC flow (if you are using third party applications, or just want an authenticating proxy in front of your applications), and allows you to use any OIDC provider to do it.</p>Jax GauthierIn this blog post, we will look at the first part of my ideal setup, which is to secure inbound communication via an authenticating reverse proxy (OAuth2_Proxy), and Keycloak.Getting Started with Kubernetes (at home) — Part 32019-05-04T12:00:00-04:002019-05-04T12:00:00-04:00https://homelab.blog/blog/devops/kubernetes-part-3<p>In the first two parts of this series, we looked at setting up a production Kubernetes cluster in our labs. In part three of this series, we are going to deploy some services to our cluster such as <a href="https://guacamole.apache.org/">Guacamole</a> and <a href="https://www.keycloak.org/">Keycloak</a>.</p>
<p>Step-by-step documentation and further service examples are <a href="https://gitlab.com/just.insane/kubernetes/blob/master/docs/services/services.md">here</a>.</p>
<h2 id="guacamole">Guacamole</h2>
<p><a href="https://guacamole.apache.org/">Guacamole</a> is a very useful piece of software that allows you to remotely connect to your devices via RDP, SSH, or other protocols. I use it extensively to access my lab resources, even when I am at home.</p>
<p>You can use this <a href="https://github.com/Just-Insane/apache-guacamole-helm-chart">Helm Chart</a> to install Guacamole on your Kubernetes cluster. The steps are as follows:</p>
<ol>
<li>Clone the Guacamole Helm Chart from <a href="https://github.com/Just-Insane/apache-guacamole-helm-chart">here</a></li>
<li>Apply any changes to <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/services/guacamole/values.yaml">values.yaml</a> such as the annotations and ingress settings.</li>
<li>Deploy the Helm Chart from the <code class="language-plaintext highlighter-rouge">apache-guacamole-helm-chart</code> directory
<ul>
<li><code class="language-plaintext highlighter-rouge">helm install . -f values.yaml --name=guacamole --namespace=guacamole</code></li>
</ul>
</li>
<li>An ingress is automatically created by the Helm Chart, and you can access it based on the <code class="language-plaintext highlighter-rouge">hosts:</code> section of <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/services/guacamole/values.yaml">values.yaml</a></li>
</ol>
<p>Once you have deployed the Helm Chart, you should be able to access Guacamole at the ingress hostname specified in values.yaml.</p>
<h2 id="keycloak">Keycloak</h2>
<p><a href="https://www.keycloak.org/">Keycloak</a> is an open-source single sign-on solution that is similar to Microsoft’s ADFS product. You can read more about Keycloak on my <a href="https://homelab.blog/blog/security/keycloak-part-1-what-is-sso/">blog</a>.</p>
<p>There is a stable Keycloak Helm Chart available in the default Helm repo, which we will be using to deploy Keycloak, you can find it <a href="https://github.com/helm/charts/tree/master/stable/keycloak">here</a>.</p>
<ol>
<li>Apply any changes to <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/services/keycloak/values.yaml">values.yaml</a></li>
<li>Deploy the helm chart stable/keycloak with values
<ul>
<li><code class="language-plaintext highlighter-rouge">helm install --name keycloak stable/keycloak --values values.yaml</code></li>
</ul>
</li>
<li>Create an <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/services/keycloak/ingress.yaml">ingress</a>
<ul>
<li><code class="language-plaintext highlighter-rouge">kubectl apply -f ingress.yaml</code></li>
</ul>
</li>
<li>Get the default password for the <code class="language-plaintext highlighter-rouge">keycloak</code> user.
<ul>
<li><code class="language-plaintext highlighter-rouge">kubectl get secret --namespace default keycloak-http -o jsonpath="{.data.password}" | base64 --decode; echo</code></li>
</ul>
</li>
</ol>
<p>Though this article is on the shorter side, hopefully it exemplifies how easy it can be to run services in Kubernetes. We mainly looked at pre-made Helm Charts in this article, however deploying a service without a Chart can also be just as easy. I prefer using charts as I find it easier to manage than straight Kubernetes manifest files.</p>
<p>You can checkout my public Kubernetes repo at <a href="https://gitlab.com/just.insane/kubernetes/">https://gitlab.com/just.insane/kubernetes/</a> for further information and more service examples.</p>Jax GauthierIn the first two parts of this series, we looked at setting up a production Kubernetes cluster in our labs. In part three of this series, we are going to deploy some services to our cluster such as Guacamole and Keycloak.Getting Started with Kubernetes (at home) - Part 12019-04-27T00:00:00-04:002019-04-27T00:00:00-04:00https://homelab.blog/blog/devops/kubernetes-part-1<p>When you think about Kubernetes, you probably think AWS or GCP, a nice managed service where you can easily spin up resources and build applications on top of them. This is great, and honestly the best way to experience Kubernetes. However, if all you need is a lab to mess around in and experiment, or learn new things in, this can be very cost inefficient. That is why we are going to look at setting up Kubernetes ourselves.</p>
<p>In this post, we are going to look at the initial deployment of Kubernetes, from creating our nodes (in this case CentOS 7 VMs) to getting a cluster up and running.</p>
<h2 id="part-1-installing-our-vms">Part 1: Installing our VMs</h2>
<p>The first step is to create some VMs. I use a custom vCenter template in my lab, but if you do not have one of those, you can follow these simple steps.</p>
<p>You will need to complete these steps on at least 1 machine, however more is certainly better to get the full benefit of Kubernetes.</p>
<p>Your machine/VM should have at least 1 core and 3Gb of RAM.</p>
<ol>
<li>Install CentOS 7 from the USB ISO image, a basic install is fine</li>
<li>Create a user for Ansible access. This user should be part of the sudo users group, and ideally have passwordless SSH authentication</li>
<li>Assign static IP Addresses to your hosts. Optionally set a hostname.</li>
<li>I hate to say it, but the official docs say to disable the firewall between the nodes, and I was unable to find documentation on which ports are needed.</li>
<li>Optionally, add your hosts to DNS.</li>
</ol>
<h2 id="part-2-kubernetes-installation">Part 2: Kubernetes Installation</h2>
<p>We are going to be using <a href="https://github.com/kubernetes-sigs/kubespray">Kubespray</a> for our cluster, as it makes creating and updating a Kubernetes cluster very simple and straightforward.</p>
<p>You can find the official docs <a href="https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting-started.md">here</a>.</p>
<p>The quick guide is as follows:</p>
<ol>
<li>
<p>Install the dependencies</p>
<p><code class="language-plaintext highlighter-rouge">sudo pip install -r requirements.txt</code></p>
</li>
<li>
<p>Copy the example Inventory</p>
<p><code class="language-plaintext highlighter-rouge">cp -rfp inventory/sample inventory/mycluster</code></p>
</li>
<li>
<p>Build the inventory, you can use the built in builder, or take a look <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/installation/hosts.ini">here</a> for an example. It is fine to have a single master, but the <code class="language-plaintext highlighter-rouge">kube-master</code> and <code class="language-plaintext highlighter-rouge">etcd</code> sections should be the same.</p>
</li>
<li>
<p>Deploy the cluster</p>
<p><code class="language-plaintext highlighter-rouge">ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml</code></p>
</li>
</ol>
<p>Then all you have to do is wait while Kubespray deploys your cluster automatically. On my 6 node cluster, it usually takes about 10–15 minutes for the cluster to be completely setup and running.</p>
<p>Note that in the Kubespray inventory there are a couple of options which are useful to enable. First, in the <code class="language-plaintext highlighter-rouge">addons.yaml</code> file, it is a good idea to enable Helm and the Kubernetes Dashboard automatic deployments. It may also be beneficial to enable <code class="language-plaintext highlighter-rouge">kube_basic_auth</code> in the <code class="language-plaintext highlighter-rouge">k8s-cluster.yaml</code> file, if you are having issues with the default token based authentication. If you decide to do this later, you can simply make the change and then re-run the deployment with the command in step 4 above.</p>
<h2 id="part-3-testing-the-cluster">Part 3: Testing the Cluster</h2>
<p>You can test that your cluster is up and running with the following commands:</p>
<ol>
<li>
<p><code class="language-plaintext highlighter-rouge">kubectl cluster-info</code> which should return something like:</p>
<p><code class="language-plaintext highlighter-rouge">Kubernetes master is running at https://10.0.40.245:6444</code></p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">kubectl get nodes</code> which displays the state of all of your nodes.</p>
</li>
</ol>
<p>You can find more information about how I have setup Kubernetes at my <a href="https://gitlab.com/just.insane/kubernetes">Gitlab repo</a>, which has helpful code snippets, full configuration files, as well as expanded documentation.</p>Jax GauthierWhen you think about Kubernetes, you probably think AWS or GCP, a nice managed service where you can easily spin up resources and build applications on top of them. This is great, and honestly the best way to experience Kubernetes. However, if all you need is a lab to mess around in and experiment, or learn new things in, this can be very cost inefficient. That is why we are going to look at setting up Kubernetes ourselves.Getting Started with Kubernetes (at home) — Part 22019-04-27T00:00:00-04:002019-04-27T00:00:00-04:00https://homelab.blog/blog/devops/kubernetes-part-2<p>In the first part of the series, we looked at installing a bare bones Kubernetes cluster in some CentOS 7 VMs. In this part, we are going to look at setting up some back-end services, like a load balancer and ingress.</p>
<p>Some important things needed to properly run a Kubernetes cluster are a <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/">storage class</a> or manual storage configuration via <a href="https://kubernetes.io/docs/concepts/storage/volumes/">volumes</a>, a <a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">load balancer</a> though not strictly needed makes accessing services much easier, and an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">ingress</a>.</p>
<p><a href="https://gitlab.com/just.insane/kubernetes/blob/master/docs/configuration/configuration.md">Step-by-step documentation is here</a>.</p>
<h4 id="setting-up-helm">Setting up Helm</h4>
<p><a href="https://helm.sh">Helm</a> is known as the “package manager for Kubernetes”, it is a tool that is used to template Kubernetes manifest files in a way that makes it easy to install new applications. I use Helm for all my service installation on Kubernetes due to it’s ease, and simplicity when getting started.</p>
<p>If you did not enable the optional Helm installation via Kubespray, we should do that first.</p>
<h4 id="1-download-helm">1. Download Helm</h4>
<p>If you are on a Mac, you can install Helm via brew install kubernetes-helm. The Helm documentation has instructions for other methods of installation (unpacking a .zip and adding it to the $PATH).</p>
<h4 id="2-setup-a-tiller-user-with-rbac">2. Setup a Tiller user with RBAC</h4>
<p>You can use this <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/rbac-config.yaml">rbac-config.yaml</a> file to setup the user, download it and run the following commands:</p>
<p><code class="language-plaintext highlighter-rouge">kubectl create -f rbac-config.yaml</code></p>
<h4 id="3-initialize-helm">3. Initialize Helm</h4>
<p>Now we have to initialize Helm by deploying Tiller. The simplest method is to run <code class="language-plaintext highlighter-rouge">helm init --service-account tiller</code>. We have to specify the service account for tiller to use due to RBAC.</p>
<h4 id="persistent-storage">Persistent Storage</h4>
<p>Since Docker, and by extension Kubernetes requires storage for any persistent data, it is important to provide Kubernetes some storage. If you were using a cloud provider, this would be handled for you, and you would use the providers default storage solution. In our case, we need to tell Kubernetes what storage to use.</p>
<p>We are going to use NFS storage for our cluster, which requires you to have an existing NFS server.</p>
<p>Once we have an NFS server, we are going to use the <code class="language-plaintext highlighter-rouge">nfs-client-provisioner</code>, which is available via Helm, to access that storage via Kubernetes <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/">storage class</a>.</p>
<p>Simply run <code class="language-plaintext highlighter-rouge">helm install stable/nfs-client-provisioner -name nfs-client -namespace kube-system -set nfs.server=10.0.40.5 -set nfs.path=/mnt/Shared/kubernetes</code></p>
<p>Let’s break that down a bit, the above command does the following:</p>
<ol>
<li>Tell Helm to install the Helm Chart <code class="language-plaintext highlighter-rouge">stable/nfs-client-provisioner</code></li>
<li>Give the release a name <code class="language-plaintext highlighter-rouge">nfs-client</code></li>
<li>Install <code class="language-plaintext highlighter-rouge">nfs-client</code> to the <code class="language-plaintext highlighter-rouge">kube-system</code> namespace</li>
<li>Set the NFS Server IP to 10.0.40.5 you should change this as necessary</li>
<li>Set the NFS Path to /mnt/Shared/kubernetes you should change this to the share path on your NFS server</li>
</ol>
<p>We should now check the status of nfs-client to ensure it is running correctly, via helm status nfs-client which should show ready.</p>
<h4 id="install-metallb">Install MetalLB</h4>
<p><a href="https://metallb.universe.tf/">MetalLB</a> is a <a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">load balancer</a> for Kubernetes that allows Kubernetes services to retrieve IP addresses when needed that are on the LAN or WAN. This greatly simplifies network access to services in a bare-metal (or VM) based Kubernetes cluster.</p>
<p>MetalLB has some requirements that we must comply with, namely, we have to have a Kubernetes cluster, a supported cluster network configuration, some IPv4 addresses, and optionally hardware that supports BGP. The first two are automatically handled by Kubespray in section 1, and for the IP addresses, you can provide a block that is outside your DHCP server’s scope on the same network as the Kubernetes nodes. I have not used the BGP option, as I do not have BGP hardware available.</p>
<p>You can install MetalLB via Helm with the following command <code class="language-plaintext highlighter-rouge">helm install -name metallb -f values.yaml stable/metallb</code> an example values.yaml file can be found <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/metallb/values.yaml">here</a>. Notice the <code class="language-plaintext highlighter-rouge">configInline</code> section that defines the IP addresses MetalLB will hand out.</p>
<h4 id="install-nginx-ingress">Install Nginx-Ingress</h4>
<p><a href="https://github.com/kubernetes/ingress-nginx">Nginx-Ingress</a> is an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> service for Kubernetes. Ingresses serve to manage external access to services in the cluster. It is important to understand how nginx-ingress works, so I recommend you read through the <a href="https://github.com/helm/charts/tree/master/stable/nginx-ingress">GitHub documentation</a>.</p>
<p>Once you have done that, we can install Nginx-Ingress via Helm. You can review the <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/nginx-ingress/values.yaml">values.yaml</a>as an example, the main thing to change being the <code class="language-plaintext highlighter-rouge">default-ssl-certificate</code> that Nginx should use.</p>
<p>Once you have done that, you can install it via <code class="language-plaintext highlighter-rouge">helm install -name nginx-ingress -f values.yaml stable/nginx-ingress -namespace nginx-ingress</code> which does the following.</p>
<ol>
<li>Tell Helm to install the Helm Chart <code class="language-plaintext highlighter-rouge">stable/nginx-ingress</code></li>
<li>With the release name <code class="language-plaintext highlighter-rouge">nginx-ingress</code></li>
<li>To the <code class="language-plaintext highlighter-rouge">nginx-ingress</code> namespace</li>
<li>With the values from <code class="language-plaintext highlighter-rouge">values.yaml</code></li>
</ol>
<p>Since the type is set to loadBalancer, MetalLB will automatically provide an external IP to the service from the provided IP addresses, which you can point your DNS or Port Forwards to in order to access your cluster’s services.</p>
<h4 id="install-cert-manager">Install Cert-Manager</h4>
<p><a href="https://docs.cert-manager.io/en/latest/">Cert-Manager</a> is a complex service that allows us to automatically retrieve and renew SSL certificates from certificate providers. It works with Nginx-Ingress to ensure that all of our services are available via a secured connection over HTTPS.</p>
<p>There are a number of steps required to setup Cert-Manager correctly. You should start by reviewing the Helm Chart documentation <a href="https://github.com/jetstack/cert-manager/tree/master/deploy/charts/cert-manager">here</a>.</p>
<p><em>Prerequisites</em></p>
<ol>
<li>Install the CustomResourceDefinition resources separately</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">kubectl apply -f [https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml](https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml)</code></li>
</ul>
<ol>
<li>Create the namespace for cert-manager</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">kubectl create namespace cert-manager</code></li>
</ul>
<ol>
<li>Label the cert-manager namespace to disable resource validation</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true</code></li>
</ul>
<ol>
<li>Add the Jetstack Helm repository</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">helm repo add jetstack [https://charts.jetstack.io](https://charts.jetstack.io)</code></li>
</ul>
<ol>
<li>Update your local Helm chart repository cache</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">helm repo update</code></li>
</ul>
<p><em>Installation via Helm</em></p>
<ol>
<li>Make any changes to <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/cert-manager/values.yaml">values.yaml</a> notably the <code class="language-plaintext highlighter-rouge">podDnsConfig/nameservers</code></li>
<li>Deploy the Helm Chart</li>
</ol>
<ul>
<li><code class="language-plaintext highlighter-rouge">helm install -name cert-manager -namespace cert-manager -f values.yaml jetstack/cert-manager</code></li>
</ul>
<p><em>Verify the Installation</em></p>
<p>This is covered in the official documentation which you can find <a href="https://docs.cert-manager.io/en/latest/getting-started/install.html#verifying-the-installation">here</a>. It has you create a test issuer and get a self-signed certificate to ensure everything is working properly.</p>
<p><em>Create the Production Issuer</em></p>
<p><strong>Note</strong> : Creation of a test issuer is omitted for brevity, but please ensure you try creating a test issuer first so you do not hit the Let’s Encrypt rate limits.</p>
<p>If you are using Let’s Encrypt and Cloudflare, you can use this <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/cert-manager/letsencrypt.yaml">letsencrypt.yaml</a> file. Remember to change the values as needed, especially anything marked with changeme.</p>
<p>Once you are ready, run <code class="language-plaintext highlighter-rouge">kubectl apply -f letsencrypt.yaml</code> to create the issuer.</p>
<p><em>Create Cloudflare API Key Secret</em></p>
<p>Now we need to add our Cloudflare API key into a Kubernetes secret. You can use <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/cert-manager/cloudflare-secret.yaml">cloudflare-secret.yaml</a> as an example.</p>
<p>Once you have made your changes, create the secret with <code class="language-plaintext highlighter-rouge">kubectl apply -f cloudflare-secret.yaml</code>.</p>
<p><em>Create the Default Certificate</em></p>
<p>We can now create the default certificate that Nginx-Ingress will use for serving our sites. See <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/cert-manager/certificate.yaml">certificate.yaml</a> for an example.</p>
<p>Once you have made your changes to the file, with regard to the names and domains, you can create the certificate with <code class="language-plaintext highlighter-rouge">kubectl apply -f certificate.yaml</code>.</p>
<h4 id="create-an-ingress">Create an Ingress</h4>
<p>Now we can create an ingress for the Kubernetes dashboard, so instead of going to <a href="https://masternode:6443/proxy/">https://masternode:6443/proxy/</a> ect, we can go directly to <a href="https://kubernetes.domain.com">https://kubernetes.domain.com</a></p>
<p>The first step is to create the dashboard ingress, which we can do with <a href="https://gitlab.com/just.insane/kubernetes/blob/master/src/configuration/dashboard/ingress.yaml">ingress.yaml</a> once you have made your changes, you can create the ingress with <code class="language-plaintext highlighter-rouge">kubectl apply -f ingress.yaml</code></p>
<p>You can now go to <a href="https://kubernetes.domain.com">https://kubernetes.domain.com</a> to access your Kubernetes dashboard, after you add the domain to DNS via the nginx-ingress external IP.</p>
<p>You now have a completely setup Kubernetes cluster running at home that you can test your services on, or use as a homelab.</p>Jax GauthierIn the first part of the series, we looked at installing a bare bones Kubernetes cluster in some CentOS 7 VMs. In this part, we are going to look at setting up some back-end services, like a load balancer and ingress.What is Helm-Vault?2019-04-25T00:00:00-04:002019-04-25T00:00:00-04:00https://homelab.blog/blog/what-is-helm-vault<p><a href="https://github.com/Just-Insane/Helm-Vault">Helm-Vault</a> is a new application designed to protect secrets contained in Helm Chart’s values.yaml files.</p>
<h4 id="the-problem">The Problem:</h4>
<p>The problem with using Helm with Kubernetes is that there is no good way to secure your private configuration items stored in the YAML configuration files.</p>
<p>There are multiple reasons you may want to do this, such as to audit who is accessing what secrets, or to prevent unauthorized modifications to the environment. Another reason may be that you want to have publicly available documentation without worrying about scrubbing the files.</p>
<h4 id="current-solutions">Current Solutions:</h4>
<p>Currently available solutions require you to significantly modify the Helm Chart, or encrypt the entire document with GPG or a hosted KMS solution.</p>
<p>This can be a pain to manage, who wants to use GPG all the time to work with files? What happens if the key is lost? What happens if the internet is down and you can’t access your KMS provider?</p>
<h4 id="what-makes-helm-vault-different">What makes Helm-Vault different:</h4>
<p>With Helm-Vault, you get full access to the structure and content of the YAML documents, even when they are in an “encrypted” state, this provides you a lot more flexibility. When publishing charts, you can seamlessly provide your encrypted YAML files, and with the deliminator, provide easily notable locations where information needs to be changed.</p>
<p>If you run Hashicorp <a href="https://www.vaultproject.io/">Vault</a> internally, you will always have access to your secret data, and don’t have to worry about working with GPG keys, which makes life a lot easier for your developers.</p>
<h4 id="getting-started">Getting Started:</h4>
<p>As long as you have Python 3.7, a working Hashicorp Vault environment, and a Vault token, you can get started with Helm-Vault in 2 easy steps:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">pip3 install git+https://github.com/Just-Insane/helm-vault</code></li>
<li><code class="language-plaintext highlighter-rouge">helm plugin install https://github.com/Just-Insane/helm-vault</code></li>
</ol>
<p>You can find information about using Helm-Vault in the <a href="https://github.com/Just-Insane/helm-vault/blob/master/README.md#usage-and-examples">README</a>.</p>
<p>If you have any questions or concerns, feel free to open an issue on the project’s <a href="https://github.com/Just-Insane/helm-vault/issues">GitHub Issue Tracker</a>.</p>Jax GauthierHelm-Vault is a new application designed to protect secrets contained in Helm Chart’s values.yaml files.The Importance of Security Testing2019-04-24T00:00:00-04:002019-04-24T00:00:00-04:00https://homelab.blog/blog/security/importance-of-security-testing<p>While setting up a new Keycloak client in my lab over the weekend, I discovered something odd, a number of users in my Keycloak database who should not have been there.</p>
<h4 id="background">Background:</h4>
<p>In my homelab environment, I have been using <a href="https://www.keycloak.org/">Keycloak</a> for securing my services, either directly for applications like Nextcloud, Confluence, and Jira, or indirectly, via a authenticated reverse proxy, for things that don’t such as single page applications like Hashicorp’s <a href="https://www.consul.io/">Consul</a> or <a href="https://jupyter.org/">Jupyter</a>. This has been a fine setup, and has allowed me to access my internal applications easily and securely from anywhere, with 2FA and seamless access when moving between services. Or so I thought.</p>
<h4 id="discovery">Discovery:</h4>
<p>After noticing some odd users in my Keycloak user’s list, I decided to do some digging. I knew I had initially allowed sign-ups to Keycloak to very locked down services, like read only access to parts of Confluence, and my internal issue tracker on Jira. These users were put into a default role in Keycloak called External-Users, which had very limited access to services on my network.</p>
<p>I decided to create a new user and see what all could be accessed, for the hell of it. I opened a new private window so that I could ensure I was not using cookies from my authenticated session, and went to work creating a new user. I was happily greeted with a <a href="https://duo.com/">Duo</a> page (Duo is a 2FA provider that you can integrate with to protect many different services), until I realized there was an option to enroll myself as a user in Duo. That is when I realized there was an issue.</p>
<h4 id="breach">Breach:</h4>
<p>After registering myself as a new user in Duo, I started to poke around, going first to Confluence and Jira, and seeing that I still had my limited permissions, was a bit relived. Then I went to Consul, and the page opened right to the main app, showing all my services humming away happily. Next I went to Jupyter, and immediately was greeted with the option of creating a new shell session.</p>
<h4 id="containment">Containment:</h4>
<p>As soon as I was able to get access to Jupyter, I disabled signups on Keycloak, cleaned out the unknown users (some of which used legitimate emails), and set Duo to not allow users to enroll themselves. I also ensured there were no active sessions in Keycloak and started locking my network down to ensure there was nobody still inside the perimeter.</p>
<h4 id="root-cause">Root Cause:</h4>
<p>The root cause of this issue was my reverse proxy. Knowing that most of my applications were secured behind it, especially the ones which do not handle their own authentication, I tore it apart. I didn’t have to look far to see the error in my ways.</p>
<p>Almost all of my proxied services had this block in the server section (private information omitted):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>access_by_lua '
local opts = {
discovery = "$URL/.well-known/openid-configuration",
redirect_uri_path = "/redirect_uri",
redirect_uri_scheme = "https",
client_id = "OpenResty",
ssl_verify = "no",
client_secret = "$ClientSecret",
session_contents = {id_token=true}
}
local res, err = require("resty.openidc").authenticate(opts)
if err then
ngx.status = 403
ngx.say(err)
ngx.exit(ngx.HTTP_FORBIDDEN)
end
';
</code></pre></div></div>
<p>Let’s break that down a bit, shall we?</p>
<p>My reverse proxy is OpenResty, a modified Nginx server setup to run Lua. On CentOS7 you have to build Nginx from source to add an OIDC authentication plugin, with OpenResty, it is built in.</p>
<p>First, we define some options for the OIDC authentication:</p>
<ol>
<li>The URL of the openid configuration information</li>
<li>The path to redirect to, this is a URI that does not exist in the backend application that should be redirected to after authentication with Keycloak so the proxy can check the authentication</li>
<li>The redirect scheme</li>
<li>The Keycloak Client ID</li>
<li>If SSL for Keycloak should be verified (this should be set to yes)</li>
<li>The client secret that you get from Keycloak</li>
<li>The session contents, in this case, we are just looking for the id_token</li>
<li>The authentication check, more on this below</li>
<li>What to do if there is an error, in this case, send a 403, print the error, and set HTTP_FORBIDDEN, then exit</li>
</ol>
<p>Looking at the authentication check, do you see the problem? It is only checking that the person accessing the protected resource is authenticated to Keycloak. It does not check for their group, roles, or even to ensure they have a valid email address for the domain.</p>
<p>Since anyone could sign up to Keycloak, and anyone could enroll themselves in Duo, they would be authenticated, and would therefore pass this check automatically, giving anyone who tried access to these internal applications.</p>
<h4 id="it-gets-worse">It Gets Worse:</h4>
<p>Knowing that I had setup external Identity Providers in Keycloak, such as Facebook and Google (adding external Identity Providers in Keycloak lets people sign in with other services which handle their own authentication), I went back and tested these out.</p>
<p>I closed out my current private window, and opened a new one, going back to my Consul URL, and clicking on the login button for Google. When prompted, I entered a spare email address and password, and once I got through the Google verification, I was redirected back to Keycloak, and then to the main Consul page.</p>
<p>After checking in Keycloak, I didn’t see a new user created with this email. Since authentication was handled by Google, all Keycloak was caring about was if the user successfully authenticated with Google, and just passed the user back to the proxy as being authenticated in Keycloak.</p>
<p>Not only could anyone sign up for an account in Keycloak and add themselves to 2FA, they could completely bypass these steps without raising suspicion and not creating any user records.</p>
<h4 id="lessons-learned">Lessons Learned:</h4>
<p>The lessons I learned through this were many, the first is to always double check everything, especially when it is security related. If I had checked these authentication flows when initially setting everything up, I would have caught these issues much sooner and this would not have been a problem.</p>
<p>Second, it is important to understand and read the documentation fully. This could have been easily mitigated or completely prevented by adding more checks to OpenResty, disabling sign up or third party authentication in Keycloak, or preventing user self-enrollment in Duo.</p>
<p>Third, check everything. If you enable a feature, ensure it does exactly what you are expecting it to do. If not, it is best to disable it, or get clarification as to why it is doing what it is.</p>
<h4 id="conclusion">Conclusion:</h4>
<p>I messed up, spectacularly. Luckily, it was in a test environment and not production, and there was nothing overly sensitive on my network.</p>
<p>This is a mistake that I will not forget, and that I will bring with me in my career.</p>
<p>There is no inherent issue with the tools that I used, or even a combination of tools, it was purely a configuration issue on my part. I would still 100% recommend all of the tools mentioned in this post, and intend to keep on using them. I have gained more respect for them, and intend to put them to use to further secure my network in the future.</p>
<h4 id="future-thoughts">Future Thoughts:</h4>
<p>As I rebuild my lab with a focus on Kubernetes and Zero Trust network design, the lessons I learned here will guide me in ensuring that everything is as secure as possible.</p>Jax GauthierWhile setting up a new Keycloak client in my lab over the weekend, I discovered something odd, a number of users in my Keycloak database who should not have been there.Keycloak SSO Part 2: Setting up Keycloak2018-12-07T18:35:00-05:002018-12-07T18:35:00-05:00https://homelab.blog/blog/security/keycloak-part-2-setting-up-keycloak<p>In part two of this series, we are going to look at setting up a standalone-ha deployment of Keycloak on two CentOS 7 servers.</p>
<p>There are three different deployment types for Keycloak, Standalone, Standalone-HA, and Domain Clustered. Standalone deployments are single servers, this is good for a dev or test environment, but not very useful for production use. Standalone-HA are one or more servers which can both be used to serve authentication requests. This method requires a shared database, and each server is configured manually. In a Domain deployment, there is a master server known as the domain controller, and one or more host controllers which serve authentication requests. This mode allows the host controllers to all have an updated configuration when it is changed on the domain controller, greatly reducing administration overhead with multiple servers.</p>
<h2 id="installation">Installation</h2>
<p>Hardware requirements, as well as distribution directory structure and operation mode information can be found at <a href="https://www.keycloak.org/docs/latest/server_installation/index.html#installation">https://www.keycloak.org/docs/latest/server_installation/index.html#installation</a></p>
<p>I use Ansible to deploy the folder structure for Keycloak and make sure all dependencies are setup correctly. This allows me to easily update and ensure that all my Keycloak servers are deployed correctly. My playbook calls <a href="https://github.com/andrewrothstein/ansible-keycloak">https://github.com/andrewrothstein/ansible-keycloak</a> with some configuration file changes.</p>
<h2 id="configuration">Configuration</h2>
<p>Reading and understanding the official documentation is essential to installing Keycloak in a secure manner, I highly recommend you follow the information there and use my configuration as a guide.</p>
<h3 id="operation-mode">Operation Mode</h3>
<p>The first thing to think about when deploying Keycloak is what operation mode you want to use. This will mostly come down to your environment, and the configuration of most modes is the same, just within different files. I am most experienced with Standalone-HA mode, so that is what we are going to work with in this series.</p>
<p>Configuration for this mode is done in the standalone-ha.xml configuration file found at <code class="language-plaintext highlighter-rouge">$keycloak_home/standalone/configuration/standalone-ha.xml</code>. This file needs to be edited on all servers in a standalone-ha cluster setup.</p>
<h3 id="relational-database-setup">Relational Database Setup</h3>
<p>The next thing we have to do is setup Keycloak to use a database, since we are going to be creating a deployment with multiple servers, we are going to need a shared database. Setting up a centrally accessible database is beyond the scope of this article. Just know that we are going to be using a PostgreSQL database that is hosted outside of both of the Keycloak servers.</p>
<h4 id="download-a-jdbc-driver">Download a JDBC driver</h4>
<p>The first step in setting up a database for Keycloak is to download a JDBC driver for the database. This allows Java to interact with the database. You can usually find these on the main site of your chosen database. For example, PostgreSQL’s JDBC driver can be found here: https://jdbc.postgresql.org/download.html</p>
<h4 id="package-the-driver-jar-and-install">Package the driver JAR and install</h4>
<p>The official documentation is a good resource for how to package the driver for use with Keycloak, and there is no point in duplicating efforts. It can be found here: <a href="https://www.keycloak.org/docs/latest/server_installation/index.html#package-the-jdbc-driver">https://www.keycloak.org/docs/latest/server_installation/index.html#package-the-jdbc-driver</a></p>
<p>This boils down to adding a folder structure, copying the .jar file, and adding an .xml file like the following:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0" ?></span>
<span class="nt"><module</span> <span class="na">xmlns=</span><span class="s">"urn:jboss:module:1.3"</span> <span class="na">name=</span><span class="s">"org.postgresql"</span><span class="nt">></span>
<span class="nt"><resources></span>
<span class="nt"><resource-root</span> <span class="na">path=</span><span class="s">"postgresql-9.4.1212.jar"</span><span class="nt">/></span> <span class="c"><!-- update the filename to match your PostgreSQL JDBC driver file name --></span>
<span class="nt"></resources></span>
<span class="nt"><dependencies></span>
<span class="nt"><module</span> <span class="na">name=</span><span class="s">"javax.api"</span><span class="nt">/></span>
<span class="nt"><module</span> <span class="na">name=</span><span class="s">"javax.transaction.api"</span><span class="nt">/></span>
<span class="nt"></dependencies></span>
<span class="nt"></module></span>
</code></pre></div></div>
<p>Making sure to update the path with the correct file name.</p>
<h4 id="declare-and-load-the-driver">Declare and load the driver</h4>
<p>This part, as well as modifying the datasource are a bit more advanced, so I will go over them in a bit more detail here, however the documentation is still very helpful.</p>
<p>We are going to look at the standalone-ha.xml file we were working on earlier, specifically the <code class="language-plaintext highlighter-rouge">drivers</code> XML block. In this block, we will be adding an additional driver. We can mostly copy the existing format of the h2 driver, and update the information for PostgreSQL. Below is an example of a driver in <code class="language-plaintext highlighter-rouge">standalone-ha.xml</code></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><drivers></span>
<span class="nt"><driver</span> <span class="na">name=</span><span class="s">"h2"</span> <span class="na">module=</span><span class="s">"com.h2database.h2"</span><span class="nt">></span>
<span class="nt"><xa-datasource-class></span>org.h2.jdbcx.JdbcDataSource<span class="nt"></xa-datasource-class></span>
<span class="nt"></driver></span>
<span class="nt"><driver</span> <span class="na">name=</span><span class="s">"postgresql"</span> <span class="na">module=</span><span class="s">"org.postgresql"</span><span class="nt">></span>
<span class="nt"><xa-datasource-class></span>org.postgresql.xa.PGXADataSource<span class="nt"></xa-datasource-class></span>
<span class="nt"></driver></span>
<span class="nt"></drivers></span>
</code></pre></div></div>
<p>As we can see, the declaration of the driver is nearly identical to the pre-configured H2 database driver.</p>
<h4 id="modify-the-keycloak-datasource">Modify the Keycloak Datasource</h4>
<p>Below we will see an example of a working PostgreSQL datasource configuration.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><datasource</span> <span class="na">jndi-name=</span><span class="s">"java:jboss/datasources/KeycloakDS"</span> <span class="na">pool-name=</span><span class="s">"KeycloakDS"</span> <span class="na">enabled=</span><span class="s">"true"</span> <span class="na">use-java-context=</span><span class="s">"true"</span><span class="nt">></span>
<span class="nt"><connection-url></span>jdbc:postgresql://$URL:$PORT/$DATABASE<span class="nt"></connection-url></span>
<span class="nt"><driver></span>postgresql<span class="nt"></driver></span>
<span class="nt"><pool></span>
<span class="nt"><max-pool-size></span>20<span class="nt"></max-pool-size></span>
<span class="nt"></pool></span>
<span class="nt"><security></span>
<span class="nt"><user-name></span>$USERNAME<span class="nt"></user-name></span>
<span class="nt"><password></span>$PASSWORD<span class="nt"></password></span>
<span class="nt"></security></span>
<span class="nt"></datasource></span>
</code></pre></div></div>
<p>$URL = The URL or IP Address of the PostgreSQL server<br />
$PORT = The port to connect to PostgreSQL database<br />
$DATABASE = The name of the database that is configured for Keycloak<br />
$USERNAME = The username that has access to the database specified above<br />
$PASSWORD = The password of the user defined above</p>
<p>In the end, you should end up with a <code class="language-plaintext highlighter-rouge">datasources</code> section that looks like the following:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><subsystem</span> <span class="na">xmlns=</span><span class="s">"urn:jboss:domain:datasources:5.0"</span><span class="nt">></span>
<span class="nt"><datasources></span>
<span class="nt"><datasource</span> <span class="na">jndi-name=</span><span class="s">"java:jboss/datasources/KeycloakDS"</span> <span class="na">pool-name=</span><span class="s">"KeycloakDS"</span> <span class="na">enabled=</span><span class="s">"true"</span> <span class="na">use-java-context=</span><span class="s">"true"</span><span class="nt">></span>
<span class="nt"><connection-url></span>jdbc:postgresql://$URL:$PORT/$DATABASE<span class="nt"></connection-url></span>
<span class="nt"><driver></span>postgresql<span class="nt"></driver></span>
<span class="nt"><pool></span>
<span class="nt"><max-pool-size></span>20<span class="nt"></max-pool-size></span>
<span class="nt"></pool></span>
<span class="nt"><security></span>
<span class="nt"><user-name></span>$USERNAME<span class="nt"></user-name></span>
<span class="nt"><password></span>$PASSWORD<span class="nt"></password></span>
<span class="nt"></security></span>
<span class="nt"></datasource></span>
<span class="nt"><drivers></span>
<span class="nt"><driver</span> <span class="na">name=</span><span class="s">"h2"</span> <span class="na">module=</span><span class="s">"com.h2database.h2"</span><span class="nt">></span>
<span class="nt"><xa-datasource-class></span>org.h2.jdbcx.JdbcDataSource<span class="nt"></xa-datasource-class></span>
<span class="nt"></driver></span>
<span class="nt"><driver</span> <span class="na">name=</span><span class="s">"postgresql"</span> <span class="na">module=</span><span class="s">"org.postgresql"</span><span class="nt">></span>
<span class="nt"><xa-datasource-class></span>org.postgresql.xa.PGXADataSource<span class="nt"></xa-datasource-class></span>
<span class="nt"></driver></span>
<span class="nt"></drivers></span>
<span class="nt"></datasources></span>
<span class="nt"></subsystem></span>
</code></pre></div></div>
<h2 id="clustering">Clustering</h2>
<p>The above steps will get a basic setup going with a shared database, however to properly cluster Keycloak, there are a few more steps that need to be completed.</p>
<p>The relevent sections from the Keycloak documentation is below:</p>
<ol>
<li>
<p><a href="https://www.keycloak.org/docs/latest/server_installation/index.html#_operating-mode">Pick an operation mode</a></p>
</li>
<li>
<p><a href="https://www.keycloak.org/docs/latest/server_installation/index.html#_database">Configure a shared external database</a></p>
</li>
<li>
<p><a href="https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy">Set up a load balancer</a></p>
</li>
<li>
<p><a href="https://www.keycloak.org/docs/latest/server_installation/index.html#multicast-network-setup">Supplying a private network that supports IP multicast</a></p>
</li>
</ol>
<p>We have already completed steps 1 and 2 in setting up a cluster. There are some additional setup needed for the next two operations, parts of which are made serviceable here, but are covered in more detail at the above links.</p>
<h3 id="set-up-a-load-balancer">Set up a load balancer</h3>
<h4 id="identifying-client-ip-addresses">Identifying client IP addresses</h4>
<p>It is very important that Keycloak is able to identify client IP addresses for various reasons, which are explained further in the docs. We will go over the changes that have to be made in <code class="language-plaintext highlighter-rouge">standalone-ha.xml</code> here.</p>
<p>You will need to configure the <code class="language-plaintext highlighter-rouge">urn:jboss:domain:undertow:6.0</code> block to look like below:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><subsystem</span> <span class="na">xmlns=</span><span class="s">"urn:jboss:domain:undertow:6.0"</span><span class="nt">></span>
<span class="nt"><buffer-cache</span> <span class="na">name=</span><span class="s">"default"</span><span class="nt">/></span>
<span class="nt"><server</span> <span class="na">name=</span><span class="s">"default-server"</span><span class="nt">></span>
<span class="nt"><ajp-listener</span> <span class="na">name=</span><span class="s">"ajp"</span> <span class="na">socket-binding=</span><span class="s">"ajp"</span><span class="nt">/></span>
<span class="nt"><http-listener</span> <span class="na">name=</span><span class="s">"default"</span> <span class="na">socket-binding=</span><span class="s">"http"</span> <span class="na">redirect-socket=</span><span class="s">"https"</span> <span class="na">proxy-address-forwarding=</span><span class="s">"true"</span><span class="nt">/></span>
...
<span class="nt"></server></span>
...
<span class="nt"></subsystem></span>
</code></pre></div></div>
<h4 id="enable-https-with-a-reverse-proxy">Enable HTTPS with a Reverse Proxy</h4>
<p>If you have a reverse proxy in front of Keycloak which handles your SSL connections and terminations, you need to make the following changes:</p>
<p>In <code class="language-plaintext highlighter-rouge">urn:jboss:domain:undertow:6.0</code> block (configured above) change the <code class="language-plaintext highlighter-rouge">redirect-socket</code> from https to a socket binding which we will define.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><subsystem</span> <span class="na">xmlns=</span><span class="s">"urn:jboss:domain:undertow:6.0"</span><span class="nt">></span>
...
<span class="nt"><http-listener</span> <span class="na">name=</span><span class="s">"default"</span> <span class="na">socket-binding=</span><span class="s">"http"</span>
<span class="na">proxy-address-forwarding=</span><span class="s">"true"</span> <span class="na">redirect-socket=</span><span class="s">"proxy-https"</span><span class="nt">/></span>
...
<span class="nt"></subsystem></span>
</code></pre></div></div>
<p>We will now need to add a new socket binding to the <code class="language-plaintext highlighter-rouge">socket-binding-group</code> element, like below:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><socket-binding-group</span> <span class="na">name=</span><span class="s">"standard-sockets"</span> <span class="na">default-interface=</span><span class="s">"public"</span>
<span class="na">port-offset=</span><span class="s">"${jboss.socket.binding.port-offset:0}"</span><span class="nt">></span>
...
<span class="nt"><socket-binding</span> <span class="na">name=</span><span class="s">"proxy-https"</span> <span class="na">port=</span><span class="s">"443"</span><span class="nt">/></span>
...
<span class="nt"></socket-binding-group></span>
</code></pre></div></div>
<h2 id="testing-the-cluster">Testing the Cluster</h2>
<p>Once the changes have been made on all of your Keycloak servers, we can manually start the Keycloak servers in any order. The command for doing so is</p>
<p><code class="language-plaintext highlighter-rouge">bin/standalone.sh --server-config=standalone-ha.xml</code> from the Keycloak home directory. The Keycloak servers will automatically configure themselves if they are connected to the same external database, and you can use your load balancer or reverse proxy to connect to either server to perform authentication operations.</p>
<h3 id="firewall">Firewall</h3>
<p>Ensure you have configured the firewall correctly, Keycloak listens on ports 8080 and 8443 by default. There may be additional ports that need to be opened based on your configuration.</p>
<h2 id="running-at-boot">Running at boot</h2>
<p>Assuming your tests have passed and you can reach both of your Keycloak servers directly and through your load balancer, you are ready to setup a systemd unit file and have Keycloak start at boot.</p>
<p>Below is a copy of the systemd unit file I am using, which is placed at <code class="language-plaintext highlighter-rouge">/etc/systemd/system/keycloak.service</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Keycloak Identity Provider
After=syslog.target network.target
Before=httpd.service
[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1 JAVA_HOME=/usr/local/java
User=keycloak
Group=keycloak
LimitNOFILE=102642
PIDFile=/var/run/keycloak/keycloak.pid
ExecStart=/usr/local/keycloak/bin/standalone.sh --server-config=standalone-ha.xml
#StandardOutput=null
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<p>Once you have completed this step, you can start and enable the service by running the below commands on all of your Keycloak servers:</p>
<p><code class="language-plaintext highlighter-rouge">systemctl enable keycloak</code></p>
<p><code class="language-plaintext highlighter-rouge">systemctl start keycloak</code></p>Jax GauthierIn part two of this series, we are going to look at setting up a standalone-ha deployment of Keycloak on two CentOS 7 servers.Keycloak SSO Part 1: What is single sign on?2018-12-06T00:00:00-05:002018-12-06T00:00:00-05:00https://homelab.blog/blog/security/keycloak-part-1-what-is-sso<p>In this series, we will take a look at Keycloak, an open-source single sign on solution similar to Microsoft’s ADFS product.</p>
<p>In this first part, we will take a brief look at what SSO and Keycloak are, and why they are used.</p>
<h2 id="what-is-single-sign-on">What is Single Sign On</h2>
<p>Single sign-on (SSO) is a property of access control of multiple related, yet independent, software systems. <a href="https://en.m.wikipedia.org/wiki/Single_sign-on">Wikipedia</a></p>
<p>What this means is that we can use a single authentication service to allow users to login to other services, without providing a password to the service that is being logged into.</p>
<p>This is different than connecting each of these applications to LDAP, as this requires providing a username and password to every service that you want to login to.</p>
<h3 id="benefits-of-sso">Benefits of SSO</h3>
<ol>
<li>
<p>Central location for authentication allows for easier auditing and security.</p>
</li>
<li>
<p>Central location for configuration of authorization, ie, Jane is able to access email but not the CRM suite</p>
</li>
<li>
<p>Authentication to external services such as a hosted CRM suite are made possible without sending LDAP requests over the internet</p>
</li>
<li>
<p>Tokens are wrapped in SSL/TLS as part of the HTTPS connection, but can also be both signed and/or encrypted using keys known only between the identity provider and service for increased security</p>
</li>
</ol>
<h2 id="what-is-keycloak">What is Keycloak</h2>
<p>Keycloak is an open source program that allows you to setup a secure single sign on provider. It supports multiple protocols such as SAML 2.0 and OpenID Connect. It can also store user credentials locally or via an LDAP or Kerberos backend.</p>
<p>These features allows Keycloak to be highly configurable, but also fairly easy to install and setup.</p>
<p>More information about Keycloak can be found here: https://www.keycloak.org</p>
<h3 id="basic-workflow">Basic Workflow</h3>
<p>The basic workflow when authenticating to a service that uses SAML/OIDC from the users perspective is as follows:</p>
<ol>
<li>
<p>User goes to the web address of a SSO protected service (known as a service provider, or SP).</p>
</li>
<li>
<p>The service asks for a cookie that the users browser may have stored, which contains a token, if it finds a valid token in the browser, it logs the user in. If this token does not exist, or is invalid, the service directs the user to the configured Identity Provider (IDp).</p>
</li>
<li>
<p>Once the user reaches the IDp, they are presented with a login page.</p>
</li>
<li>
<p>The user logins in with their provided credentials. The IDp can compare these to locally stored credentials, or against an LDAP or Kerberos backend.</p>
</li>
<li>
<p>The user is given a token upon successful login which is stored as a cookie in the browser, and gets automatically redirected back to the service they were initially attempting to access. This token usually contains a username, as well as information regarding what the user has access to, if using a protocol like OIDC that supports authorization.</p>
</li>
<li>
<p>The service requests the token from the browser and if the token is valid, allows the user access to the service.</p>
</li>
</ol>
<p>Token validation is done on a secure back channel between the service and Identity Provider, without involving the browser, to increase security.</p>Jax GauthierIn this series, we will take a look at Keycloak, an open-source single sign on solution similar to Microsoft’s ADFS product.Backing up FreeNAS to Backblaze B22017-09-27T19:09:00-04:002017-09-27T19:09:00-04:00https://homelab.blog/blog/freenas-b2-backup<p>If you use FreeNAS, it’s probably because you care about your data. Part of data security is ensuring the availability of your data. To that end, you need to ensure that said data is backed up. There are generally two reasonable ways to backup your data from FreeNAS. One, local backup (using ZFS replication), and two, cloud backup.</p>
<p>In this article, we will look at setting up cloud backups to Backblaze B2, an economical cloud backup solution similar to Amazon S3.</p>
<h2 id="step-1-sign-up">Step 1: Sign up</h2>
<p>Sign up for a Backblaze account <a href="https://www.backblaze.com/b2/sign-up.html">here</a>. Once you have created an account, go to the “My Settings” tab, and under “Enabled Products”, check the box beside B2 Cloud Storage. This enables your account for using B2.</p>
<h2 id="step-2-create-a-bucket">Step 2: Create a Bucket</h2>
<p>Once you have enabled your account for B2, you need to create a bucket (where your files are stored). To do this, on the left side of the screen, select “Buckets” under B2 Cloud Storage. Then, select Create a Bucket. Also on this page, be sure to click on “Show Account ID and Application Key”, and mark down your Account ID and click “Create Application Key”. Also mark this down, as you will not be able to see it again, and will need it when we setup rclone in a later step.</p>
<h3 id="step-2a-setup-caps-and-alerts">Step 2a: Setup Caps and Alerts</h3>
<p>While this step is optional, it is highly recommended so that you get notified about any charges against your account that you may not be expecting. I set mine to a cap of $1 a day for each section. This will give you 6Tb of storage, and a good number of API calls.</p>
<h3 id="step-2b-have-a-look-around-your-account">Step 2b: Have a look around your account</h3>
<p>Have a look around your Backblaze account, there is a great get started guide available <a href="https://www.backblaze.com/b2/docs/">here</a>.</p>
<h2 id="step-3-setting-up-a-freebsd-jail-on-freenas">Step 3: Setting up a FreeBSD Jail on FreeNAS</h2>
<ol>
<li>Login to your FreeNAS GUI, and go to the Jails section.</li>
<li>Click “Add Jail”.</li>
<li>Enter a Jail Name. I called mine “b2-backups”.</li>
<li>Click Ok, and your jail will be created. Note that this may take a little bit of time. You should be able to close the dialog box if needed, the jail will be created in the background.</li>
<li>Click on your Jail and click the “shell” button in the bottom left. This will open a shell session to the Jail.</li>
<li>Enter <code class="language-plaintext highlighter-rouge">vi /etc./rc.conf</code> and change <code class="language-plaintext highlighter-rouge">sshd_enable="NO"</code> to <code class="language-plaintext highlighter-rouge">sshd_enable="YES"</code>. This will enable SSH to the jail.</li>
</ol>
<p>! FreeBSD uses vim as a text editor, use <code class="language-plaintext highlighter-rouge">i</code> to insert text, <code class="language-plaintext highlighter-rouge">del</code> to delete the rest of the line, and the arrow keys to scroll. Save and exit by pressing the <code class="language-plaintext highlighter-rouge">ESC</code> key and then <code class="language-plaintext highlighter-rouge">:we</code> to save and quit.</p>
<p>! You will need to run <code class="language-plaintext highlighter-rouge">passwd root</code> and reboot in order to have SSH access, as well as <code class="language-plaintext highlighter-rouge">PermitRootLogin yes</code> in <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd_config</code>.</p>
<p>!!!! At this point, you can switch over to SSH, if you prefer that to the shell in the FreeNAS GUI.</p>
<ol>
<li>Install <code class="language-plaintext highlighter-rouge">wget</code> using <code class="language-plaintext highlighter-rouge">pkg install wget</code>, this will allow you to download the rclone binary.</li>
<li>Download the latest rclone binary: <code class="language-plaintext highlighter-rouge">cd /tmp && wget https://downloads.rclone.org/rclone-v1.37-freebsd-amd64.zip</code>.</li>
<li>run <code class="language-plaintext highlighter-rouge">unzip rclone-v1.37-freebsd-amd64.zip</code> to extract the binary.
!!!! rclone version 1.37 is the latest stable release at the time of this writing</li>
<li>Copy the rclone executable to <code class="language-plaintext highlighter-rouge">/usr/bin</code> by running <code class="language-plaintext highlighter-rouge">cd rclone-v1.37-freebsd-amd64 && cp ./rclone /usr/bin</code></li>
</ol>
<h3 id="step-3a-adding-storage-to-the-jail">Step 3a: Adding storage to the Jail</h3>
<ol>
<li>Create a new folder structure in the Jail, I put mine in <code class="language-plaintext highlighter-rouge">/mnt/storage</code>, where you will mount your FreeNAS datastores. It is a good idea to make a folder for each dataset you want to mount.</li>
<li>In the FreeNAS GUI, go to the Jails tab, and then the Storage sub-tab.</li>
<li>Click “Add Storage”</li>
<li>Select the Jail you want to add the storage to.</li>
<li>Select the source dataset.</li>
<li>Select the destination (this will be the folder structure in the jail that you created in Step 3a-1).</li>
<li>Optionally select read-only.</li>
<li>Leave “Create Directory” selected.</li>
<li>Click “Ok”.</li>
<li>Repeat steps 3a-4 to 3a-9 for each dataset you want to backup to B2.</li>
</ol>
<h2 id="step-4-configuring-rclone">Step 4: Configuring rclone</h2>
<ol>
<li>Run <code class="language-plaintext highlighter-rouge">rclone config</code> to initiate the configuration of rclone</li>
<li>Press <code class="language-plaintext highlighter-rouge">n</code> to create a new remote (a remote is what rclone uses to know where to copy/sync your files).</li>
<li>Enter a name, I choose <code class="language-plaintext highlighter-rouge">b2</code>.</li>
<li>Press <code class="language-plaintext highlighter-rouge">3</code>.</li>
<li>Enter your account ID from your B2 account.</li>
<li>Enter your application Key from your B2 account.</li>
<li>Leave endpoint blank.</li>
<li>Press <code class="language-plaintext highlighter-rouge">y</code> to save the config.</li>
</ol>
<h3 id="step-4a-configuring-encryption">Step 4a: Configuring encryption</h3>
<ol>
<li>Follow steps 1-3 from Step 4.
! Note, name this new remote different than the previous remote.</li>
<li>Press <code class="language-plaintext highlighter-rouge">5</code>.</li>
<li>Enter the name of the remote you created in Step 4, number 3, followed by the name of your bucket. For example, <code class="language-plaintext highlighter-rouge">b2:storage</code> in my case.</li>
<li>Choose whether or not you want to encrypt the file names, selecting <code class="language-plaintext highlighter-rouge">1</code> does not encrypt file names. Selecting <code class="language-plaintext highlighter-rouge">2</code> encrypts the file names. I choose <code class="language-plaintext highlighter-rouge">2</code>.</li>
<li>Choose <code class="language-plaintext highlighter-rouge">y</code> to type in your own password, choose <code class="language-plaintext highlighter-rouge">g</code> to generate a strong password randomly. If you choose <code class="language-plaintext highlighter-rouge">g</code>, you are given an option as to how strong of a password you want to generate.</li>
<li>Create a password for the salt. This is recommended if you have chosen to enter your own password in the previous section. Note that for security, these passwords should not be the same.</li>
<li>Select <code class="language-plaintext highlighter-rouge">y</code> to accept the configuration.</li>
</ol>
<p>!! Note: The rclone config file is not encrypted by default, and Application Keys and your encryption passwords are stored in plaintext. It is recommended to set a password for the config file, and/or ensure the security of the rclone.conf file.</p>
<p>!! If you need to recover encrypted files from B2, you NEED both passwords (if you set two), otherwise your files will be completely unrecoverable.</p>
<h3 id="step-4b-creating-the-bash-script">Step 4b: Creating the bash script</h3>
<p>In this section, we will look at creating the bash script we will use with cron in order to backup any changes to our local storage to B2.</p>
<ol>
<li>Create a new file in <code class="language-plaintext highlighter-rouge">/root</code>, I called mine <code class="language-plaintext highlighter-rouge">rclone-cron.sh</code>.</li>
<li>Copy the following:</li>
</ol>
<p><code class="language-plaintext highlighter-rouge">!/bin/sh
if pidof -o %PPID -x "rclone-cron.sh"; then
exit 1
fi
echo starting storage sync
rclone copy {/path/to/local/storage} {name of your crypt remote}: -v --log-file={/path/to/log/file} --min-age 15m --copy-links
exit</code></p>
<p>Let’s break that down a bit and look at what the script actually does.</p>
<p><code class="language-plaintext highlighter-rouge">!/bin/sh</code> - run the script with the <code class="language-plaintext highlighter-rouge">sh</code> terminal.</p>
<p><code class="language-plaintext highlighter-rouge">if pidof -o %PPID -x "rclone-cron.sh"; then</code> - If the script is currently being run, then:</p>
<p><code class="language-plaintext highlighter-rouge">exit 1</code> - Do not run the script currently. This is good if your initial backup will take a while to run, as it won’t try to run rclone again.</p>
<p><code class="language-plaintext highlighter-rouge">fi</code> - closes the if statement.</p>
<p><code class="language-plaintext highlighter-rouge">echo start storage sync</code> - print to the terminal that the clone is starting.</p>
<p><code class="language-plaintext highlighter-rouge">rclone copy {/path/to/local/storage} {name of your crypt remote}: -v --log-file={/path/to/log/file} --min-age 15m --copy-links</code> - runs <code class="language-plaintext highlighter-rouge">rclone</code> with the copy parameter (does not delete files deleted locally, alternatively change <code class="language-plaintext highlighter-rouge">copy</code> to <code class="language-plaintext highlighter-rouge">sync</code> to keep an exact copy on B2 (deletes files from B2 that are deleted locally)). Uses the <code class="language-plaintext highlighter-rouge">-v</code> flag for verbosity. <code class="language-plaintext highlighter-rouge">--log-file={/path/to/log/file}</code> Tells rclone where to create a log file. <code class="language-plaintext highlighter-rouge">--min-age 15m</code> Tells rclone not to sync files less than 15 minutes old, useful to ensure copied files are probably complete, instead of semi-completed. <code class="language-plaintext highlighter-rouge">--copy-links</code> Tells rclone to follow slinks.</p>
<p><code class="language-plaintext highlighter-rouge">exit</code> - exits the script when the copy is finished.</p>
<h3 id="step-4c-creating-the-cron-entry">Step 4c: Creating the cron entry</h3>
<ol>
<li>Run <code class="language-plaintext highlighter-rouge">crontab -e</code> to open the cron editor.</li>
<li>Enter <code class="language-plaintext highlighter-rouge">0 1 * * * /root/rclone-cron.sh</code> - This will run the script we created in 4b once a day.</li>
</ol>
<h2 id="step-5-run-the-script">Step 5: Run the script!</h2>
<ol>
<li><code class="language-plaintext highlighter-rouge">chmod +x /root/rclone-cron.sh</code> - makes the script executable</li>
<li><code class="language-plaintext highlighter-rouge">cd /root/ && ./rclone-cron.sh</code> - runs the script.
! rclone does not run in the background. It is recommended to run the script in tmux or similar, or wait for the crontab to run, as the initial backup will probably take a long time if you have a lot of data like I do.</li>
</ol>
<h3 id="step-5a-check-b2-console-to-see-if-its-working">Step 5a: Check B2 console to see if it’s working.</h3>
<ol>
<li>Log into your back blaze account, and take a look at your bucket. You should see that files are being copied to B2.</li>
</ol>
<p>This completes the guide on setting up rclone to backup to B2 on FreeNAS. Rclone can backup to many cloud providers, have a look at different providers if Backblaze is not your cup of tea.</p>Jax GauthierIf you use FreeNAS, it’s probably because you care about your data. Part of data security is ensuring the availability of your data. To that end, you need to ensure that said data is backed up. There are generally two reasonable ways to backup your data from FreeNAS. One, local backup (using ZFS replication), and two, cloud backup.