OCI – Oracle Container Engine for Kubernetes (OKE) – Using Client Certificates and RBAC

Kubernetes has been proven the best way that we have today to run microservices deployments, whether it is via a Serverless approach or by managing your own deployments in the cluster. This has solidified with the strong adoption of Kubernetes by all the major Cloud Vendors, as the strategic way to orchestrate containers and run serverless functions.

However, one of the situations that we need to be mindful, is that kubernetes creates by default a super powerful user that has full access to almost every resource in the cluster (accessible via kubectl or directly though APIs). This is very convenient for most dev & test scenarios, but it is imperative that for production workloads, we limit such power and use Role Base Access Control (RBAC), stable since version 1.8, for fine-grained authorisation access control to kubernetes cluster resources.

For the purpose of this demo, I am assuming some familiarity with Kubernetes and kubectl. I will mainly focus on the Authentication and Authorisation aspects that allow us to use Client certificates to get access to protected resources in a Kubernetes cluster.

In a nutshell this is what I am going to do:

  1. Create and use Client certificates to authenticate into a Kubernetes cluster
  2. Create a Role Base Access Control to fine grain authorise resources in the Kubernetes cluster
  3. Configure kubectl with the new security context, to properly limit access to resources in the Kubernetes cluster.

This is a super simplified visual representation:

Going a bit deeper into the actual steps to achieve this outcome, I need to highlight a couple of points:

  • In order to create the client certificate to authenticate into Kubernetes, we need to:
    • Create a new RSA private key for our new user. Let’s call it id_rsa.pri
    • Then, we will use this private key to create a new certificate signing request (CSR). Let’s call it id_rsa.csr
    • Then, we will use our CSR to create a CertificateSigningRequest in Kubernetes, so that it produces a Pending Certificate, to then be approved or rejected by the cluster administrator. Cool huh?
    • Finally, we need to approve the pending Certificate, so that Kubernetes can issue our full certificate to authenticate against the cluster.

  • Once we the Kubernetes cluster has approved and issued certificate, we need to configure kubectl to use our private key in the context of the cluster.

    For this, in the kubeconfig used by kubectl:

    • Using id_rsa.pri, we need to configure a new set of credentials.
    • Then we need to associate the new set of credentials into a new context in the cluster
    • Finally, we will swap context from the super powerful user to our better controlled mortal one.
  • Ok, that’s all good for Authentication, but we also need to Authorise access to resources. As part of this, we need to create roles and role bindings in the kubernetes cluster. These have to map the user or group used originally in the certificate signing request, with a set of rules or permission to access resources in the cluster.

    Wow, that sounded a bit complex, but it is not. Let’s do it in practice to make it easier to digest.

Ok, let’s have fun!!!

Before we start

In this blog I assume the following:

  • You have an Oracle Cloud account, if not request a free account. Here: https://cloud.oracle.com/tryit
  • In this blog, I assume that you are familiar with basic Kubernetes concepts. If you are not, refer to one of our previous blogs.
  • I will try to cover the important bits and pieces from Kubernetes point of view, but if you wish more information, refer to the official documentation.

Let’s create a Client Certificate for Kubernetes Authentication

As discussed, this client certificate needs to be signed by the Kubernetes CA certificate, this is how our incoming user requests will be authenticated by Kubernetes.

  • The first thing we need to do is to create a new RSA Private key, let’s use openssl for that:

    openssl genrsa -out id_rsa.pri 2048

  • Now, let’s use this certificate to create a certificate signing request (CSR).
  • First, we need to create a file that will help us configure the CSR. You can gran it here.

    Make sure to replace your username and group in the fields CN and O – This is how Kubernetes will see you coming. These values need to match the name of the user or group used in Kubernetes while building the Role Bindings (we’ll do this later).

  • Now, let’s use that configuration file (csr.config) to generate the CSR.

    openssl req -new -key id_rsa.pri -out id_rsa.csr -config ./csr.conf

  • Now we need to sign our new CSR with the Kubernetes CA certificate. There are multiple ways to achieve this, the way I would recommend is submitting a formal CertificateSigningRequest into Kubernetes, which then needs to either be approved or rejected by the Kubernetes administrator.
  • We are going to submit a CertificateSigningRequest, by building a new YAML file. You can download a sample here.
  • Notice in line 6 that we have a command that basically extracts the signing request certificate. Let’s run this command ourselves and edit this YAML file to contain the real value as part of the certificate request.

    cat id_rsa.csr | base64 | tr -d ‘\n’

  • Copy this output and edit your YAML file replacing the actual command. Also feel free to rename your certificate signing request and namespace the way you want.

    Mine looks like this:

  • Now, let’s use kubectl to apply this certificate signing request. I saved this YAML file with the name cri_csr.yml

    kubectl apply -f cri_csr.yml

  • You can query all existing certificate signing requests. You will see your as Pending.

    kubectl get csr

    kubectl describe csr

  • Also notice that as part of the Subjects, it acknowledges that this certificate signing request is for:
    • User: carlos
    • Organization: carlosgroup

    Just like we defined while building our certificate signing request.

  • Obviously at this point, it is up to the master Kubernetes administrator to either accept or reject our CSR. However, given we are that person too, let’s approve it =)

    kubectl certificate approve [csr_name]

Where [csr_name] is the full name that we gave our CertificateSigningRequest earlier.

  • Now if we get our CSR again it will appear as Approved and Issued!

  • Great! Now that our certificate has been issued, let’s simply download it.

    kubectl get csr [csr_name] -o jsonpath='{.status.certificate}’ | base64 –decode > id_rsa.crt

Where:

  • [csr_name]: Is the full name I gave previously to my CertificateSigningRequest
  • id_rsa.crt: Is the name of the result certificate that we will use for further authentication.

  • Congratulations, we have the final client certificate, for user carlos, as part of organisation carlosgroup and it is signed by the Kubernetes cluster CA certificate, so when exchanged with the Kubernetes cluster we get properly authenticated. The next step is to use this certificate to configure a new user context in kubectl.
  • If you want to, you can decode your certificate. It will show you things like the subject, validity, issuer, etc.

    openssl x509 -in id_rsa.crt -text -noout

Let’s configure kubectl to use the new client certificate

In this section we are going to configure kubectl to use the client certificate as part of a new context.

  • Let’s create a new user associated with our client certificate.

    kubectl config set-credentials carlos –client-certificate=id_rsa.crt –client-key=id_rsa.pri –embed-certs=true

    Where:

  • carlos is the nickname
  • id_rsa.crt is our client certificate – Notice that this is optional, as the server already has knowledge of our CSR. We could only use the private key at this stage.
  • id_rsa.pri is our private key – This one is mandatory of course.
  • You can see your new user if you look into the kube config $HOME/.kube/config file and search for users.

  • Optionally, you can use the command:

    kubectl config view

  • Now, let’s associate our new user with the cluster. First, we need the cluster name. For this, once again run kubectl config view or go into your current admin kube config $HOME/.kube/config and get the cluster name.

  • Now let’s associate the user to the cluster

    kubectl config set-context carlos –cluster=[CLUSTER-NAME] –user=carlos

  • If you check the $HOME/.kube/config again, you will see that your username is now associated within the context of the kubernetes cluster.

  • We are ready to swap contexts and prove our new user, but first, let’s prove that our still super powerful kubectl user can bring us information. Let’s bring all pods.

    kubectl get pods –all-namespaces


Obviously depending on what you are currently running on your cluster, you will see the outcome. I have lots of pods, as you can see, some of them running for almost a year.

  • Now, let’s try to get all pods using the “carlos” context. For this you can either swap the context:

    kubectl config use-context carlos

  • Or, simply, add the –context flag at the end to specify a different context:

    kubectl get pods –context=carlos

Notice that we are authenticated, but not authorised, as it is forbidden to access such resources:

  • Why? You may ask… I’m glad you did! This is exactly what we were expecting. With our client certificate, we can authenticate properly, given our certificate was signed by the CA certificate of the kubernetes cluster. However, we have not yet created the roles and binding needed to provide fine-grain control over resources in kubernetes. That is what we are going to do next in the following section.

Let’s create a Role and Role Binding in Kubernetes to Authorise access to resource

We are in the last step of our blog, so far, we have successfully created a client certificate signed by the Kubernetes cluster CA certificate. We have also configured this certificate under a new context in kubectl, so that we can simply swap from our ultra-powerful user into a mortal user with specific permissions. For this, we are going to create a role and role binding in Kubernetes, associating what our new user can do.

  • Let’s firs create a ClusterRole, that will give us ability to access cluster-wide across all namespaces. Depending on the use case, it might be better to create a Role, instead of a ClusterRole, that way we can limit the power of user by namespaces. In this example, I want to get all Pods cluster-wide, that’s why I am using a ClusterRole.

    You can grab this file here.

Important lines to mention out of this file:

  • Line 4: Set the name of the cluster role that you want.
  • Lines 7 and 9: We are creating a rule on pods, services and deployments across the cluster (all namespaces)
  • Line 8 and 11: We are allowing certain verbs on these resources.
  • We need to apply this file. For this you might need to use the original admin context in kubectl, otherwise you won’t have permission either as you are still under “carlos” context. You can grab the name of your original context in $HOME/.kube/config or by calling kubectl config view

    kubectl apply -f cri_clusterRole.yml –context=context [Admin-Context]

  • Finally, let’s create the role binding that maps our incoming user carlos, from carlosgroup to bind our previously created ClusterRole. Similarly, as Roles, depending on the use case, you might want to limit role bindings based on namespaces, in which case you will use a RoleBinding, rather than a ClusterRoleBinding. In this example, I am targeting all global contexts, so I am using a ClusterRoleBinding

    You can get this file here.

Important lines to mention out of this file:

  • Line 4: Set the name of the cluster role binding that you want.
  • Line 6: You can apply subjects buy “User” or “Group”. In this case I am creating a User binding
  • Line 7: This is important. This is the name of the user or the name of the organisation that you used when creating your certificate signing request. Make sure they match. Notice that it is case sensitive. If you chose Group, then any user who belongs to carlosgroup organisation will be bound. Otherwise if you used User, like I did, only carlos will be authorised.
  • Line 11: This is pointing to the ClusterRole that we created previously. That gives permission to read, list and watch pods.

In other words, what we are implying is that only carlos (if used as User) or any user who belongs to carlosgroup organisation (if used Group), will have permission to read, list and watch Pods cluster wide.

  • Ok, let’s apply this file.

    kubectl apply -f cri_clusterRoleBinding.yml

  • Great, so everything is in order to switch back to our new context and try to get all Pods again.

    kubectl config use-context carlos

  • Now, we should be authorised to see all Pods cluster-wide

    kubectl get pods –all-namespaces


Amazing! Our new user can only access the allowed resources in the Kubernetes cluster, congratulations!!!

I hope you found this blog useful. If you have any question or comment, feel free to contact me directly at https://www.linkedin.com/in/citurria/

Thanks for your time.

Author: Carlos Rodriguez Iturria

I am extremely passionate about people, technology and the most effective ways to connect the two by sharing my knowledge and experience. Working collaboratively with customers and partners inspires and excites me, especially when the outcome is noticeable valuable to a business and results in true innovation. I enjoy learning and teaching, as I recognise that this is a critical aspect of remaining at the forefront of technology in the modern era. Over the past 10+ years, I have developed and defined solutions that are reliable, secure and scalable, working closely with a diverse range of stakeholders. I enjoy leading engagements and am very active in the technical communities – both internal and external. I have stood out as a noticeable mentor running technology events across major cities in Australia and New Zealand, including various technology areas such as, Enterprise Integrations, API Management, Cloud Integration, IaaS and PaaS adoption, DevOps, Continuous Integration, Continuous Automation among others. In recent years, I have shaped my role and directed my capabilities towards educating and architecting benefits for customers using Oracle and AWS Cloud technologies. I get especially excited when I am able to position both as a way to exceed my customers’ expectations. I hold a bachelor degree in Computer Science and certifications in Oracle and AWS Solutions Architecture.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s