[How To] Prevent Container with Privileged Mode to Run on Kubernetes Cluster

My Kubernetes exploration brought me to the topic of privileged pods. Privileged pod, or container running in privileged mode is a configuration option of K8s deployment which can be useful, but can also be dangerous. 

This is an excerpt of deployment specification where privileged mode defined.

kind: Deployment
...
spec:
  template:
    spec:
      containers:
      - name: ...
        image: nginx:1.14.2
        securityContext:
          privileged: true

Found this article:

https://www.cncf.io/blog/2020/10/16/hack-my-mis-configured-kubernetes-privileged-pods/

which explains about what's the true intent of running privileged pod and the security risk it caused, including how to exploit privileged pod to do malicious intent.

Now the question is, how can we prevent container with privileged mode to run on our Kubernetes cluster?

[lunar.lab] Cannot Resolve ".local" Domain from TKGm Workload Cluster

Problem Statement

• Kubernetes Pod Status ImagePullBackOff 
• Describe pod show error message:

dial tcp: lookup harbor-01a.corp.local: Temporary failure in name resolution

• Container image pulled from local container registry with ".local" domain suffix

[lunar.lab] Allow TKGm Workload Cluster to Pull Image from Harbor Configured with Self-signed Certificate

Disclaimer

• This method is kind of a hack and hence ** Unsupported **.
• I do this only within my lab or PoC with controlled environment.

Problem Statement

TKGm Workload Cluster do not allow pulling image from Container Registry configured with Self-signed Certificate.

Doing such thing will throw error message as follows:

x509: certificate signed by unknown authority

[How To] Enhance Online Boutique App to Use Persistent Volume

Online Boutique (https://github.com/GoogleCloudPlatform/microservices-demo) is a web-based e-commerce microservices demo app built by folks at Google. I use this as demo app to deploy on top of Tanzu Kubernetes platform. One of the demo scenario I do is how to consume vSphere datastore as persistent storage for Kubernetes app, in easy, on-demand, fully automated, and scalable fashion. This can be done by a feature called Cloud Native Storage (CNS). Read more about CNS here:

https://blogs.vmware.com/virtualblocks/2019/08/14/introducing-cloud-native-storage-for-vsphere/

One of Online Boutique service is redis-cart. This is the service in charge for Shopping Cart. If any item added to Shopping Cart, the record will be handled by this service. With default configuration, the data volume used by redis-cart do not use persistent volume. If redis-cart is failed, Shopping Cart data will be lost. This article explains how to alter this and use vSphere datastore to provide persistent storage for redis-cart service.

[How To] Avoid Hitting Docker Pull Rate Limit by Authenticate Pull Request

When demoing kubernetes platform, I definitely need sample application to deploy. There are some great reference here: https://williamlam.com/2020/06/interesting-kubernetes-application-demos.html, where most of the source container images are coming from Docker registry. If you try to deploy the app manifests, you might hit error like the following:

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit.

[lunar.lab] Deploy TKG Management Cluster on vSphere

Now all the preparation completed, I finally able to deploy TKG management cluster. The recommended (and easiest) way to do this for the first time is using the installer interface. From the bootstrap machine prepared earlier (https://dy.si/TAg1M72), I type this:

tanzu management-cluster create --ui --browser none --bind 192.168.110.101:8081

• 192.168.110.101 is my bootstrap machine IP Address
• I set the installer interface to be accessible on port 8081
• --browser none to tell the installer not to open browser locally as my bootstrap machine is without GUI
• Open http://192.168.110.101:8081 on a browser with network access to bootstrap machine
• As always, this is the official documentation that I refer to: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-deploy-ui.html

Step 1 - IaaS Provider

[lunar.lab] Prepare to Deploy TKG Management Clusters to vSphere

• Created a base image template that matches the management cluster’s Kubernetes version.
• Official documentation: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-mgmt-clusters-vsphere.html#import-a-base-image-template-into-vsphere-4
• Import latest available version and -1 to test version upgrade
• In my case for TKGm 1.5.1, I import version Kubernetes v1.21.8 and v1.22.5
• I found out that Ubuntu template is not recognized when trying to deploy workload cluster from TMC, so I also import Photon image.