This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Setup

Install and configure the Capsule Proxy

1 - Installation

Installation guide for the capsule-proxy

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed CustomResourceDefinitions. The following Artifacthub repository are official:

Perform the following steps to install the capsule Operator:

  1. Add repository:

     helm repo add projectcapsule https://projectcapsule.github.io/charts
    
  2. Install Capsule-Proxy:

     helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system --create-namespace
    

    or (OCI)

     helm install capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy -n capsule-system --create-namespace
    
  3. Show the status:

     helm status capsule-proxy -n capsule-system
    
  4. Upgrade the Chart

     helm upgrade capsule-proxy projectcapsule/capsule-proxy -n capsule-system
    

    or (OCI)

     helm upgrade capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy --version 0.13.0
    
  5. Uninstall the Chart

     helm uninstall capsule-proxy -n capsule-system
    

GitOps

There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.

ArgoCD

Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: capsule
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: system
  source:
    repoURL: ghcr.io/projectcapsule/charts
    targetRevision: 0.12.4 
    chart: capsule
    helm:
      valuesObject:
        ...
        proxy:
          enabled: true
          webhooks:
            enabled: true
          certManager:
            generateCertificates: true
          options:
            generateCertificates: false
            oidcUsernameClaim: "email"
            extraArgs:
            - "--feature-gates=ProxyClusterScoped=true"
          serviceMonitor:
            enabled: true
            annotations:
              argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

  destination:
    server: https://kubernetes.default.svc
    namespace: capsule-system

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - ServerSideApply=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
  name: capsule-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: ghcr.io/projectcapsule/charts
  name: capsule
  project: system
  type: helm
  enableOCI: "true"

FluxCD

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: capsule
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  targetNamespace: "capsule-system"
  interval: 10m
  releaseName: "capsule"
  chart:
    spec:
      chart: capsule
      version: "0.12.4"
      sourceRef:
        kind: HelmRepository
        name: capsule
      interval: 24h
  install:
    createNamespace: true
  upgrade:
    remediation:
      remediateLastFailure: true
  driftDetection:
    mode: enabled
  values:
    proxy:
      enabled: true
      webhooks:
        enabled: true
      certManager:
        generateCertificates: true
      options:
        generateCertificates: false
        oidcUsernameClaim: "email"
        extraArgs:
        - "--feature-gates=ProxyClusterScoped=true"
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: capsule
  namespace: flux-system
spec:
  type: "oci"
  interval: 12h0m0s
  url: oci://ghcr.io/projectcapsule/charts

Considerations

Consdierations when deploying capsule-proxy

Exposure

Depending on your environment, you can expose the capsule-proxy by:

  • Ingress
  • NodePort Service
  • LoadBalance Service
  • HostPort
  • HostNetwork

Here how it looks like when exposed through an Ingress Controller:

                +-----------+          +-----------+         +-----------+
 kubectl ------>|:443       |--------->|:9001      |-------->|:6443      |
                +-----------+          +-----------+         +-----------+
                ingress-controller     capsule-proxy         kube-apiserver

User Authentication

The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server.

It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.

If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.

Certificate Management

By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule.This relates to certifiacates for the proxy and the admissions server. However, you can also use a job to generate self-signed certificates and store them in a Kubernetes Secret:

options:
  generateCertificates: true
certManager:
  generateCertificates: false  

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

HTTP Support

NOTE: kubectl will not work against a http server.

Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.

After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.

For example:

TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces

Metrics

Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.

The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.

Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.

capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:

path: the HTTP path of every single request that Capsule Proxy passes to the upstream capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.

path: the HTTP path of every single request that Capsule Proxy passes to the upstream status: the HTTP status code of the request

2 - Controller Options

Configure the Capsule Proxy Controller

You can customize the Capsule Proxy with the following configurations.

Controller Options

You can provide additional options via the helm chart:

options:
  extraArgs:
    - --disable-caching=true

Options are also available as dedicated configuration values:

# Controller Options
options:
  # -- Set the listening port of the capsule-proxy
  listeningPort: 9001
  # -- Set leader election to true if you are running n-replicas
  leaderElection: false
  # -- Set the log verbosity of the capsule-proxy with a value from 1 to 10
  logLevel: 4
  # -- Name of the CapsuleConfiguration custom resource used by Capsule, required to identify the user groups
  capsuleConfigurationName: default
  # -- Define which groups must be ignored while proxying requests
  ignoredUserGroups: []
  # -- Specify if capsule-proxy will use SSL
  oidcUsernameClaim: preferred_username
  # -- Specify if capsule-proxy will use SSL
  enableSSL: true
  # -- Set the directory, where SSL certificate and keyfile will be located
  SSLDirectory: /opt/capsule-proxy
  # -- Set the name of SSL certificate file
  SSLCertFileName: tls.crt
  # -- Set the name of SSL key file
  SSLKeyFileName: tls.key
  # -- Specify if capsule-proxy will generate self-signed SSL certificates
  generateCertificates: false
  # -- Specify additional subject alternative names for the self-signed SSL
  additionalSANs: []
  # -- Specify an override for the Secret containing the certificate for SSL. Default value is empty and referring to the generated certificate.
  certificateVolumeName: ""
  # -- Set the role bindings reflector resync period, a local cache to store mappings between users and their namespaces. [Use a lower value in case of flaky etcd server connections.](https://github.com/projectcapsule/capsule-proxy/issues/174)
  rolebindingsResyncPeriod: 10h
  # -- Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector.
  disableCaching: false
  # -- Enable the rolebinding reflector, which allows to list the namespaces, where a rolebinding mentions a user.
  roleBindingReflector: false
  # -- Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
  authPreferredTypes: "BearerToken,TLSCertificate"
  # -- QPS to use for interacting with Kubernetes API Server.
  clientConnectionQPS: 20
  # -- Burst to use for interacting with kubernetes API Server.
  clientConnectionBurst: 30
  # -- Enable Pprof for profiling
  pprof: false

The following options are available for the Capsule Proxy Controller:

      --auth-preferred-types string           Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
                                              First match is used and can be specified multiple times as comma separated values or by using the flag multiple times. (default "[TLSCertificate,BearerToken]")
      --capsule-configuration-name string     Name of the CapsuleConfiguration used to retrieve the Capsule user groups names (default "default")
      --capsule-user-group strings            Names of the groups for capsule users (deprecated: use capsule-configuration-name)
      --client-connection-burst int32         Burst to use for interacting with kubernetes apiserver. (default 30)
      --client-connection-qps float32         QPS to use for interacting with kubernetes apiserver. (default 20)
      --disable-caching                       Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector (default: false)
      --enable-leader-election                Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
      --enable-pprof                          Enables Pprof endpoint for profiling (not recommend in production)
      --enable-reflector                      Enable rolebinding reflector. The reflector allows to list the namespaces, where a rolebinding mentions a user
      --enable-ssl                            Enable the bind on HTTPS for secure communication (default: true) (default true)
      --feature-gates mapStringBool           A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
                                              AllAlpha=true|false (ALPHA - default=false)
                                              AllBeta=true|false (BETA - default=false)
                                              ProxyAllNamespaced=true|false (ALPHA - default=false)
                                              ProxyClusterScoped=true|false (ALPHA - default=false)
                                              SkipImpersonationReview=true|false (ALPHA - default=false)
      --ignored-impersonation-group strings   Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
      --ignored-user-group strings            Names of the groups which requests must be ignored and proxy-passed to the upstream server
      --impersonation-group-regexp string     Regular expression to match the groups which are considered for impersonation
      --listening-port uint                   HTTP port the proxy listens to (default: 9001) (default 9001)
      --metrics-addr string                   The address the metric endpoint binds to. (default ":8080")
      --oidc-username-claim string            The OIDC field name used to identify the user (default: preferred_username) (default "preferred_username")
      --rolebindings-resync-period duration   Resync period for rolebindings reflector (default 10h0m0s)
      --ssl-cert-path string                  Path to the TLS certificate (default: /opt/capsule-proxy/tls.crt)
      --ssl-key-path string                   Path to the TLS certificate key (default: /opt/capsule-proxy/tls.key)
      --webhook-port int                      The port the webhook server binds to. (default 9443)
      --zap-devel                             Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder                   Zap log encoding (one of 'json' or 'console')
      --zap-log-level level                   Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level            Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding       Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

Feature GateDefault ValueDescription
SkipImpersonationReviewfalseSkipImpersonationReview allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems.
ProxyClusterScopedfalseProxyClusterScoped allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings