Proxy

• 1: Setup
• 1.1: Installation
• 1.2: Controller Options
• 2: ProxySettings
• 3: Gangplank
• 4: API Reference

Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope

1 - Setup

1.1 - Installation

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed CustomResourceDefinitions. The following Artifacthub repository are official:

• Artifacthub Page (OCI)
• Artifacthub Page (Legacy - Best Effort)

Perform the following steps to install the capsule Operator:

1. Add repository:

    helm repo add projectcapsule https://projectcapsule.github.io/charts
   

2. Install Capsule-Proxy:

    helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system --create-namespace
   

   or (OCI)

    helm install capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy -n capsule-system --create-namespace
   

3. Show the status:

    helm status capsule-proxy -n capsule-system
   

4. Upgrade the Chart

    helm upgrade capsule-proxy projectcapsule/capsule-proxy -n capsule-system
   

   or (OCI)

    helm upgrade capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy --version 0.13.0
   

5. Uninstall the Chart

    helm uninstall capsule-proxy -n capsule-system
   

GitOps

There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.

ArgoCD

Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: capsule
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: system
  source:
    repoURL: ghcr.io/projectcapsule/charts
    targetRevision: 0.12.4 
    chart: capsule
    helm:
      valuesObject:
        ...
        proxy:
          enabled: true
          webhooks:
            enabled: true
          certManager:
            generateCertificates: true
          options:
            generateCertificates: false
            oidcUsernameClaim: "email"
            extraArgs:
            - "--feature-gates=ProxyClusterScoped=true"
          serviceMonitor:
            enabled: true
            annotations:
              argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

  destination:
    server: https://kubernetes.default.svc
    namespace: capsule-system

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - ServerSideApply=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
  name: capsule-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: ghcr.io/projectcapsule/charts
  name: capsule
  project: system
  type: helm
  enableOCI: "true"

FluxCD

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: capsule
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  targetNamespace: "capsule-system"
  interval: 10m
  releaseName: "capsule"
  chart:
    spec:
      chart: capsule
      version: "0.12.4"
      sourceRef:
        kind: HelmRepository
        name: capsule
      interval: 24h
  install:
    createNamespace: true
  upgrade:
    remediation:
      remediateLastFailure: true
  driftDetection:
    mode: enabled
  values:
    proxy:
      enabled: true
      webhooks:
        enabled: true
      certManager:
        generateCertificates: true
      options:
        generateCertificates: false
        oidcUsernameClaim: "email"
        extraArgs:
        - "--feature-gates=ProxyClusterScoped=true"
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: capsule
  namespace: flux-system
spec:
  type: "oci"
  interval: 12h0m0s
  url: oci://ghcr.io/projectcapsule/charts

Considerations

Consdierations when deploying capsule-proxy

Exposure

Depending on your environment, you can expose the capsule-proxy by:

• Ingress
• NodePort Service
• LoadBalance Service
• HostPort
• HostNetwork

Here how it looks like when exposed through an Ingress Controller:

                +-----------+          +-----------+         +-----------+
 kubectl ------>|:443       |--------->|:9001      |-------->|:6443      |
                +-----------+          +-----------+         +-----------+
                ingress-controller     capsule-proxy         kube-apiserver

User Authentication

The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server.

It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.

If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.

Certificate Management

By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule.This relates to certifiacates for the proxy and the admissions server. However, you can also use a job to generate self-signed certificates and store them in a Kubernetes Secret:

options:
  generateCertificates: true
certManager:
  generateCertificates: false  

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

• Kubernetes Reflector

HTTP Support

NOTE: kubectl will not work against a http server.

Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.

After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.

For example:

TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces

Metrics

Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.

The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.

Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.

capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:

path: the HTTP path of every single request that Capsule Proxy passes to the upstream capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.

path: the HTTP path of every single request that Capsule Proxy passes to the upstream status: the HTTP status code of the request

1.2 - Controller Options

You can customize the Capsule Proxy with the following configurations.

Controller Options

You can provide additional options via the helm chart:

options:
  extraArgs:
    - --disable-caching=true

Options are also available as dedicated configuration values:

# Controller Options
options:
  # -- Set the listening port of the capsule-proxy
  listeningPort: 9001
  # -- Set leader election to true if you are running n-replicas
  leaderElection: false
  # -- Set the log verbosity of the capsule-proxy with a value from 1 to 10
  logLevel: 4
  # -- Name of the CapsuleConfiguration custom resource used by Capsule, required to identify the user groups
  capsuleConfigurationName: default
  # -- Define which groups must be ignored while proxying requests
  ignoredUserGroups: []
  # -- Specify if capsule-proxy will use SSL
  oidcUsernameClaim: preferred_username
  # -- Specify if capsule-proxy will use SSL
  enableSSL: true
  # -- Set the directory, where SSL certificate and keyfile will be located
  SSLDirectory: /opt/capsule-proxy
  # -- Set the name of SSL certificate file
  SSLCertFileName: tls.crt
  # -- Set the name of SSL key file
  SSLKeyFileName: tls.key
  # -- Specify if capsule-proxy will generate self-signed SSL certificates
  generateCertificates: false
  # -- Specify additional subject alternative names for the self-signed SSL
  additionalSANs: []
  # -- Specify an override for the Secret containing the certificate for SSL. Default value is empty and referring to the generated certificate.
  certificateVolumeName: ""
  # -- Set the role bindings reflector resync period, a local cache to store mappings between users and their namespaces. [Use a lower value in case of flaky etcd server connections.](https://github.com/projectcapsule/capsule-proxy/issues/174)
  rolebindingsResyncPeriod: 10h
  # -- Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector.
  disableCaching: false
  # -- Enable the rolebinding reflector, which allows to list the namespaces, where a rolebinding mentions a user.
  roleBindingReflector: false
  # -- Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
  authPreferredTypes: "BearerToken,TLSCertificate"
  # -- QPS to use for interacting with Kubernetes API Server.
  clientConnectionQPS: 20
  # -- Burst to use for interacting with kubernetes API Server.
  clientConnectionBurst: 30
  # -- Enable Pprof for profiling
  pprof: false

The following options are available for the Capsule Proxy Controller:

      --auth-preferred-types string           Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
                                              First match is used and can be specified multiple times as comma separated values or by using the flag multiple times. (default "[TLSCertificate,BearerToken]")
      --capsule-configuration-name string     Name of the CapsuleConfiguration used to retrieve the Capsule user groups names (default "default")
      --capsule-user-group strings            Names of the groups for capsule users (deprecated: use capsule-configuration-name)
      --client-connection-burst int32         Burst to use for interacting with kubernetes apiserver. (default 30)
      --client-connection-qps float32         QPS to use for interacting with kubernetes apiserver. (default 20)
      --disable-caching                       Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector (default: false)
      --enable-leader-election                Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
      --enable-pprof                          Enables Pprof endpoint for profiling (not recommend in production)
      --enable-reflector                      Enable rolebinding reflector. The reflector allows to list the namespaces, where a rolebinding mentions a user
      --enable-ssl                            Enable the bind on HTTPS for secure communication (default: true) (default true)
      --feature-gates mapStringBool           A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
                                              AllAlpha=true|false (ALPHA - default=false)
                                              AllBeta=true|false (BETA - default=false)
                                              ProxyAllNamespaced=true|false (ALPHA - default=false)
                                              ProxyClusterScoped=true|false (ALPHA - default=false)
                                              SkipImpersonationReview=true|false (ALPHA - default=false)
      --ignored-impersonation-group strings   Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
      --ignored-user-group strings            Names of the groups which requests must be ignored and proxy-passed to the upstream server
      --impersonation-group-regexp string     Regular expression to match the groups which are considered for impersonation
      --listening-port uint                   HTTP port the proxy listens to (default: 9001) (default 9001)
      --metrics-addr string                   The address the metric endpoint binds to. (default ":8080")
      --oidc-username-claim string            The OIDC field name used to identify the user (default: preferred_username) (default "preferred_username")
      --rolebindings-resync-period duration   Resync period for rolebindings reflector (default 10h0m0s)
      --ssl-cert-path string                  Path to the TLS certificate (default: /opt/capsule-proxy/tls.crt)
      --ssl-key-path string                   Path to the TLS certificate key (default: /opt/capsule-proxy/tls.key)
      --webhook-port int                      The port the webhook server binds to. (default 9443)
      --zap-devel                             Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder                   Zap log encoding (one of 'json' or 'console')
      --zap-log-level level                   Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level            Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding       Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

2 - ProxySettings

The configuration for the Proxy is also declarative via CRDs. This allows both Administrators and Tenant Owners to create flexible rules.

GlobalProxysettings

As an administrator, you might have the requirement to allow users to query cluster-scoped resources which are not directly linked to a tenant or anything like that. In that case you grant cluster-scoped LIST privileges to any subject, no matter what their tenant association is. For example:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - "*"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

With this rule the User alice can list any cluster-scoped resource which match the selector condition. The apiGroups and resources work the same as known from Kubernetes ClusterRoles. All of these are valid expressions:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - ""
      resources:
      - "pods"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev
    - apiGroups:
      - "kyverno.io/v1"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

A powerful tool to enhance the user-experience for all your users.

Proxysettings

ProxySettings are created in a namespace of a tenant, if it’s not in a namespace of a tenant it’s not regarded as valid. With the ProxySettings Tenant Owners can further improve the experience for their fellow tenant users.

apiVersion: capsule.clastix.io/v1beta1
kind: ProxySettings
metadata:
  name: solar-proxy
  namespace: solar-prod
spec:
  subjects:
  - kind: User
    name: alice
    proxySettings:
    - kind: IngressClasses
      operations:
      - List

Primitives

This will be refactored

Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.

The proxy setting kind is an enum accepting the supported resources:

• Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
• StorageClasses: Perform actions on the allowed StorageClasses for the tenant
• IngressClasses: Perform actions on the allowed IngressClasses for the tenant
• PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant PriorityClasses
• RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
• PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant

Each Resource kind can be granted with several verbs, such as:

• List
• Update
• Delete

Special routes for kubectl describe

When issuing a kubectl describe node, some other endpoints are put in place:

• api/v1/pods?fieldSelector=spec.nodeName%3D{name}
• /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/{name}

These are mandatory to retrieve the list of the running Pods on the required node and provide info about its lease status.

3 - Gangplank

Gangplank is a web application that allows users to authenticate with an OIDC provider and configure their kubectl configuration file with the OpenID Connect Tokens. Gangplank is based on Gangway, which is no longer maintained.

Prerequisites

For Authentication you will need a Confidential OIDC client configured in your OIDC provider, such as Keycloak, Dex, or Google Cloud Identity. By default the Kubernetes API only validates tokens against a Public OIDC client, so you will need to configure your OIDC provider to allow the Gangplank client to issue tokens. You must make use of the Kubernetes Authentication Configuration, which allows to define multiple audiences (clients). This way we can issue tokens for a gangplank client, which is Confidential, and a kubernetes client, which is Public. The Kubernetes API will validate the tokens against both clients. The Config might look like this:

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://keycloak/realms/realm-name
    audiences:
    - kubernetes
    - gangplank
    audienceMatchPolicy: MatchAny # This one is important
  claimMappings:
    username:
      claim: 'email'
      prefix: ""
    groups:
      claim: 'groups'
      prefix: ""

Read More

Integration

We provide the option to install Gangplank alongside the Capsule Proxy. This allows users to authenticate with their OIDC provider and configure their kubectl configuration file with the OpenID Connect Tokens, which are valid for the Capsule Proxy Ingress. This way users can access the Kubernetes API through the Capsule Proxy without having to worry about the authentication and token management. To install gangplank, you must enable it:

gangplank:
  enabled: true

Gangplank won’t just work out of the box. You will need to provide some configuration values, which are required for gangplank to work properly. These values are:

• GANGPLANK_CONFIG_AUTHORIZE_URL: https://keycloak/realms/realm-name/protocol/openid-connect/auth
• GANGPLANK_CONFIG_TOKEN_URL: https://keycloak/realms/realm-name/protocol/openid-connect/token
• GANGPLANK_CONFIG_REDIRECT_URL: https://gangplank.example.com/callback
• GANGPLANK_CONFIG_CLIENT_ID: gangplank
• GANGPLANK_CONFIG_CLIENT_SECRET: <SECRET>
• GANGPLANK_CONFIG_USERNAME_CLAIM: The JWT claim to use as the username. (we use email in the authentication config above, so this should also be email)
• GANGPLANK_CONFIG_APISERVER_URL: The URL Capsule Proxy Ingress. Since the users probably want to access the Kubernetes API from outside the cluster, you should use the Capsule Proxy Ingress URL here.

When using the Helm chart, you can set these values in the values.yaml file:

gangplank:
  enabled: true
  config:
     clusterName: "tenant-cluster"
     apiServerURL: "https://capsule-proxy.company.com:443"
     scopes: ["openid", "profile", "email", "groups", "offline_access"]
     redirectURL: "https://gangplank.company.com/callback"
     usernameClaim: "email"
     clientID: "gangplank"
     authorizeURL: "https://keycloak/realms/realm-name/protocol/openid-connect/auth"
     tokenURL: "https://keycloak/realms/realm-name/protocol/openid-connect/token"

  # Mount The Client Secret as Environment Variables (GANGPLANK_CONFIG_CLIENT_SECRET)
  envFrom:
  - secretRef:
       name: gangplank-secrets

Now the only thing left to do is to change the CA certificate which is provided. By default the CA certificate is set to the Kubernetes API server CA certificate, which is not valid for the Capsule Proxy Ingress. For this we can simply override the CA certificate in the Helm chart. You can do this by creating a Kubernetes Secret with the CA certificate and mounting it as a volume in the Gangplank deployment.

gangplank:
  volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: token-ca
  volumes:
    - name: token-ca
      projected:
        sources:
        - serviceAccountToken:
            path: token
        - secret:
            name: proxy-ingress-tls
            items:
            - key: tls.crt
              path: ca.crt

Note: In this example we used the tls.crt key of the proxy-ingress-tls secret. This is a classic Cert-Manager TLS secret, which contains only the Certificate and Key for the Capsule Proxy Ingress. However the Certificate contains the CA certificate as well (Certificate Chain), so we can use it to verify the Capsule Proxy Ingress. If you use a different secret, make sure to adjust the key accordingly.

If that’s not possible you can also set the CA certificate as an environment variable:

gangplank:
  config:
    clusterCAPath: "/capsule-proxy/ca.crt"
  volumeMounts:
    - mountPath: /capsule-proxy/
      name: token-ca
  volumes:
    - name: token-ca
      projected:
        sources:
        - secret:
            name: proxy-ingress-tls
            items:
            - key: tls.crt
              path: ca.crt

4 - API Reference

Packages:

• capsule.clastix.io/v1beta1

capsule.clastix.io/v1beta1

Resource Types:

• GlobalProxySettings

• ProxySetting

GlobalProxySettings

GlobalProxySettings is the Schema for the globalproxysettings API.

GlobalProxySettings.spec

GlobalProxySettingsSpec defines the desired state of GlobalProxySettings.

GlobalProxySettings.spec.rules[index]

GlobalProxySettings.spec.rules[index].subjects[index]

GlobalProxySettings.spec.rules[index].clusterResources[index]

GlobalProxySettings.spec.rules[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

GlobalProxySettings.spec.rules[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting

ProxySetting is the Schema for the proxysettings API.

ProxySetting.spec

ProxySettingSpec defines the additional Capsule Proxy settings for additional users of the Tenant. Resource is Namespace-scoped and applies the settings to the belonged Tenant.

ProxySetting.spec.subjects[index]

ProxySetting.spec.subjects[index].clusterResources[index]

ProxySetting.spec.subjects[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

ProxySetting.spec.subjects[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting.spec.subjects[index].proxySettings[index]