This is the multi-page printable view of this section. Click here to print.
Setup
1 - Installation
Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.
The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.
We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed CustomResourceDefinitions. The following Artifacthub repository are official:
Perform the following steps to install the capsule Operator:
Add repository:
helm repo add projectcapsule https://projectcapsule.github.io/chartsInstall Capsule-Proxy:
helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system --create-namespaceor (OCI)
helm install capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy -n capsule-system --create-namespaceShow the status:
helm status capsule-proxy -n capsule-systemUpgrade the Chart
helm upgrade capsule-proxy projectcapsule/capsule-proxy -n capsule-systemor (OCI)
helm upgrade capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy --version 0.13.0Uninstall the Chart
helm uninstall capsule-proxy -n capsule-system
GitOps
There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.
ArgoCD
Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: capsule
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: system
source:
repoURL: ghcr.io/projectcapsule/charts
targetRevision: 0.12.4
chart: capsule
helm:
valuesObject:
...
proxy:
enabled: true
webhooks:
enabled: true
certManager:
generateCertificates: true
options:
generateCertificates: false
oidcUsernameClaim: "email"
extraArgs:
- "--feature-gates=ProxyClusterScoped=true"
serviceMonitor:
enabled: true
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
destination:
server: https://kubernetes.default.svc
namespace: capsule-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
name: capsule-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
url: ghcr.io/projectcapsule/charts
name: capsule
project: system
type: helm
enableOCI: "true"
FluxCD
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: capsule
namespace: flux-system
spec:
serviceAccountName: kustomize-controller
targetNamespace: "capsule-system"
interval: 10m
releaseName: "capsule"
chart:
spec:
chart: capsule
version: "0.12.4"
sourceRef:
kind: HelmRepository
name: capsule
interval: 24h
install:
createNamespace: true
upgrade:
remediation:
remediateLastFailure: true
driftDetection:
mode: enabled
values:
proxy:
enabled: true
webhooks:
enabled: true
certManager:
generateCertificates: true
options:
generateCertificates: false
oidcUsernameClaim: "email"
extraArgs:
- "--feature-gates=ProxyClusterScoped=true"
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: capsule
namespace: flux-system
spec:
type: "oci"
interval: 12h0m0s
url: oci://ghcr.io/projectcapsule/charts
Considerations
Consdierations when deploying capsule-proxy
Exposure
Depending on your environment, you can expose the capsule-proxy by:
IngressNodePort ServiceLoadBalance ServiceHostPortHostNetwork
Here how it looks like when exposed through an Ingress Controller:
+-----------+ +-----------+ +-----------+
kubectl ------>|:443 |--------->|:9001 |-------->|:6443 |
+-----------+ +-----------+ +-----------+
ingress-controller capsule-proxy kube-apiserver
User Authentication
The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server.
It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.
If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.
Certificate Management
By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule.This relates to certifiacates for the proxy and the admissions server. However, you can also use a job to generate self-signed certificates and store them in a Kubernetes Secret:
options:
generateCertificates: true
certManager:
generateCertificates: false
Distribute CA within the Cluster
The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.
Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:
kubectl get secret capsule-proxy -n capsule-system -o json \
| jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
| kubectl apply -n tekton-pipelines -f -
This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:
HTTP Support
NOTE: kubectl will not work against a http server.
Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.
After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.
For example:
TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces
Metrics
Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.
The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.
Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.
capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:
path: the HTTP path of every single request that Capsule Proxy passes to the upstream capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.
path: the HTTP path of every single request that Capsule Proxy passes to the upstream status: the HTTP status code of the request
2 - Controller Options
You can customize the Capsule Proxy with the following configurations.
Controller Options
You can provide additional options via the helm chart:
options:
extraArgs:
- --disable-caching=true
Options are also available as dedicated configuration values:
# Controller Options
options:
# -- Set the listening port of the capsule-proxy
listeningPort: 9001
# -- Set leader election to true if you are running n-replicas
leaderElection: false
# -- Set the log verbosity of the capsule-proxy with a value from 1 to 10
logLevel: 4
# -- Name of the CapsuleConfiguration custom resource used by Capsule, required to identify the user groups
capsuleConfigurationName: default
# -- Define which groups must be ignored while proxying requests
ignoredUserGroups: []
# -- Specify if capsule-proxy will use SSL
oidcUsernameClaim: preferred_username
# -- Specify if capsule-proxy will use SSL
enableSSL: true
# -- Set the directory, where SSL certificate and keyfile will be located
SSLDirectory: /opt/capsule-proxy
# -- Set the name of SSL certificate file
SSLCertFileName: tls.crt
# -- Set the name of SSL key file
SSLKeyFileName: tls.key
# -- Specify if capsule-proxy will generate self-signed SSL certificates
generateCertificates: false
# -- Specify additional subject alternative names for the self-signed SSL
additionalSANs: []
# -- Specify an override for the Secret containing the certificate for SSL. Default value is empty and referring to the generated certificate.
certificateVolumeName: ""
# -- Set the role bindings reflector resync period, a local cache to store mappings between users and their namespaces. [Use a lower value in case of flaky etcd server connections.](https://github.com/projectcapsule/capsule-proxy/issues/174)
rolebindingsResyncPeriod: 10h
# -- Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector.
disableCaching: false
# -- Enable the rolebinding reflector, which allows to list the namespaces, where a rolebinding mentions a user.
roleBindingReflector: false
# -- Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
authPreferredTypes: "BearerToken,TLSCertificate"
# -- QPS to use for interacting with Kubernetes API Server.
clientConnectionQPS: 20
# -- Burst to use for interacting with kubernetes API Server.
clientConnectionBurst: 30
# -- Enable Pprof for profiling
pprof: false
The following options are available for the Capsule Proxy Controller:
--auth-preferred-types string Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate]
First match is used and can be specified multiple times as comma separated values or by using the flag multiple times. (default "[TLSCertificate,BearerToken]")
--capsule-configuration-name string Name of the CapsuleConfiguration used to retrieve the Capsule user groups names (default "default")
--capsule-user-group strings Names of the groups for capsule users (deprecated: use capsule-configuration-name)
--client-connection-burst int32 Burst to use for interacting with kubernetes apiserver. (default 30)
--client-connection-qps float32 QPS to use for interacting with kubernetes apiserver. (default 20)
--disable-caching Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector (default: false)
--enable-leader-election Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
--enable-pprof Enables Pprof endpoint for profiling (not recommend in production)
--enable-reflector Enable rolebinding reflector. The reflector allows to list the namespaces, where a rolebinding mentions a user
--enable-ssl Enable the bind on HTTPS for secure communication (default: true) (default true)
--feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
ProxyAllNamespaced=true|false (ALPHA - default=false)
ProxyClusterScoped=true|false (ALPHA - default=false)
SkipImpersonationReview=true|false (ALPHA - default=false)
--ignored-impersonation-group strings Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
--ignored-user-group strings Names of the groups which requests must be ignored and proxy-passed to the upstream server
--impersonation-group-regexp string Regular expression to match the groups which are considered for impersonation
--listening-port uint HTTP port the proxy listens to (default: 9001) (default 9001)
--metrics-addr string The address the metric endpoint binds to. (default ":8080")
--oidc-username-claim string The OIDC field name used to identify the user (default: preferred_username) (default "preferred_username")
--rolebindings-resync-period duration Resync period for rolebindings reflector (default 10h0m0s)
--ssl-cert-path string Path to the TLS certificate (default: /opt/capsule-proxy/tls.crt)
--ssl-key-path string Path to the TLS certificate key (default: /opt/capsule-proxy/tls.key)
--webhook-port int The port the webhook server binds to. (default 9443)
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
Feature Gates
Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:
| Feature Gate | Default Value | Description |
|---|---|---|
SkipImpersonationReview | false | SkipImpersonationReview allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems. |
ProxyClusterScoped | false | ProxyClusterScoped allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings |