Importing untrusted TLS certificates to Che

External communications between Che components are, by default, encrypted with TLS. Communications of Che components with external services such as proxies, source code repositories, identity providers may require using of TLS. Those communications require the use of TLS certificates signed by trusted Certificate Authorities.

When the certificates used by Che components or by an external service are signed by an untrusted CA it can be necessary to import the CA certificate in the Che installation, so that every Che component will consider them as signed by a trusted CA.

Typical cases that may require this addition are:

  • when the underlying Kubernetes cluster uses TLS certificates signed by a CA that is not trusted,

  • when Che server or workspace components connect to external services such as Keycloak or a Git server that use TLS certificates signed by an untrusted CA.

Che uses labeled ConfigMaps in namespace as sources for TLS certificates. The ConfigMaps can have arbitrary number of keys with arbitrary number of certificates each.

On OpenShift, when the cluster contains cluster-wide trusted CA certificates added through the cluster-wide-proxy configuration, Che Operator detects them and automatically injects them into this ConfigMap:

  • Che automatically labels the ConfigMap with the config.openshift.io/inject-trusted-cabundle="true" label.

  • Based on this annotation, OpenShift automatically injects the cluster-wide trusted CA certificates inside the ca-bundle.crt key of ConfigMap

Some Che components require to have full certificate chain to trust the endpoint. If the cluster is configured with an intermediate certificate, then the whole chain (including self-signed root) should be added to Che.

Adding new CA certificates into Che

This guide can be used before the installations of Che or when Che is already installed and running.

If you are using Che version lower than v7.23.0 see this guide on how to apply additional TLS certificates.
Prerequisites
  • The kubectl tool is available.

  • Namespace for Che exists.

Procedure
  1. Save the certificates you need to import, to a local file system.

    • Certificate files are typically stored as Base64 ASCII files, such as .pem, .crt, .ca-bundle. But, they can also be binary-encoded, for example, as .cer files. All Secrets that hold certificate files should use the Base64 ASCII certificate rather than the binary-encoded certificate.

    • Che already uses some reserved file names to automatically inject certificates into the ConfigMap, so you should avoid using the following reserved file names to save your certificates:

      • ca-bundle.crt

      • ca.crt

  2. Create a new ConfigMap with the required TLS certificates:

    $ kubectl create configmap custom-certs --from-file=<bundle-file-path> -n=eclipse-che

    To apply more than one bundle, add another --from-file=<bundle-file-path> flag to the above command. Or you may create another ConfigMap.

  3. Label created ConfigMaps with both app.kubernetes.io/part-of=che.eclipse.org and app.kubernetes.io/component=ca-bundle labels:

    $ kubectl label configmap custom-certs app.kubernetes.io/part-of=che.eclipse.org app.kubernetes.io/component=ca-bundle -n <che-namespace-name>
  4. Deploy Che if it has not been deployed before otherwise wait until rollout of Che components finishes. If there are running workspaces, they should be restarted in order the changes take effect.

Adding CA certificates in case of Helm installer

For a Che Helm Chart deployment only one ConfigMap supported. It is recommended to add the certificates at installation time.

At installation time

The ConfigMap should be pre-created in the Eclipse Che installation namespace. Ensure you override the global.tls.serverTrustStoreConfigMapName Helm Chart field with the name of the ConfigMap when installing the Che Helm Chart. For this you should add the following arguments to the Helm command line:

--set global.tls.serverTrustStoreConfigMapName=custom-certs

Or, in case of chectl usage, provide a Helm patch --helm-patch-yaml flag that points to an yaml file with:

global:
  tls:
    enabled: true
    serverTrustStoreConfigMapName: custom-certs

On already running installations

Upgrade Che Helm Chart:

  1. Clone the che project.

  2. Go to the deploy/kubernetes/helm/che directory.

  3. Update the name of the configMap Che will use, by editing the global.tls.serverTrustStoreConfigMapName Helm Chart property to match the created or updated ConfigMap:

    $ helm upgrade che -n eclipse-che --set global.tls.serverTrustStoreConfigMapName=<config-map-name> .

    When using Minikube to run Che, substitute <kubernetes-cluster-domain> with $(minikube ip).nip.io.

Verification at the Che installation level

If after adding the certificates something does not work as expected, here is a list of things to verify:

  • In case of a Che Operator deployment, namespace where CheCluster located contains labeled ConfigMaps with right content:

    $ kubectl get cm --selector=app.kubernetes.io/component=ca-bundle,app.kubernetes.io/part-of=che.eclipse.org -n eclipse-che

    And to check content of ConfigMap:

$ {orch-cli} get cm __<name>__ -n {prod-namespace} -o yaml
  • Che Pod Volumes list contains a volume that uses ca-certs-merged ConfigMap as data-source. To get the list of Volumes of the Che Pod:

    $ kubectl get pod -o json <che-pod-name> -n eclipse-che | jq .spec.volumes
  • Che mounts certificates in folder /public-certs/ of the Che server container. This command returns the list of files in that folder:

    $ kubectl exec -t <che-pod-name> -n eclipse-che -- ls /public-certs/
  • In the Che server logs there is a line for every certificate added to the Java truststore, including configured Che certificates.

    $ kubectl logs <che-pod-name> -n eclipse-che
  • Che server Java truststore contains the certificates. The certificates SHA1 fingerprints are among the list of the SHA1 of the certificates included in the truststore returned by the following command:

    $ kubectl exec -t <che-pod-name> -n eclipse-che -- keytool -list -keystore /home/user/cacerts
    Your keystore contains 141 entries
    
    (...)

    To get the SHA1 hash of a certificate on the local filesystem:

    $ openssl x509 -in <certificate-file-path> -fingerprint -noout
    SHA1 Fingerprint=3F:DA:BF:E7:A7:A7:90:62:CA:CF:C7:55:0E:1D:7D:05:16:7D:45:60

Verification at the workspace level

  • Start a workspace, get the namespace in which it has been created, and wait for it to be started

  • Get the name of the workspace Pod with the following command:

    $ kubectl get pods -o=jsonpath='{.items[0].metadata.name}' -n <workspace namespace> | grep '^workspace.*'
  • Get the name of the Theia IDE container in the workspace Pod with the following command:

    $ kubectl get -o json pod <workspace pod name>  -n <workspace namespace> | \
        jq -r '.spec.containers[] | select(.name | startswith("theia-ide")).name'
  • Look for a ca-certs ConfigMap that should have been created inside the workspace namespace:

    $ kubectl get cm ca-certs <workspace namespace>
  • Check that the entries in the ca-certs ConfigMap contain all the additional entries you added before. In addition it may contain ca-bundl.crt entry which is a reserved one:

    $ kubectl get cm ca-certs -n <workspace namespace> -o json | jq -r '.data | keys[]'
    ca-bundle.crt
    source-config-map-name.data-key.crt
  • Make sure that the ca-certs ConfigMap has been added as a volume in the workspace Pod:

    $ kubectl get -o json pod <workspace pod name> -n <workspace namespace> | \
        jq '.spec.volumes[] | select(.configMap.name == "ca-certs")'
    {
      "configMap": {
        "defaultMode": 420,
        "name": "ca-certs"
      },
      "name": "che-self-signed-certs"
    }
  • Confirm that the volume has been mounted into containers, especially in the Theia IDE container:

    $ kubectl get -o json pod <workspace pod name> -n <workspace namespace> | \
       jq '.spec.containers[] | select(.name == "<theia ide container name>").volumeMounts[] | select(.name == "che-self-signed-certs")'
    {
      "mountPath": "/public-certs",
      "name": "che-self-signed-certs",
      "readOnly": true
    }
  • Inspect /public-certs folder in the Theia IDE container and check that its contents match the list of entries in the ca-certs ConfigMap:

    $ kubectl exec <workspace pod name> -c <theia ide container name> -n <workspace namespace> -- ls /public-certs
    ca-bundle.crt
    source-config-map-name.data-key.crt