Importing untrusted TLS certificates to Che

This obsolete method is kept for backward compatibility . See Importing untrusted TLS certificates to Che

Internal communications between Che components are, by default, encrypted with TLS. Communications of Che components with external services such as proxies, source code repositories, identity providers may require TLS tool. Those communications require the use of TLS certificates signed by trusted Certificate Authorities.

When the certificates used by Che components or by an external service are signed by an untrusted CA it can be necessary to import the CA certificate in the Che installation, so that every Che component will consider them as signed by a trusted CA.

Typical cases that may require this addition are:

  • when the underlying Kubernetes cluster uses TLS certificates signed by a CA that is not trusted,

  • when Che server or workspace components connect to external services such as Keycloak or a Git server that use TLS certificates signed by an untrusted CA.

To store those certificates, Che uses a dedicated ConfigMap. Its default name is ca-certs but Che allows configuring its name.

On OpenShift, when the cluster contains cluster-wide trusted CA certificates added using the cluster-wide-proxy configuration, Che Operator detects them and automatically injects them into this ConfigMap:

  • Che automatically labels the ConfigMap with the config.openshift.io/inject-trusted-cabundle="true" label.

  • Based on this annotation, OpenShift automatically injects the cluster-wide trusted CA certificates inside the ca-bundle.crt key of ConfigMap

At Che installation time

Prerequisites
  • The kubectl tool is available.

  • You are ready to create CheCluster custom resource.

Procedure
  1. Save the certificates you need to import, to a local file system.

    • Certificate files are typically stored as Base64 ASCII files, such as .pem, .crt, .ca-bundle. But, they can also be binary-encoded, for example, as .cer files. All Secrets that hold certificate files should use the Base64 ASCII certificate rather than the binary-encoded certificate.

    • Che already uses some reserved file names to automatically inject certificates into the ConfigMap, so you should avoid using the following reserved file names to save your certificates:

      • ca-bundle.crt

      • ca.crt

  2. Create a new ConfigMap with the required TLS certificates:

    $ kubectl create configmap ca-certs --from-file=<certificate-file-path> -n=eclipse-che

    To apply more than one certificate, add another --from-file=<certificate-file-path> option to the above command.

  3. During the installation process, when creating the CheCluster custom resource, configure the right name for the created ConfigMap.

    For a Che Operator deployment, ensure you add the spec.server.ServerTrustStoreConfigMapName field with the name of the ConfigMap, to the CheCluster Custom Resource you will create during the installation:

    spec:
      server:
        ...
        spec.server.ServerTrustStoreConfigMapName: ca-certs

On already-running Che installations

Prerequisites
  • The kubectl tool is available.

  • You should first gather the name of the ConfigMap used to import certificates:

    On instances of Che deployed with the Che Operator, retrieve the name of the ConfigMap by reading the spec.server.ServerTrustStoreConfigMapName CheCluster Custom Resource property:

    $ kubectl get checluster eclipse-che -n eclipse-che -o jsonpath={.spec.server.serverTrustStoreConfigMapName}

    If the existing installation did not define any name for the ConfigMap, just use ca-certs.

Procedure
  1. Save the certificates you need to import, to a local file system.

    • Certificate files are typically stored as Base64 ASCII files, such as .pem, .crt, .ca-bundle. But, they can also be binary-encoded, for example, as .cer files. All Secrets that hold certificate files should use the Base64 ASCII certificate rather than the binary-encoded certificate.

    • Che already uses some reserved file names to automatically inject certificates into the ConfigMap, so you should avoid using the following reserved file names to save your certificates:

      • ca-bundle.crt

      • ca.crt

  2. Add the required TLS certificates in the ConfigMap:

    $ kubectl create configmap <config-map-name> --from-file=<certificate-file-path> -n=eclipse-che -o yaml --dry-run | kubectl apply -f -

    To apply more than one certificate, add another --from-file=<certificate-file-path> option to the above command.

  3. Configure the Che installation to use the ConfigMap:

    For a Che Operators deployment:

    1. Edit the spec.server.ServerTrustStoreConfigMapName CheCluster Custom Resource property to match the name of the ConfigMap:

      $ kubectl patch checluster/eclipse-che -n eclipse-che --type=json -p '[{"op": "replace", "path": "/spec/server/serverTrustStoreConfigMapName", "value": "<config-map-name>"}]'
  4. Restart the Che Operator, the Che server and Keycloak to load the new certificates:

    $ kubectl rollout restart -n eclipse-che deployment/che-operator
    $ kubectl rollout restart -n eclipse-che deployment/keycloak
    $ kubectl rollout restart -n eclipse-che deployment/che

    Restarting the Che components is not required anymore on Che v7.20.0 and higher.

Verification at the Che installation level

If you added the certificates without error, the Che server starts and obtains Keycloak configuration over https. Otherwise here is a list of things to verify:

  1. In case of a Che Operator deployment, the CheCluster attribute serverTrustStoreConfigMapName value matches the name of the ConfigMap. Get the value using the following command :

    $ kubectl get -o json checluster/eclipse-che -n eclipse-che | jq .spec.server.serverTrustStoreConfigMapName
  2. Che Pod Volumes list contains one Volume that uses the ConfigMap as data-source. To get the list of Volumes of the Che Pod:

    $ kubectl get pod -o json <che-pod-name> -n eclipse-che | jq .spec.volumes
  3. Che mounts certificates in folder /public-certs/ of the Che server container. This command returns the list of files in that folder:

    $ kubectl exec -t <che-pod-name> -n eclipse-che -- ls /public-certs/
  4. In the Che server logs there is a line for every certificate added to the Java truststore, including configured Che certificates.

    $ kubectl logs <che-pod-name> -n eclipse-che
    (...)
    Found a custom cert. Adding it to java trust store based on /usr/lib/jvm/java-1.8.0/jre/lib/security/cacerts
    (...)
  5. Che server Java truststore contains the certificates. The certificates SHA1 fingerprints are among the list of the SHA1 of the certificates included in the truststore returned by the following command:

    $ kubectl exec -t <che-pod-name> -n eclipse-che -- keytool -list -keystore /home/user/cacerts
    Your keystore contains 141 entries
    
    (...)

    To get the SHA1 hash of a certificate on the local filesystem:

    $ openssl x509 -in <certificate-file-path> -fingerprint -noout
    SHA1 Fingerprint=3F:DA:BF:E7:A7:A7:90:62:CA:CF:C7:55:0E:1D:7D:05:16:7D:45:60

Verification at the workspace level

  1. Start a workspace, get the namespace in which it has been created, and wait for it to be started.

  2. Get the name of the workspace Pod:

    $ kubectl get pods -o=jsonpath='{.items[0].metadata.name}' -n <workspace namespace> | grep '^workspace.*'
  3. Get the name of the Che-Theia IDE container in the workspace Pod:

    $ kubectl get -o json pod <workspace pod name>  -n <workspace namespace> | \
        jq -r '.spec.containers[] | select(.name | startswith("theia-ide")).name'
  4. Look for a ca-certs ConfigMap inside the workspace namespace:

    $ kubectl get cm ca-certs <workspace namespace>
  5. Check that the entries in the ca-certs ConfigMap contain all the additional entries you added in the certificate ConfigMap at the Che installation level, in addition to the reserved ca-bundle.crt entry:

    $ kubectl get cm ca-certs -n <workspace namespace> -o json | jq -r '.data | keys[]'
    ca-bundle.crt
    manually-added-certificate.crt
  6. Verify that the ca-certs ConfigMap has been added as a volume in the workspace Pod:

    $ kubectl get -o json pod <workspace pod name> -n eclipse-che | \
        jq '.spec.volumes[] | select(.configMap.name == "ca-certs")'
    {
      "configMap": {
        "defaultMode": 420,
        "name": "ca-certs"
      },
      "name": "che-self-signed-certs"
    }
  7. Confirm that the volume has been mounted into containers, especially in the Che-Theia IDE container:

    $ kubectl get -o json pod <workspace pod name> -n <workspace namespace> | \
       jq '.spec.containers[] | select(.name == "<theia ide container name>").volumeMounts[] | select(.name == "che-self-signed-certs")'
    {
      "mountPath": "/public-certs",
      "name": "che-self-signed-certs",
      "readOnly": true
    }
  8. Inspect /public-certs folder on the Che-Theia IDE container and check that its contents match the list of entries in the ca-certs ConfigMap:

    $ kubectl exec <workspace pod name> -c <theia ide container name> -n <workspace namespace> -- ls /public-certs
    ca-bundle.crt
    manually-added-certificate.crt