Importing untrusted TLS certificates to Che

By default, external communications between Che components are encrypted with TLS. Communications of Che components with external services such as proxies, source code repositories, and identity provider might also require TLS. All communications encrypted with TLS require the use of TLS certificates signed by trusted Certificate Authorities (CA).

When the certificates used by Che components or by an external service are signed by an untrusted CA, you must import the CA certificate into the Che instance so that every Che component treats the certificates as signed by a trusted CA. You have to do this in the following cases:

The underlying Kubernetes cluster uses TLS certificates signed by an untrusted CA. Che server or workspace components connect to external OIDC providers or a Git server that use TLS certificates signed by an untrusted CA.

Che uses labeled ConfigMaps in namespace as sources for TLS certificates. The ConfigMaps can have an arbitrary number of keys with a random number of certificates each.

When an OpenShift cluster contains cluster-wide trusted CA certificates added through the cluster-wide-proxy configuration, Che Operator detects them and automatically injects them into a ConfigMap. Che automatically labels the ConfigMap with the config.openshift.io/inject-trusted-cabundle="true" label. Based on this annotation, OpenShift automatically injects the cluster-wide trusted CA certificates inside the ca-bundle.crt key of ConfigMap.

Some Che components require a full certificate chain to trust the endpoint. If the cluster is configured with an intermediate certificate, add the whole chain, including self-signed root, to Che.

Adding new CA certificates into Che

The following procedure is applicable for already installed and running instances and for instances that are to be installed.

Prerequisites

An active kubectl session with administrative permissions to the destination Kubernetes cluster. See Overview of kubectl.
Namespace for Che exists.

Procedure

Save the certificates you need to import to a local file system.

A certificate with the introductory phrase BEGIN TRUSTED CERTIFICATE is likely in the PEM TRUSTED CERTIFICATE format, which is not supported by Java. Convert it to the supported CERTIFICATE format with the following command:
- openssl x509 -in cert.pem -out cert.cer

Create a new ConfigMap with the required TLS certificates:
```
$ kubectl create configmap custom-certs --from-file=<bundle-file-path> -n=eclipse-che
```
To apply more than one bundle, add another -from-file=<bundle-file-path>. Alternatively, create another ConfigMap.

Label created ConfigMaps with the app.kubernetes.io/part-of=che.eclipse.org and app.kubernetes.io/component=ca-bundle labels:

$ kubectl label configmap custom-certs app.kubernetes.io/part-of=che.eclipse.org app.kubernetes.io/component=ca-bundle -n <che-namespace-name>

Deploy Che if it hasn’t been deployed before. Otherwise wait until the rollout of Che components finishes.
Restart running workspaces for the changes to take effect.

Troubleshooting imported certificate issues

If issues occur after adding the certificates, verify the specified values at the Che instance level and workspace level.

Verifying imported certificates at the Che instance level

In case of a Che Operator deployment, the namespace where the CheCluster is located contains labeled ConfigMaps with the correct content:
```
$ kubectl get cm --selector=app.kubernetes.io/component=ca-bundle,app.kubernetes.io/part-of=che.eclipse.org -n eclipse-che
```
Check the content of ConfigMap by entering:
```
$ kubectl get cm <name> -n eclipse-che -o yaml
```
Che Pod Volumes list contains a volume that uses ca-certs-merged ConfigMap as data-source. To get the list of Volumes of the Che Pod, run:
```
$ kubectl get pod -o json <che-pod-name> -n eclipse-che | jq .spec.volumes
```
Che mounts certificates in the /public-certs/ folder of the Che server container. To view the list of files in this folder, enter:
```
$ kubectl exec -t <che-pod-name> -n eclipse-che -- ls /public-certs/
```
In the Che server logs, there is a line for every certificate added to the Java truststore, including configured Che certificates. View them:
```
$ kubectl logs <che-pod-name> -n eclipse-che
```

Che server Java truststore contains the certificates. The certificates SHA1 fingerprints are among the list of the SHA1 of the certificates included in the truststore. View the list:

$ kubectl exec -t <che-pod-name> -n eclipse-che -- keytool -list -keystore /home/user/cacerts
Your keystore contains 141 entries:
+
(...)

To get the SHA1 hash of a certificate on the local filesystem, run:

$ openssl x509 -in <certificate-file-path> -fingerprint -noout
SHA1 Fingerprint=3F:DA:BF:E7:A7:A7:90:62:CA:CF:C7:55:0E:1D:7D:05:16:7D:45:60

Verifying imported certificates at the workspace level

Start a workspace, obtain the namespace name in which it has been created and wait for the workspace to be started.

Get the name of the workspace Pod:

$ kubectl get pods -o=jsonpath='{.items[0].metadata.name}' -n <workspace namespace> | grep '^workspace.*'

Get the name of the Che-Theia IDE container in the workspace Pod:

$ kubectl get -o json pod <workspace pod name>  -n <workspace namespace> | \
    jq -r '.spec.containers[] | select(.name | startswith("theia-ide")).name'

Look for a ca-certs ConfigMap inside the workspace namespace:
```
$ kubectl get cm ca-certs <workspace namespace>
```
Check that the entries in the ca-certs ConfigMap contain all the additional entries you added before. In addition, it can contain ca-bundle.crt reserved entry. View the entries:
```
$ kubectl get cm ca-certs -n <workspace namespace> -o json | jq -r '.data | keys[]'
ca-bundle.crt
source-config-map-name.data-key.crt
```

Confirm that the ca-certs ConfigMap is added as a volume in the workspace Pod:

$ kubectl get -o json pod <workspace pod name> -n <workspace namespace> | \
    jq '.spec.volumes[] | select(.configMap.name == "ca-certs")'
{
  "configMap": {
    "defaultMode": 420,
    "name": "ca-certs"
  },
  "name": "che-self-signed-certs"
}

Confirm that the volume is mounted into containers, especially in the Che-Theia IDE container:

$ kubectl get -o json pod <workspace pod name> -n <workspace namespace> | \
   jq '.spec.containers[] | select(.name == "<theia ide container name>").volumeMounts[] | select(.name == "che-self-signed-certs")'
{
  "mountPath": "/public-certs",
  "name": "che-self-signed-certs",
  "readOnly": true
}

Inspect the /public-certs folder in the Che-Theia IDE container and check if its contents match the list of entries in the ca-certs ConfigMap:

$ kubectl exec <workspace pod name> -c <theia ide container name> -n <workspace namespace> -- ls /public-certs
ca-bundle.crt
source-config-map-name.data-key.crt