Migration from PostgreSQL 9 to PostgreSQL 13

By the 11th of November, 2021, the PostgreSQL version 9.6 came out of support, and Che team recommends that all users undergo migrating to version 13.

Follow the procedure below to migrate to a newer version of PostgreSQL successfully without any data loss.

Prerequisites

The kubectl tool is available.
An instance of Che running in Kubernetes.

Procedure

Save and push changes back to the Git repositories for all running workspaces of the Che instance.
Stop all workspaces in the Che instance.

Scale down the Che and Keycloak deployments:

kubectl scale deployment che --replicas=0 -n eclipse-che
kubectl scale deployment keycloak --replicas=0 -n eclipse-che

Backup available databases:

POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
CHE_POSTGRES_DB=$(kubectl get checluster/eclipse-che -n eclipse-che -o json  | jq '.spec.database.chePostgresDb')
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump $CHE_POSTGRES_DB > /tmp/che.sql"
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump keycloak > /tmp/keycloak.sql"

Copy the obtained backups to a local file system:

kubectl cp eclipse-che/$POSTGRES_POD:/tmp/che.sql che.sql
kubectl cp eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql keycloak.sql

Scale down the PostgreSQL deployment:

kubectl scale deployment postgres --replicas=0 -n eclipse-che

Delete the corresponding PVC unit to clean up old data:
```
kubectl delete pvc postgres-data -n eclipse-che
```
After deleting the PVC from the step above, a new PVC will automatically appear in a few seconds.

Set the version of the new PostgreSQL database to 13.3:

kubectl patch checluster eclipse-che -n eclipse-che --type=json -p '[{"op": "replace", "path": "/spec/database/postgresVersion", "value": "13.3"}]'

Scale up the PostgreSQL deployments:

kubectl scale deployment postgres --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=postgres -n eclipse-che --timeout=120s

Provision a database:

POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
OPERATOR_POD=$(kubectl get pods -n eclipse-che | grep che-operator | awk '{print $1}')

IDENTITY_POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresSecret')
IDENTITY_POSTGRES_PASSWORD=$(if [ -z "$IDENTITY_POSTGRES_SECRET" ] || [ $IDENTITY_POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresPassword'; else kubectl get secret $IDENTITY_POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.password' | base64 -d; fi)

kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE USER keycloak WITH PASSWORD '$IDENTITY_POSTGRES_PASSWORD'\""
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE DATABASE keycloak\""
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak\""

POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.database.chePostgresSecret')
CHE_USER=$(if [ -z "$POSTGRES_SECRET" ] || [ $POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.database.chePostgresUser'; else kubectl get secret $POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.user' | base64 -d; fi)

kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"ALTER USER $CHE_USER WITH SUPERUSER\""

Copy the backups to the PostgreSQL Pod:

kubectl cp che.sql eclipse-che/$POSTGRES_POD:/tmp/che.sql
kubectl cp keycloak.sql eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql

Restore the database:

kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql keycloak < /tmp/keycloak.sql"
kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql $CHE_POSTGRES_DB < /tmp/che.sql"

Scale up the Keycloak and Che deployments:

kubectl scale deployment keycloak --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=keycloak -n eclipse-che --timeout=120s
kubectl scale deployment che --replicas=1 -n eclipse-che
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=che -n eclipse-che --timeout=120s