Migration from PostgreSQL 9 to PostgreSQL 13

By the 11th of November, 2021, the PostgreSQL version 9.6 came out of support, and Che team recommends that all users undergo migrating to version 13.

Follow the procedure below to migrate to a newer version of PostgreSQL successfully without any data loss.

Prerequisites
  • The kubectl tool is available.

  • An instance of Che running in Kubernetes.

Procedure
  1. Save and push changes back to the Git repositories for all running workspaces of the Che instance.

  2. Stop all workspaces in the Che instance.

  3. Scale down the Che and Keycloak deployments:

    kubectl scale deployment che --replicas=0 -n eclipse-che
    kubectl scale deployment keycloak --replicas=0 -n eclipse-che
  4. Backup available databases:

    POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
    CHE_POSTGRES_DB=$(kubectl get checluster/eclipse-che -n eclipse-che -o json  | jq '.spec.database.chePostgresDb')
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump $CHE_POSTGRES_DB > /tmp/che.sql"
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "pg_dump keycloak > /tmp/keycloak.sql"
  5. Copy the obtained backups to a local file system:

    kubectl cp eclipse-che/$POSTGRES_POD:/tmp/che.sql che.sql
    kubectl cp eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql keycloak.sql
  6. Scale down the PostgreSQL deployment:

    kubectl scale deployment postgres --replicas=0 -n eclipse-che
  7. Delete the corresponding PVC unit to clean up old data:

    kubectl delete pvc postgres-data -n eclipse-che

    After deleting the PVC from the step above, a new PVC will automatically appear in a few seconds.

  8. Set the version of the new PostgreSQL database to 13.3:

    kubectl patch checluster eclipse-che -n eclipse-che --type=json -p '[{"op": "replace", "path": "/spec/database/postgresVersion", "value": "13.3"}]'
  9. Scale up the PostgreSQL deployments:

    kubectl scale deployment postgres --replicas=1 -n eclipse-che
    kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=postgres -n eclipse-che --timeout=120s
  10. Provision a database:

    POSTGRES_POD=$(kubectl get pods -n eclipse-che | grep postgres | awk '{print $1}')
    OPERATOR_POD=$(kubectl get pods -n eclipse-che | grep che-operator | awk '{print $1}')
    
    IDENTITY_POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresSecret')
    IDENTITY_POSTGRES_PASSWORD=$(if [ -z "$IDENTITY_POSTGRES_SECRET" ] || [ $IDENTITY_POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.auth.identityProviderPostgresPassword'; else kubectl get secret $IDENTITY_POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.password' | base64 -d; fi)
    
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE USER keycloak WITH PASSWORD '$IDENTITY_POSTGRES_PASSWORD'\""
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"CREATE DATABASE keycloak\""
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak\""
    
    POSTGRES_SECRET=$(kubectl get checluster/eclipse-che -n eclipse-che -o json | jq -r '.spec.database.chePostgresSecret')
    CHE_USER=$(if [ -z "$POSTGRES_SECRET" ] || [ $POSTGRES_SECRET = "null" ]; then kubectl get checluster/eclipse-che  -n eclipse-che -o json | jq -r '.spec.database.chePostgresUser'; else kubectl get secret $POSTGRES_SECRET -n eclipse-che -o json | jq -r '.data.user' | base64 -d; fi)
    
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql postgres -tAc \"ALTER USER $CHE_USER WITH SUPERUSER\""
  11. Copy the backups to the PostgreSQL Pod:

    kubectl cp che.sql eclipse-che/$POSTGRES_POD:/tmp/che.sql
    kubectl cp keycloak.sql eclipse-che/$POSTGRES_POD:/tmp/keycloak.sql
  12. Restore the database:

    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql keycloak < /tmp/keycloak.sql"
    kubectl exec -it $POSTGRES_POD -n eclipse-che  -- bash  -c "psql $CHE_POSTGRES_DB < /tmp/che.sql"
  13. Scale up the Keycloak and Che deployments:

    kubectl scale deployment keycloak --replicas=1 -n eclipse-che
    kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=keycloak -n eclipse-che --timeout=120s
    kubectl scale deployment che --replicas=1 -n eclipse-che
    kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=che -n eclipse-che --timeout=120s