Hi,
In short - It's not looking very good... but let's start with something positive - We've got steps 1-3 automated and PR for step 4 is on the way (should land in repo this week IMO). Everything is running fine locally...
But now for the not-so-good news.
For running this in CI, we are using standard infrastructure (standardized through whole QE org in Red Hat), called Central CI. This is basically just jenkins master and Openstack cluster, where slaves are provisioned automatically and per demand (when no jobs are running, no slaves are running ;-)).
We've spent last few days/weeks making the tests run on this infra. We've caught some problems and fixed them, but now we've come to the point, where we don't know how to proceed further and we think we will not be able to run our testsuite on this infra (ever?).
In the beginning we had lots of troubles when running minikube, starting che server (using chectl)... we were blaming the slow infra and we've increased some timeouts to the point it seemed everything should "just work ®", but we've started seeing weird failures in test. I'll try to describe one of them here, so you have better understanding of what we are dealing with:
In our testcase, we have springboot project in the workspace, we run mvn build on that project and check output of this build and waiting for successful build message there. But often, in this process, the tests fail.
Here is a screenshot of one of such failures. You can see in the bottom left, that the workspace is in "offline" mode - indicating something went really wrong.
When tests fail, we run few `kubectl` commands to gather as many logs as possible, but these commands often fail with different messages (that's really weird!) such as:
kubectl logs workspacehmlslj468jnbik67.workspace-584f547d7f-kpjb9 theia-ide73s -n che
10:58:11 Unable to connect to the server: http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug=""
or
kubectl --namespace=che get events
12:51:37 Unable to connect to the server: dial tcp 192.168.39.178:8443: i/o timeout or
kubectl --namespace=che get events
10:19:46 error: the server doesn't have a resource type "events"
This leads me to believing minikube is REALLY misbehaving. I started looking for possible causes, but I have no idea what could be wrong. The machines we are running tests on are virtual machines on Openstack with 16 VCPUs and 32G of RAM. Minikube there is started with 12G of RAM and 4CPUs.
I tried to sum up everything important, but I can provide more info if needed.
Right now, we are in a state, where any help is appreciated...
Radim
