Dave
It is quite common to do interop testing *during* an OASIS TC's lifecycle rather than after. Interop testing is a great way of uncovering issues in the specification and unclarities. I remember an interesting case in WSRX where we had two ways of doing one thing (yes I know it seems a bad idea put that way!) and it was only during interop testing that a side effect of this became known.
So I think it would be good to do interop testing whenever we can. In addition, it takes some effort to produce an interop test suite and we should definitely get on with that.
In my experience there are two ways of doing that:
1) create a list of scenarios and then a matrix of implX<-->implY on scenario Z, so you end up with a three dimensional matrix where every cell is either a pass / fail / not tested. The scenarios need to be complete enough so that they cover all the conformance clauses.
2) create a independent test suite that tests all the conformance clauses of the spec.
They both have advantages and disadvantages. #2 is easier to test, more repeatable to test and more self contained. However it has two big disadvantages: firstly, it is a lot of upfront work independent of any implementation. Secondly it has to be very complete to really work, because its not actually testing any two real implementations together!
Paul
PS I think I probably should cross-post this to OASIS as well.
