Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Jubula » The good, the bad and the Ugly - Summary after 1 month of jubula(Big list of bugs, feature requests etc.)
The good, the bad and the Ugly - Summary after 1 month of jubula [message #811436] Fri, 02 March 2012 05:44 Go to next message
Johann Vogel is currently offline Johann Vogel
Messages: 20
Registered: February 2012
Junior Member
I had to work for 1 month with jubula, here is my "short" summary. - I hope some things will be fixed, because i have to work with it in summer again.


The good:

* Concept seems well thought
* Usable for non-programmers
* No recursions - can't run endless
* It's free, customizeable etc.
* Search functions in various views speed up the working process
* Object maping works better than expected, I'd even say excellent
* Database works fine
* Stable


The bad:

* Extending it is hard if you have small knowledge about RCP plugins... at least it seems for me
* Screenshots should be in the results-dir of the server too
* Setting parameters with the autrun/testexec would be nice
* Test Jobs don't work at all for me. Starting them simply does nothing at all.
* Sometimes when started from Jubula in eclipse (never at cmd-line-start) - 10 Error-
* Messages with "An error occured" appear, no nearer specification
* Invalid regex / other commands mostly show up at runtime
* Multilang doesn't really work: Suggestion: Other languages link to the default-language-values and overwrite them if you select it - probably some dialogue like the externalize language dialogue from java-eclipse
* Object Mapping (missing/possible) only if the test case is in a test suite
* Database refresh takes quite long for my project (size 6mb @ export to xml)
* Why isn't username/password in the db-config and has to be reentered every time i connect (no "remember my password"?)
* Why doesn't jubula connect to the last used database on default? Usless extra step.
* Running test cases should always be enabled and if you aren't connected to aut agent you will automatically do this, and start the aut.
* No option to check workspace-properties of RCP applications

Missing functions: (See thread#283202 of this forum)
* Absolute/Relative mouse movement
* Loop/Iteration "handler" - for example to double-click all elements in a tree/list, with a running-variable that can be defined
* Some kind of "if" like IF(CONDITION (via java-evaluate, see lower), TEST CASE !, TEST CASE 2))
* Get number of elements for list/tree(nodes)
* Application - Window Funcions (Move/Min/Maximize/Resize)
* Application - Java-Evaluate - to make checks and calculations like <, >, +, -
* Check all cells of table

Test Handlers:
* In Test Case Browser displayed as "the first" - Should be marked with the icon from context menu and prefix: HANDLER <TRIGGERD WHEN> or at least indention in the tree
* A "Continue without error" would be nice, for example if you try to press 5 buttons, and it isn't a program-bug if one of them is disabled
* Empty test-handlers / Just test steps as handlers would be nice - sometimes, in case of connect-timeout a retry without handling is enough

Conceptual
Instead of having Test Cases and Test Suits (and nonworking jobs) you could make runable test cases (only if they have no parameters) and remove the test suite browser and replace it by a test case browser with an additional root node "Runable test cases" that can be run/set (un)run-able by context menu

The ugly:

.. user interface - had to explain Jubula the user-interface and every 3rd question was like "Why is it here (usually: context menu) and not there (usually: bound to double click)".

* Various "1-Option-Context-Menus" of "Do-Nothing-Buttons" - Connect to AUT agent/Select AUT etc. - If you haven't previously selected something, the buttons do simply nothing - and you wait forever
* The change between test specification/execution is quite senseless/annyoing
* Two Run Buttons - One in Running AUTs, one in Running AUT, one in Test Suit
* Browser - The possible runs are distributed - No context menu option of a test suite/job with "Run Suite"
* Data Sets/Component view should be outsourced to Properites - If you have 10 Data Sets defined, properties shows just the first (should be something like "multible values")
* Marking the writeable fields/cells in Properties would be nice - new users have to search the parameter-field
* Problems: An exact position where the problem appears - with opening on double click would be nice - no need to browse threw the test suites

TC (editor)-Critics:
* is built with a tree, but can't expand it's test cases - doesn't make much sense to me
* double click on empty background does exactly - nothing - suggestion: add new test step here
* double click on a test case would be expected to open it instead of adding a new
* adding a new should be done by double-clicking the root node only
* Copying actions doesn't work, just moving

TC Browser-Critics:
* Clicking on a step inside a test case opens read-only-properties - writeable would improve the workflow, because you don't have to open test cases for minor changes
* Doubleclicking a referenced function in a test case opens the "root case", the behaviour of "Context menu -> open menu" would be my expected action

[Updated on: Fri, 02 March 2012 06:15]

Report message to a moderator

Re: The good, the bad and the Ugly - Summary after 1 month of jubula [message #816747 is a reply to message #811436] Fri, 09 March 2012 03:04 Go to previous message
Alexandra Schladebeck is currently offline Alexandra Schladebeck
Messages: 1055
Registered: July 2009
Senior Member
Hi,

First of all, thanks for the comments and the feedback. For some of the points, I think it's definitely worth entering enhancements, some other points are already planned, and for some I'll try to explain the motivation or reasoning behind them.

Quote:
Extending it is hard if you have small knowledge about RCP plugins... at least it seems for me


This is one of the few areas in Jubula where experienced RCP developers are required. We do have an enhancement request to improve the tooling that you can follow.

Quote:
Screenshots should be in the results-dir of the server too


Can I ask what your use case is for this? Is there a problem (usability or otherwise) with viewing the results directly in the reporting perspective? If the reporting perspective is problematic, please enter tickets for that. If you have a specific use case that can't be achieved with the reporting perspective, then enter an enhancement.

Quote:
Setting parameters with the autrun/testexec would be nice


This is (for most parameters) possible for testexec via the configuration file. It's not currently possible for autrun though. Again, enter an enhancement.

Quote:
Test Jobs don't work at all for me. Starting them simply does nothing at all.


That's odd. You could try reading this blog about test jobs - if you find errors with the execution of them, then enter a ticket. I am unaware of any issues with test jobs in the current release (except for this which would explain why they don't execute anything if it doesn't get saved).

Quote:

Messages with "An error occured" appear, no nearer specification


Error handling is admittedly one area that is not so amazing in older features. We have much more of a focus on it for newer features. If you can name specific areas where bad error handling or descriptions are shown, then enter tickets for those areas. Unfortunately, a general "make error handling better" is quite hard to plan!

Quote:

Invalid regex / other commands mostly show up at runtime


We've played around with the idea of some kind of test suite validation where you can "run" the test suite but only in terms of its data. I don't think there's an enhancement, feel free to enter one.

Quote:
Multilang doesn't really work: Suggestion: Other languages link to the default-language-values and overwrite them if you select it - probably some dialogue like the externalize language dialogue from java-eclipse


You're right here with the multilang. We're aware of the issue, but it's never received that much attention from us because not many people seem to want to test multiple language versions of their AUT, and for those that do, they don't generally want to translate everything - just the use cases that are language specific (like dealing with currency, date formats etc), so the effort is kept to a minimum.

Quote:
Object Mapping (missing/possible) only if the test case is in a test suite


I'll deal with this point in the section on concepts of execution etc.

Quote:
Database refresh takes quite long for my project (size 6mb @ export to xml)


We're hoping to solve refresh problems once and for all with a change to our data model. It's not 100% definite that we'll be going ahead with the change, but it's looking promising.

Quote:
Why isn't username/password in the db-config and has to be reentered every time i connect (no "remember my password"?)


We have an enhancement with a patch for this. It won't be in Juno though.

Quote:
Why doesn't jubula connect to the last used database on default? Usless extra step.

You could put in an enhancement for this.

Quote:
Running test cases should always be enabled and if you aren't connected to aut agent you will automatically do this, and start the aut.


The problem we have here is deciding which Agent to connect to (there could be multiple defined / running / not running, on multiple ports, on various machines) and which AUT (there could be multiple defined) in which configuration (linux, windows, mac, with argument A or argument B...). If you can come up with meaningful defaults for this kind of a constellation, then enter an enhancement.

Missing functions

Most of the problems here seem to be related to knowing how many elements you have in a tree / list / table etc. You can usually find out how many elements you have via "check property" (there are properties like rowCount, itemCount etc that can help).

What it seems like you're missing is the ability to add that information as a loop i.e execute this test case as many times as $itemCount, once on each item. You could put an enhancement in, but I do know that one of the things we say about testing is that you should know your data.

You can do if-then using the retry event handler. If the checkbox is not active, then activate it. See more on retry event handlers here . If then else has been deliberately not implemented to preserve the deterministic-ness of the tests. Either something is correct or it is not correct, and I should be able to know when looking at a test result report that it always ran in the same way.

Quote:
Application - Window Funcions (Move/Min/Maximize/Resize)


This is something that we probably won't implement because use cases for it are not generally part of functional testing and because there are no platform / window-system independent ways of doing it.

Quote:
Application - Java-Evaluate - to make checks and calculations like <, >, +, -


There's an enhancement request for this.

Event Handlers:
"Continue without error" is only possible via a successful retry. If something fails, it's not ok unless I can do something (usually small) to change the situation.

Quote:
Empty test-handlers / Just test steps as handlers would be nice - sometimes, in case of connect-timeout a retry without handling is enough


You can use empty test cases as event handlers. They do nothing except react to *this error* at *this place*

Conceptual

The break between specification and execution comes right from the beginnings of the tool. The idea is that you can have Test Cases that are used in various places, for various AUTs (which is why validation does not happen in the Test Case Browser most especially not for Object Mapping).
You are right that the two browsers could indeed be combined as one tree. The thought has occurred to us a few times as well, but hasn't been a high enough priority compared to other requirements to actually be looked at in any more detail.

The ugly:

Quote:
Various "1-Option-Context-Menus" of "Do-Nothing-Buttons" - Connect to AUT agent/Select AUT etc. - If you haven't previously selected something, the buttons do simply nothing - and you wait forever

You could put a ticket in for this.
Quote:


Browser - The possible runs are distributed - No context menu option of a test suite/job with "Run Suite"


This is a good enhancement request.


Quote:
Marking the writeable fields/cells in Properties would be nice - new users have to search the parameter-field


We have an enhancement request with a patch to change the structure of the properties view to make it more readable and hopefully more user friendly.

Quote:

Problems: An exact position where the problem appears - with opening on double click would be nice - no need to browse threw the test suites


Good idea. Put in an enhancement.

TC (editor)-Critics:

Quote:
double click on a test case would be expected to open it instead of adding a new


At the moment, Shift F6 (just F3 in the upcoming version) opens a Test Case Specification and ENTER opens the Test Case Reference dialog. The workflow:
ENTER to open ref TC dialog - type filter string for Test Case I want - ENTER to select - ENTER to open ref TC dialog for the next test case is quite well established. You could enter an enhancement to make the behavior configurable though.

Quote:
Copying actions doesn't work, just moving


Have a look at this forum for the copy-paste debate. We've relented somewhat for Juno with this feature, by adding a "Save As new Test Case" option.

TC Browser-Critics:

Quote:
Clicking on a step inside a test case opens read-only-properties - writeable would improve the workflow, because you don't have to open test cases for minor changes


This is basically to do with multi-user support in the database so that conflicting changes can't happen.

(For some of the usability issues I've not replied directly. For many of them, the behavior one user would expect would be the opposite of what another would expect. That's not to say either user is necessarily wrong, but we don't change behaviors based on single comments we try to collect them over time and see how many people would be helped / hindered by the change or make it configurable.)

Hopefully the answers have helped to clear up some questions. Any enhancements / tickets / patches you want to enter are welcome. Thanks again for taking the time to provide detailed feedback.

Best regards,
Alex


Previous Topic:GEF Inspector Problem
Next Topic:Preparing config.ini
Goto Forum:
  


Current Time: Sun Apr 20 12:36:16 EDT 2014

Powered by FUDForum. Page generated in 0.01564 seconds