Following the tips described on this page: Making good Eclipse RCP apps for Mac, I was able to add two modifications into the SWT rendering engine of scout.
Sheet style for message box
Notification of unsaved changes
Reacting on IForm.PROP_SAVE_NEEDED in handleScoutPropertyChange(..)
For now this changes are implemented in my application, by overriding some methods of the AbstractSwtEnvironment in my SwtEnvironment.
Any interest in a patch for the common SWT UI plug-in ?