Do you know where your time is spent on startup?
Is it bytecode scanning? (by servlet spec requirements in jetty? or by your 3rd party libs?)
Is it annotation scanning? (by servlet spec requirements in jetty? or by your 3rd party libs?)
I'm not familiar with "ContextLoaderListener", as that's not a term in the servlet spec.
If startup time is important, then start moving away from runtime assembly of the webapp to build-time, such as doing things with statically defined configurations (many build tools can do this in an automatic fashion for you).
For pure Servlet behaviors, Jetty even has a quickstart featureset that will perform the initialization phase of the servlet spec during build-time of your war, and will then produce a WEB-INF/jetty-quickstart.xml that short circuits all of the servlet spec bytecode scanning and annotation scanning, making startup of servlet features super quick.
Whatever decision you go with, be aware that changes to the configuration of the WebApp can only occur during the servlet initialization phase, once that's done you can no longer change the state of the configuration (such as removing/adding servlets and filters).
