Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] HTTP/2 requirements

One additional question: do you think it is worthwhile to support HTTP/2 for newer browsers (which most of our users have) while at the same time trying to uphold support for IE9?



On 20-11-18 19:26, Joakim Erdfelt wrote:
Inline ...

On Tue, Nov 20, 2018 at 9:26 AM Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx> wrote:
Hello Joakim,

Firstly, this is the setup (Scala) code for the SslContextFactory etc.:

/***** start code *****/
val sslContextFactory = new SslContextFactory
        val ciphers = Array(
            "TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
            "TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 - IE7/8/9 - Android <= 4.3 */

Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)


Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.

Use defaults, none of those Cipher suites would be selected anyway due to default Cipher Suite exclusions in SslContextFactory AND the JVM disabled cipher suites.

    /** Default Excluded Cipher Suite List */
    private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
            // Exclude weak / insecure ciphers
            // Exclude ciphers that don't support forward secrecy
            // The following exclusions are present to cleanup known bad cipher
            // suites that may be accidentally included via include patterns.
            // The default enabled cipher list in Java will not include these
            // (but they are available in the supported list).



This is the in the default excluded protocols values, and also excluded/disabled at the JVM level, don't set it.

    /** Default Excluded Protocols List */
    private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL", "SSLv2", "SSLv2Hello", "SSLv3"};


This is only relevant for a client initiating a connection to a destination machine with TLS.

Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols are disabled in so many places (User-Agents, Browsers, OS level, Java VM level, Server level, various hardware intermediates) now its pointless to attempt to support them.   If this is still important to you, make a different connector for legacy behaviors.

        val httpsConfig = new HttpConfiguration
        httpsConfig.addCustomizer(new SecureRequestCustomizer)
        val http1 = new HttpConnectionFactory(httpsConfig)
        val connector =
            if (httpVersion != "http/2")
                val ssl = new SslConnectionFactory(sslContextFactory,"http/1.1")
                val http = new HttpConnectionFactory(httpsConfig)
                new ServerConnector(server,ssl,http)
                val http2 = new HTTP2ServerConnectionFactory(httpsConfig)
            val alpn = new ALPNServerConnectionFactory

Use alpn.setDefaultProtocol(http1.getDefaultProtocol())

                val ssl = new SslConnectionFactory(sslContextFactory,alpn.getProtocol)
                new ServerConnector(server,ssl,alpn,http2,http1)
        if (address != null) connector.setHost(address)
        connector.setIdleTimeout(1000L * idleTimeout)
/***** end code *****/

Then for the ThreadPool part:

/***** start code *****/
    val pool = new QueuedThreadPool(maxThreads,minThreads,idleTimeout,new ArrayBlockingQueue(queueSize))
    val server = new Server(pool)
/***** end code *****/

HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better then a raw Executor with regards to HTTP/2.

Threads are 10-100, idletimeouts I have tried are from 1000 to 20000, queueSize is now 500 but I have tried both much smaller and larger values.

I have no idea what values would be sensible. The number of concurrent users is relatively low: up to a couple of hundred, rarely more. Users can do requests that take longish to complete (sometimes up to 30 or even 60 seconds) but those are very rare. The majority is relatively quick (< 50ms). Current load on the problematic servers is very low (perhaps 10 users) and as said: low memory and CPU close to zero.
No problems with HTTP/1.1 with this code/settings, problematic with HTTP/2. Strangely enough some servers seem fine (or users just did not report any problems yet).

Start by using a default QueuedThreadPool, no configuration, no min thread, no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.

Any help would be welcome.

Kind regards,


Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2

When it comes to the ALPN suppport, you have to worry about what JVM runtime you are on first.

If you are using Java 8 then you have to align your alpn-boot version to the specific Java 8 JVM you are using.
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn work properly.

However, if you are using Java 11 then you can use the jetty-alpn-java-server artifact instead (no bootclasspath requirement) to get ALPN behaviors required for HTTP/2.

You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which will work the same regardless of Java 8 or Java 11.

When it comes to too restrictive of an SslContextFactory we have the following advice:

* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like 2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working, then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Understand the OpenJDK JVM Crypto Roadmap and its implications -
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know what the current state of your SslContextFactory is (including all selected cipher suites and protocols)
* Test your running server on to see where you are still having configuration issues with SslContextFactory

We have an example codebase that shows how to start an HTTP/2 server connector.

You can check the Jenkins project for their usage of HTTP/2 on Embedded Jetty as well. (their self executing WAR file does just this)

You could also run h2spec against your server and see if its behaving properly.

- Joakim


On 20-11-18 16:06, Joakim Erdfelt wrote:
How do you enable HTTP/2 in your embedded application?

Also, If you have configured your SslContextFactory, what does this configuration look like?

Lastly, what does your ThreadPool / Executor configuration look like?

Joakim Erdfelt / joakim@xxxxxxxxxxx

On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx> wrote:
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,


jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit

jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit

jetty-users mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit

Back to the top