[jetty-dev] Concurrency issue causing EnvConfiguration.configure to thro

[Matthew Sheppard <msheppard@xxxxxxxxxxxxxx>]

Hi Folks,

I'm currently debugging an issue we've seen running our system, which contains two embedded jetty servers, where rarely (about one in 6000 restarts is the best estimate I have at the moment) we see one of our web apps fail to deploy with jetty throwing a NameNotFoundException with the message "remaining name 'env'" (Full stack trace included for reference below).

If I understand the design of the code correctly, that shouldn't be possible because the EnvConfiguration should be ensuring that the 'env' subcontext is always created in preConfigure before its configure starts trying to bind Env Entries. (See https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63)

I went looking through some of the relevant code and came to wondering whether https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249 was possibly the cause. I think in writing this I've convinced myself, however, that since we start each jetty server in a separate thread there should not be a problem with messing with the current thread's context class loader as I initially though, but maybe there's something else there I'm overlooking.

I'm working at the moment to try to reduce the problem down into a form I can demonstrate it outside our system, but figured I'd post this in the meanwhile in the hope someone might have bumped into this before or have some pointers about where I ought to be looking.

Many thanks,

Matt Sheppard

—-

The full stack trace we see in jetty's log is as follows. I believe it's using jetty version 9.4.6.v20170531. I'm working on setting up a version with the latest jetty to reproduce the problem there, but it'll take some time given the infrequency with which it occurs.

javax.naming.NameNotFoundException; remaining name 'env'
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
        at org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
        at javax.naming.InitialContext.lookup(InitialContext.java:417)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
        at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:452)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
        at org.eclipse.jetty.server.Server.doStart(Server.java:419)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at (our code which calls start)