Re: [jetty-users] Jetty Cannot Find File In Web Application (Duo Outage)

[Joakim Erdfelt <joakim@xxxxxxxxxxx>]

Sounds like you have deployed your webapps in a way that the work directory is in System temp (aka "/tmp" on redhat).

But now you have something that is cleaning out system temp periodically. (which your system didn't do before)

If you are using a ${jetty.base} directory, add a directory under it (not your ${jetty.home}, jetty-home, or jetty-distribution directory) called "work" and restart.

Or ... you can configure a different location for java's java.io.tmpdir.

If you want to know more, look at how webapp temp directories function -> https://stackoverflow.com/questions/19232182/jetty-starts-in-c-temp/19232771#19232771

Joakim Erdfelt / joakim@xxxxxxxxxxx

On Thu, Nov 1, 2018 at 12:20 PM David C Fuhs <dfuhs@xxxxxxxxxxxx> wrote:

Good morning all:

 

We are running Shibboleth Identity Provider (IdP) V3.3.3 on Jetty 9.3.9 on a Red Hat Linux platform.  A multifactor authentication flow added to the IdP redirects any user found in the MFA group in LDAP to duo.vm.  This has worked without error from when we installed it until a failure last week.

 

The nature of the failure:  A user in the MFA group clicked to login to a federated application.  The IdP displayed its login page as expected.  The user entered his uid + password and was successfully authenticated.  The search of the MFA group succeeded, but only a partial copy of duo.vm was displayed: the external part of the page was displayed but the inner section with the green buttons (e.g., “send me a push”) never loaded.

 

A _javascript_ file that is part of the IdP web application calls to Duo to retrieve the content for the inner section of the page.  That call never happened.

 

The IdP warn and process logs did not show anything interesting, but the log levels are set pretty low.  The Jetty access log did show that on the server experiencing the outage, the IdP got an HTTP 404 not found error on this call:

 

GET /idp/js/Duo-Web-v2.min.js

 

Access logs on the other IdP servers at the same time all showed HTTP 200 codes on the same GET.

 

That file exists and is readable.  Nothing in the Jetty or Duo configurations has been touched in a long time.

 

Restarting Jetty on the problematic IdP resolved the problem, but we don’t know why the problem occurred in the first place, or how to prevent it in the future.  The problematic IdP had been successfully authenticating MFA users for months with no reported problems.

 

We tested right after the failure against our development IdP instances and a hot spare production instance, but all of those systems behaved as expected.

 

We have made no Jetty configuration changes since we installed version 9.3.9.

 

The only recent changes to the production IdP server where the problem occurred was routine RHEL OS patching a week earlier (with no reported problems in the intervening week) plus ongoing additions to the CAS registry (conf/cas-protocol.xml), also with no reported side-effects.  We have not added configurations for any new service providers recently, and have made no changes to any Duo pages or configuration items in months.

 

What would cause Jetty to not find that _javascript_ file?

 

How can we prevent a recurrence?

 

Any insights greatly appreciated.

 

David Fuhs

Enterprise Applications

Information Security

California State University, Chico

dfuhs@xxxxxxxxxxxx

530-898-4852

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users