Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] WAR reloading leaves stale file descriptors (file system quota not working as expected)

I wonder if this is the same issue that was reported a few years ago, that we could never get to the bottom of:  https://github.com/eclipse/jetty.project/issues/1425

Jan

On Fri, 26 Jun 2020 at 15:06, Joakim Erdfelt <joakim@xxxxxxxxxxx> wrote:
Hot reload on Java has many pitfalls.

Most of them come from WebApp ClassLoader Pinning.
Where a class being loaded from the WebApp is keeping a reference to the WebApp ClassLoader, preventing it from being recycled.

We have many LeakPreventers implemented that prevent the WebApp ClassLoader Pinning, for the common JVM and Servlet/Java EE classes, but there are still more in various other 3rd party libraries.


Give them a try, if it works out for you GREAT.
But if you still have this issue, then you'll have to figure out which class is holding the reference to the WebAppClassLoader and perhaps implement a LeakPreventere for it to let you have hot reload success.

Joakim Erdfelt / joakim@xxxxxxxxxxx


On Fri, Jun 26, 2020 at 7:54 AM Kiao Desouzza <kiao.desouzza@xxxxxxxxx> wrote:
Hi,

I am having an issue with file system quota tools not reporting correctly on the directory where jetty is deployed. After some investigation I think it's down to left over file descriptors like these (output from lsof):

java      39467                 sclient    5r      REG                8,1    134017  101365281 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar (deleted)
java      39467                 sclient    5r      REG                8,1    134017  101365281 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar (deleted)
java      39467                 sclient   45r      REG                8,1     69409  101365257 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/activation-1.1.1.jar (deleted)
java      39467                 sclient   46r      REG                8,1    267827  101365259 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/admin_2.13.jar (deleted)
java      39467                 sclient   47r      REG                8,1    499844  101365280 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar (deleted)
java      39467                 sclient   48r      REG                8,1    246918  101365265 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar (deleted)
java      39467                 sclient   49r      REG                8,1    347669  101365260 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar (deleted)
java      39467                 sclient   50r      REG                8,1    588337  101365276 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar (deleted)
java      39467                 sclient   51r      REG                8,1    208475  101365250 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar (deleted)
java      39467                 sclient   52r      REG                8,1    146108  101365282 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar (deleted)
java      39467                 sclient   53r      REG                8,1    214788  101365256 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-io-2.6.jar (deleted)
java      39467                 sclient   54r      REG                8,1     61829  101365278 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-logging-1.2.jar (deleted)
java      39467                 sclient   55r      REG                8,1    129592  101365268 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-pool2-2.7.0.jar (deleted)
java      39467                 sclient   56r      REG                8,1    186077  102297951 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-validator-1.6.jar (deleted)
java      39467                 sclient   57r      REG                8,1    580870  101365271 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/flyway-core-6.3.3.jar (deleted)
java      39467                 sclient   58r      REG                8,1   1586948  101365273 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/freemarker-2.3.29.jar (deleted)
java      39467                 sclient   59r      REG                8,1     80511  101365272 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/geny_2.13-0.6.0.jar (deleted)
java      39467                 sclient   60r      REG                8,1    219146  101365253 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/javax.mail-api-1.6.2.jar (deleted)
java      39467                 sclient   61r      REG                8,1     25464  101365283 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/jetty-continuation-9.4.28.v20200408.jar (deleted)
java      39467                 sclient   62r      REG                8,1    624036  101365275 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/mchange-commons-java-0.2.15.jar (deleted)
java      39467                 sclient   63r      REG                8,1    927447  101365252 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/postgresql-42.2.10.jar (deleted)
java      39467                 sclient   64r      REG                8,1    134017    1352185 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar (deleted)
java      39467                 sclient   65r      REG                8,1      4250  101365255 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-collection-compat_2.13-2.1.6.jar (deleted)
java      39467                 sclient   66r      REG                8,1   5865349  101365264 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-library-2.13.2.jar (deleted)
java      39467                 sclient   67r      REG                8,1    225338  101365277 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-parser-combinators_2.13-1.1.2.jar (deleted)
java      39467                 sclient   68r      REG                8,1    129113  101365279 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalaj-http_2.13-2.4.2.jar (deleted)
java      39467                 sclient   69r      REG                8,1   3625052  101365254 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-reflect-2.13.2.jar (deleted)
java      39467                 sclient   70r      REG                8,1   1801315  101365266 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-core_2.13-3.4.2.jar (deleted)
java      39467                 sclient   71r      REG                8,1      6504  102297938 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation-macro_2.13-3.4.2.jar (deleted)
java      39467                 sclient   72r      REG                8,1    201322  101365263 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation_2.13-3.4.2.jar (deleted)
java      39467                 sclient   73r      REG                8,1     19036  101365261 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc_2.13-3.4.2.jar (deleted)
java      39467                 sclient   74r      REG                8,1     41472  101365274 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/slf4j-api-1.7.30.jar (deleted)
java      39467                 sclient   75r      REG                8,1     35063  101365284 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/stats_collector_2.13-0.1.0-SNAPSHOT.jar (deleted)
java      39467                 sclient   76r      REG                8,1    187761  101365270 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/ujson_2.13-1.1.0.jar (deleted)
java      39467                 sclient   77r      REG                8,1    114284  101365269 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/upickle-core_2.13-1.1.0.jar (deleted)
java      39467                 sclient   94r      REG                8,1     69409    1220495 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/activation-1.1.1.jar (deleted)
java      39467                 sclient   95r      REG                8,1    267827    1352165 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/admin_2.13.jar (deleted)
java      39467                 sclient   96r      REG                8,1    499844    1352184 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar (deleted)
java      39467                 sclient   97r      REG                8,1    246918    1352170 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar (deleted)
java      39467                 sclient   98r      REG                8,1    347669    1352166 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar (deleted)
java      39467                 sclient   99r      REG                8,1    588337    1352180 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar (deleted)
java      39467                 sclient  100r      REG                8,1    208475     957623 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar (deleted)
java      39467                 sclient  101r      REG                8,1    146108    1352186 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar (deleted)
java      39467                 sclient  102r      REG                8,1    214788    1212954 /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-io-2.6.jar (deleted)
...

The file system tools(du) correctly report used disk space but file system quota is not seeing space being released after each update of deployed WAR file (used space grows). After I restart jetty file system quota is updated correctly. It's only JAR files from what I can tell so could it be related to the class loader in some way?

FS is xfs on centos8 with pquota (project quota)

I am using latest release 9.4.30.v20200611

Regards
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users


--
Jan Bartel <janb@xxxxxxxxxxx>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


Back to the top