Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Upload of a big file (~5GB) via Multipart upload is getting stuck
Hi Debraj,

I think what Joakim told you is exactly what you need to do:  start by removing all dependencies to commons-fileupload and work your way from there.

If you don`t, you'll end up wasting a lot of time only to create patchwork.

Regards,

Nicolas Therrien

Senior Software Developer

https://www.motorolasolutions.com/content/dam/msi/images/logos/corporate/msiemailsignature.png

o: +1.819.931.2053  



On Sat, Oct 13, 2018 at 12:19 PM Debraj Manna <subharaj.manna@xxxxxxxxx> wrote:
Thanks Joakim for a detailed response. 

I don't think there is a problem in http-client as I can see that about 3.5-3.8 GB out 5 GB is written to disk and then it gets stuck. I will verify your other suggestions. But the problem is the issue does not come always. It is intermittent. 

On Sat, Oct 13, 2018 at 5:49 PM Joakim Erdfelt <joakim@xxxxxxxxxxx> wrote:
The relevant stacktrace ...

"dw-150 - POST /management/upgrade/uploadbundle" #150 prio=5 os_prio=0 tid=0x00007ffff0f7e800 nid=0x3a46 in Object.wait() [0x00007ffee0728000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.jetty.server.HttpInput.blockForContent(HttpInput.java:565)
at org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput.java:1084)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:306)
- locked <0x0000000677a5d220> (a java.util.ArrayDeque)
at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999)
at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1465)
at com.vnera.restapilayer.ManagementResource.flushFileToDisk(ManagementResource.java:2515)
at com.vnera.restapilayer.ManagementResource.storeFileToDisk(ManagementResource.java:2441)
at com.vnera.restapilayer.ManagementResource.uploadBundle(ManagementResource.java:2366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308)
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)

Looks like you are using dropwizard + jersey + com.vnera.restapilayer + commons-fileupload to handle your file upload.

Your setup isn't using Jetty for multipart/form-data upload in the way you think.

First, don't use commons-fileupload, that's for Servlet 2.4 (and older) environments.
Support for multipart/form-data parsing is built into the Servlet spec as HttpServletRequest.getParts() and getPart(name) since Servlet 3.0

If your code, or ANY of its Filters or 3rd party libraries uses any one of the HttpServletRequest.getParameter*() methods before your attempt at using commons-fileupload then your code will not work and fail.
Why?  That's because the Servlet spec says we have to read the Request URI query and Request form data (Input Stream) to generate the the parameter list and Parts list (if request is a multipart/form)

Right now, based on your stacktrace, Jetty is only involved in giving you a HttpServletRequest InputStream which is blocked waiting for more content from the client.
This could be a bug in commons-fileupload (a common scenario)
A bug in your use of the servlet spec (based on the number of layers you have going on, this is a high probability, try putting a breakpoint on Jetty's org.eclipse.jetty.server.Request.getParameters() method to see if the above example is going on)
Or a bug in your http client (a common scenario where the client says `Content-Length: <big-number>` but never sends the complete <big-number> of bytes, hence the blocking for content)

You can test Jetty's behavior in isolation easily enough.

$ mkdir jetty-downloads
$ cd jetty-downloads
$ tar -zxf tar -zxvf jetty-distribution-9.4.12.v20180830.tar.gz
$ cd jetty-distribution-9.4.12.v20180830/demo-base/
$ java -jar ../start.jar

now open a web browser to http://localhost:8080/test/dump/info
scroll down to "Form to generate UPLOAD content"
use "File 1" or "File 2" on that form to select your 5gb data file.
then "submit"

On chrome you'll see an "Uploading" percentage at the bottom of the window.

when it's done uploading, you'll get a "Dump Servlet" response.
scroll down to "Parts:" (about 1/3rd down the page)
you'll see the results of HttpSerletRequest.getParts() here.

example (for mine):
Parts:
Action: Part{n=Action,fn=null,ct=null,s=6,tmp=true,file=null}
file2: Part{n=file2,fn=,ct=application/octet-stream,s=0,tmp=true,file=null}
TextField: Part{n=TextField,fn=null,ct=null,s=7,tmp=true,file=null}
file1: Part{n=file1,fn=bigger.iso,ct=application/octet-stream,s=6806166040,tmp=true,file=/private/var/folders/gm/676mvm6j6131h37d0msqk1k40000gn/T/jetty-0.0.0.0-8080-test.war-_test-any-432009796042963862.dir/upload/MultiPart7642561650112679943}

That tells me that 6806166040 bytes were sent as part of "file1".

If I look at the original (pre-uploaded) file, I have ...

$ ls -la bigger.iso
-rw-r--r--  1 joakim  staff  6806166040 Oct 13 07:09 bigger.iso

I can see that the upload completed, the entire 6GB file was sent.

Joakim Erdfelt / joakim@xxxxxxxxxxx


On Sat, Oct 13, 2018 at 6:17 AM Debraj Manna <subharaj.manna@xxxxxxxxx> wrote:
POST /management/upgrade/uploadbundle is the api in context

On Sat, Oct 13, 2018 at 4:26 PM Debraj Manna <subharaj.manna@xxxxxxxxx> wrote:
I am using a jetty-server behind nginx reverse proxy. On trying to upload a big file about 5 GB via Multi-part I am seeing sometimes the upload is getting stuck. Please find attached the full thread dump when the jetty server was stuck.

I have also placed the thread dump in the below gist


Can someone let me know if this is some known issue in Jetty? 

Environment
Ubuntu 14.04
Java 8
Jetty Server - 9.4.10, 9.4.8.v20171121
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwICAg&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=

Back to the top