Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Request parsing into buffer is taking time leading to p99 latency issues

Hi Joakim,

Here are the logs of a request which took 5.4 seconds after upgrading to jetty version - 9.4.46.v20220331.
Is there a way to solve this slow clients issue from server side ?



DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.server.HttpChannel: new HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0} -> SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=0}-><null>,null,HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0}
DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.io.AbstractEndPoint: onOpen SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.io.AbstractConnection: onOpen HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.io.AbstractConnection: fillInterested HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.io.FillInterest: interested FillInterest@4089e08a{AC.ReadCB@4048662e{HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}}}
DEBUG [2022-05-11 11:50:56,560] org.eclipse.jetty.io.ChannelEndPoint: changeInterests p=false 0->1 for SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ManagedSelector: Created SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ChannelEndPoint: Key interests updated 0 -> 1 on SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/1,kio=1,kro=0}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ManagedSelector: selected 1 sun.nio.ch.SelectionKeyImpl@25cf1073 SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/1,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ChannelEndPoint: onSelected 1->0 r=true w=false for SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ChannelEndPoint: task CEP:SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill: EatWhatYouKill@921dbcf/SelectorProducer@6529b078/IDLE/p=true/InstrumentedQueuedThreadPool[dw]@2f60cbf2{STARTED,8<=10<=100000,i=1,r=-1,q=0}[ReservedThreadExecutor@e97f115{reserved=3/8,pending=0}][pc=698,pic=0,pec=1,epc=3276]@2022-05-11T11:50:56.561Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.FillInterest: fillable FillInterest@4089e08a{AC.ReadCB@4048662e{HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}}}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.io.ChannelEndPoint: Key interests updated 1 -> 0 on SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0} onFillable enter HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0} filled 8192 HeapByteBuffer@1e55b489[p=0,l=8192,c=8192,r=8192]={<<<POST /gameplay/103618036/...Busts":0},{"cardShown":>>>}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=START,0 of -1},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@1e55b489[p=0,l=8192,c=8192,r=8192]={<<<POST /gameplay/103618036/...Busts":0},{"cardShown":>>>} {}
DEBUG [2022-05-11 11:50:56,561] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,0 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=//global-api.local.live/gameplay/103618036/submitscore,age=0} parsed true HttpParser{s=CONTENT,0 of 14196}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,0 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parse HeapByteBuffer@1e55b489[p=1124,l=8192,c=8192,r=7068]={POST /gam...zip\r\n\r\n<<<{"gameSession":{"gameSess...Busts":0},{"cardShown":>>>} {}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,7068 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parsed false HttpParser{s=CONTENT,7068 of 14196}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,7068 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,7068 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} filled 1124 HeapByteBuffer@36f622a1[p=0,l=1124,c=8192,r=1124]={<<<9,"cardPlacement":4,"card...t":1,"cardsAlreadyprese>>>20,10,20,...:2,"car}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,7068 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parse HeapByteBuffer@36f622a1[p=0,l=1124,c=8192,r=1124]={<<<9,"cardPlacement":4,"card...t":1,"cardsAlreadyprese>>>20,10,20,...:2,"car} {}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parsed false HttpParser{s=CONTENT,8192 of 14196}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} filled 0 HeapByteBuffer@253ea3ff[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...{},\\"Po}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parse HeapByteBuffer@253ea3ff[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...{},\\"Po} {}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2} parsed false HttpParser{s=CONTENT,8192 of 14196}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.io.ChannelEndPoint: changeInterests p=false 0->1 for SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2}
DEBUG [2022-05-11 11:50:56,563] org.eclipse.jetty.io.ChannelEndPoint: Key interests updated 0 -> 1 on SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=0/30000}{io=1/1,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=2}
DEBUG [2022-05-11 11:51:02,014] org.eclipse.jetty.io.ManagedSelector: selected 1 sun.nio.ch.SelectionKeyImpl@25cf1073 SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=5450/30000}{io=1/1,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5453}
DEBUG [2022-05-11 11:51:02,014] org.eclipse.jetty.io.ChannelEndPoint: onSelected 1->0 r=true w=false for SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=5451/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5453}
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.io.ChannelEndPoint: task CEP:SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=5451/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454}:runFillable:NON_BLOCKING
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill: EatWhatYouKill@921dbcf/SelectorProducer@6529b078/PRODUCING/p=false/InstrumentedQueuedThreadPool[dw]@2f60cbf2{STARTED,8<=10<=100000,i=1,r=-1,q=0}[ReservedThreadExecutor@e97f115{reserved=3/8,pending=0}][pc=698,pic=0,pec=1,epc=3278]@2022-05-11T11:51:02.015Z m=PRODUCE_CONSUME t=CEP:SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=FI,flush=-,to=5451/30000}{io=1/0,kio=1,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454}:runFillable:NON_BLOCKING/NON_BLOCKING
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.io.ChannelEndPoint: Key interests updated 1 -> 0 on SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=5451/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454}
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454} filled 6004 HeapByteBuffer@253ea3ff[p=0,l=6004,c=8192,r=6004]={<<<nt":"12","laneWiseTotal":...jQ3ZGUzN2E3ZWU5NDI3"}}}>>>Count":39...13X":0,}
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=CONTENT,8192 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454} parse HeapByteBuffer@253ea3ff[p=0,l=6004,c=8192,r=6004]={<<<nt":"12","laneWiseTotal":...jQ3ZGUzN2E3ZWU5NDI3"}}}>>>Count":39...13X":0,} {}
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.server.HttpConnection: HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454} parsed false HttpParser{s=END,14196 of 14196}
DEBUG [2022-05-11 11:51:02,015] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=START}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5454}
DEBUG [2022-05-11 11:51:02,054] org.eclipse.jetty.io.ChannelEndPoint: flushed 32890 SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=W,to=38/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=COMMITTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5493}
DEBUG [2022-05-11 11:51:02,054] org.eclipse.jetty.io.ChannelEndPoint: flushed 32776 SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=W,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=COMMITTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5493}
DEBUG [2022-05-11 11:51:02,054] org.eclipse.jetty.io.ChannelEndPoint: flushed 23618 SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=W,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=COMMITTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5493}
DEBUG [2022-05-11 11:51:02,054] org.eclipse.jetty.server.HttpConnection: abort HttpConnection@4048662e::SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=ABORTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5493} {}
DEBUG [2022-05-11 11:51:02,055] org.eclipse.jetty.io.AbstractEndPoint: close SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=ABORTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5494}
DEBUG [2022-05-11 11:51:02,055] org.eclipse.jetty.io.AbstractEndPoint: close(null) SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=ABORTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5494}
DEBUG [2022-05-11 11:51:02,055] org.eclipse.jetty.io.ChannelEndPoint: doClose SocketChannelEndPoint@8e77fda{l=/172.31.16.150:8080,r=/172.31.29.207:10512,CLOSED,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@4048662e[p=HttpParser{s=END,14196 of 14196},g=HttpGenerator@2a0900de{s=COMMITTED}]=>HttpChannelOverHttp@4f92d404{s=HttpChannelState@7c12df1a{s=HANDLING rs=BLOCKING os=ABORTED is=READY awp=false se=false i=true al=1},r=1,c=false/false,a=HANDLING,uri=https://global-api.local.live:443/gameplay/103618036/submitscore,age=5494}


On Wed, May 11, 2022 at 12:29 AM sanketh reddy <sanre6@xxxxxxxxx> wrote:
Thanks will upgrade and recheck the issue. 

On Tue, 10 May, 2022, 9:28 pm Joakim Erdfelt, <joakim@xxxxxxxxxxx> wrote:
First, upgrade your version of Jetty.

9.4.11 is very old and is missing countless bug fixes and updates.
And is also subject to several security issues. See: https://www.eclipse.org/jetty/security_reports.php

Use at least 9.4.46.

The output shows that you have a request being processed by a webapp, consisting of 26090 bytes, and it's reading until the the content is closed by the client.
The client doesn't appear to be sending your the content in a timely manner, the sending is slow.
That's as far as we can go with this limited debug logs on that very old Jetty version.

Upgrade, use DEBUG again, and we should see more details that might reveal something else.

Joakim Erdfelt / joakim@xxxxxxxxxxx


On Tue, May 10, 2022 at 10:32 AM sanketh reddy <sanre6@xxxxxxxxx> wrote:
Hi Jetty Users,

I am facing a peculiar problem in jetty server 9.4.11.v20180605. There seems to be a delay in request parsing AFAIK from the debug logs affecting about 1% of the requests.
How do I solve this issue ? Do we have any timeout option to restrict this on the server side. Attaching the debug logs below of one of the request where you see the age getting bumped up till 15 seconds.

Any suggestions are welcome.


DEBUG [2022-05-10 11:30:43,805] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,7144 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=2/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,7144 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}
DEBUG [2022-05-10 11:30:43,805] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,8192 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,8192 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}
DEBUG [2022-05-10 11:30:43,805] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,8192 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,8192 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2}
DEBUG [2022-05-10 11:30:45,941] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,16384 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2138}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,16384 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2138}
DEBUG [2022-05-10 11:30:45,941] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,16384 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2138}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,16384 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=2138}
DEBUG [2022-05-10 11:30:59,012] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,24576 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15209}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,24576 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15209}
DEBUG [2022-05-10 11:30:59,012] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,24576 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15209}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=CONTENT,24576 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15209}
DEBUG [2022-05-10 11:30:59,251] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=END,26090 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15448}<-SocketChannelEndPoint@60b4f649{/172.31.23.178:22996<->/172.31.16.150:8080,OPEN,fill=-,flush=-,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@608e8c9f[p=HttpParser{s=END,26090 of 26090},g=HttpGenerator@27cf3879{s=START}]=>HttpChannelOverHttp@399cc4bf{r=1,c=false,a=DISPATCHED,uri=https://global-api.mpl.live/gameplay/103619231/submitscore,age=15448}
DEBUG [2022-05-10 11:30:59,307] org.eclipse.jetty.server.HttpConnection: releaseRequestBuffer HttpConnection@608e8c9f[p=HttpParser{s=CLOSED,0 of -1}





--
Thanks,
sanre6
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users


--
regards
sanketh

Back to the top