B4J Question Excessive org.eclipse.jetty.io.EofException after switch to HTTP/2?

[gravel]

I've a B4J server running a website and serving as the backend for a B4A app. The main content of the site is a 'live' map, AJAX updated every minute and it's been running well for a number of years with HTTP/1.

I switched to HTTP/2 (Java 11 without Conscrypt) early on the 5th July and in the following 42 hours the server threw 547 org.eclipse.jetty.io.EofException: errors, 391 with "Reset" & 156 with HTTP/2 CANCEL_STREAM_ERROR

I would expect to see some errors, but nearly 550 in 42 hours seems rather high as during the 42 hours, according to analytics, the website had approx 1,300 visitors ... one error for every 2-3 visitors?

But during the time, the server seemed to operate as normal with no unusual signs of CPU or memory problems and when I loaded the site myself it seemed to behave normally and indeed the site did seem a bit faster ... good!.

But this level of org.eclipse.jetty.io.EofException surely can't be normal can it? Internet searches didn't throw up anything about frequency of error and I'd expect to find at least some comment about it.

So is this normal, or does anyone have any ideas about what might be causing this volume of errors?

If the volume of error is 'normal' is there any way to supress the stacktrace for them?

Regarding supressing the stacktrace I think this misleadingly titled Jetty issue committed Feb 4, 2019 regarding stacktrace logging might apply to these io errors
https://github.com/eclipse/jetty.project/issues/3317
The version of Jetty in B4J is from before that date I think ... is there a plan to update the B4J Jetty version any time soon?

A sample check of the 2 types of error ( "Reset" & HTTP/2 CANCEL_STREAM_ERROR) indicates they are all the same as the 2 below:

org.eclipse.jetty.io.EofException: reset
    at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:177)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
    at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:695)
    at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:667)
    at org.eclipse.jetty.http2.HTTP2Stream.headers(HTTP2Stream.java:107)
    at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.sendHeadersFrame(HttpTransportOverHTTP2.java:235)
    at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.send(HttpTransportOverHTTP2.java:134)
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:787)
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:843)
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240)
    at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:777)
    at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:876)
    at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:753)
    at org.eclipse.jetty.server.ResourceService.sendData(ResourceService.java:732)
    at org.eclipse.jetty.server.ResourceService.doGet(ResourceService.java:295)
    at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:458)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
    at anywheresoftware.b4j.object.JServlet$Handle.run(JServlet.java:135)
    at anywheresoftware.b4j.object.JServlet.Handle(JServlet.java:110)
    at anywheresoftware.b4j.object.JServlet.doFilter(JServlet.java:72)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at anywheresoftware.b4j.object.JServlet$Handle.run(JServlet.java:135)
    at anywheresoftware.b4j.object.JServlet.Handle(JServlet.java:110)
    at anywheresoftware.b4j.object.JServlet.doFilter(JServlet.java:72)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)

org.eclipse.jetty.io.EofException: HTTP/2 CANCEL_STREAM_ERROR
    at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:171)
    at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:521)
    at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:308)
    at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:245)
    at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:299)
    at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66)
    at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:182)
    at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:110)
    at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
    at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:231)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
    at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:165)
    at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:121)
    at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:290)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
    at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.lang.Throwable: HttpInput failure
        at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:821)
        at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:338)
        at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:219)
        ... 32 more

 

[Erel]

I haven't encountered this issue. Based on this discussion: http://jetty.4.x6.nabble.com/Jetty-9-4-24-prints-warning-messages-to-stderr-td4968368.html it might be over-verbose logging.

 

[gravel]

Thanks for that link Erel.

Yes it does seem very similar to the issue discussed in that link and I think it is only happening on full initial load of assets, but only in some particular set of circumstances.

What the circumstances are is a bit hazy as I can only replicate it occasionly and unpredictably. In my tests I've not seen it happen with any of the AJAX requests. From the client end there doesn't seem to be a problem, and probably it's as you suggest, Jetty being over-zealous in 'calling errors'.

So, for the moment I've switched off HTTP/2 until I can work out how to filter the logs to exclude the stack traces for these errors, but not stack traces for other errors I do want to know about.

If I learn anything more I'll post again.