NettyIO Exception Spam

[applelove]

Howdy folks,

I've been running a private mod pack (forge, no bukkit/spigot/etc) for friends for a bit over a year now with updates here and there. Recently we've run into a problem where if a user "lags out" or logs out during particularly bad network lag, the server seems to go nuts spamming and NettyIO Thread error:

[21:53:26] [Netty IO #7/ERROR] [FML/]: NetworkDispatcher exception
java.io.IOException: An established connection was aborted by the software in your host machine
	at sun.nio.ch.SocketDispatcher.writev0(Native Method) ~[?:1.8.0_92]
	at sun.nio.ch.SocketDispatcher.writev(Unknown Source) ~[?:1.8.0_92]
	at sun.nio.ch.IOUtil.write(Unknown Source) ~[?:1.8.0_92]
	at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[?:1.8.0_92]
	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:249) ~[NioSocketChannel.class:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:680) [AbstractChannel$AbstractUnsafe.class:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:266) [AbstractNioChannel$AbstractNioUnsafe.class:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:488) [NioEventLoop.class:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:447) [NioEventLoop.class:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:341) [NioEventLoop.class:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [SingleThreadEventExecutor$2.class:?]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_92]

This and the rest of the stack trace will spam at about 1,000+ lines per second indefinitely. The above is the full stacktrace that gets repeated thousands of times. My record for log file size is 7GB before the server started behaving strangely and desyncing with clients/

Once the client disconnects from their end, they remain as a "ghost" player - their entity still exists in the world and cannot be kicked or removed. They can even log back in and see their old entity, but quickly lag out again. This only seems to impact the user that lagged out originally, but has been encountered by all of the users at one point or another. It doesn't seem like the log spam ever subsides, or that the client can "wait out" the lag instead of logging out. 

Things I've tried or looked into:

• Server resources are fine - 11GB RAM allocated, 7 used - CPU ~70%
• All necessary ports are open
• Firewall exceptions are made for appropriate ports
• No AV or third party firewalls or other network managing software
• No recent ISP change
• Network seems to be fine - 150 down, 10 up - We seem to use about 1-4mbit upload at peak usage.

 

The only way to fix the issue has been to restart the server, which isn't very good since I am not available 24/7 to reboot every time a player lags a bit too much. This is a fairly recent issue, and while mods have changed, none are things that I would expect to cause server issues like this. Morpheus is the only server-only mod that has been added, and other mods are all either updates or benign things like Advanced Genetics.

I've tried everything I can think of. We can replicate the lagout issue, but I can't identify a cause or solution. It's almost like the server just can't register that the player has left, and tries to send a billion block and chunk updates to their dead session. Anyone here ever run into something similar? Any suggestions?

Edited February 28, 2017 by applelove