Lag Diagnosis Tools

[littleweseth]

What tools exist to diagnose lag on a Tekkit server?

I'm aware of the NoLagg plugin for Bukkit, but that seems mainly focused on diagnosing plugin lag. I don't believe it will diagnose lag caused by mods like RP2, IC2 or BC.

I believe that someone mentioned it was possible to profile the Minecraft server and get performance statistics broken down by Java class, which lets you see which mod is chewing all the CPU cycles. Anyone have a pointer to this or any other tool that will let me diagnose mod-related lag?

Edit: Background - my server started spewing "can't keep up" messages today. No-one was actually on the server when it started. A restart fixes it for a while but it just comes back. Already tried WorldEdit //remove items -1 (spilled items are not the cause.)

[Nentify]

I'd suggest using VisualVM, at least, it's what I use. If you need further assistance, just drop me a PM. Also - that message occurs any time the TPS drops below 20, you can disable it in the bukkit.yml.

[littleweseth]

I'd suggest using VisualVM, at least, it's what I use. If you need further assistance, just drop me a PM. Also - that message occurs any time the TPS drops below 20, you can disable it in the bukkit.yml.

Yes, I know that the "can't keep up!" message occurs when the ticks per second drops below 20. The problem is that the TPS, as measured by NoLagg, is on the order of **0.8 TPS**. D:

I'll look into using VisualVM. My server runs on FreeBSD so I may need to take a local copy of it to use VisualVM on my Windows box.

[Expozay]

I'd suggest giving giving LagMeter a go for monitoring your server performance.

http://dev.bukkit.org/server-mods/lagmeter/

[Nentify]

I'd suggest giving giving LagMeter a go for monitoring your server performance.

http://dev.bukkit.org/server-mods/lagmeter/

LagMeter simple reports the TPS, which NoLagg also does. That's all fine and dandy, but they want to find out what is causing the lag, and any possible solutions to solving it.

[prenetic]

A couple of nice built-in options to invoke at run-time will let you know what's going on with garbage collection within your JVM. I've used this data to finely-tune which GC methods are used as well as YG/OG sizes.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

If you see a lot of full GCs it's likely your OG has been completely consumed and attempts to prune are failing due to resources in the YG becoming promoted and tenured, effectively undoing the work that was just done to clean up the OG. Best thing to do at that point is restart your server after tuning your GC configuration.

This is assuming you're suffering from memory issues, of course.

[Aerik]

Hmm, same issue, but OG/YG sounds to me like the noise a zombie would make... Any chance of a practical solution/explanation for a noob who is willing to learn?

[Expozay]

LagMeter simple reports the TPS, which NoLagg also does. That's all fine and dandy, but they want to find out what is causing the lag, and any possible solutions to solving it.

It's still a tool that helps diagnose, for someone who runs their own server, I'm surprized. It's one way to see if it's just your connection or the server. Little, still useful.

[littleweseth]

Hmm, same issue, but OG/YG sounds to me like the noise a zombie would make... Any chance of a practical solution/explanation for a noob who is willing to learn?

"OG" and "YG" refer to the internal details of Java memory management. The Java garbage collector divides objects in memory into a young generation (YG) and and old generation (OG). These are garbage collected at different rates. Generally the default rates work fine, but the garbage collection can be tuned in the manner @prenetic is suggesting.

Google keywords for this are "Java garbage collector tuning". It's black magic.

[prenetic]

+1 to the above.

Black magic demystified a bit, in the context of Minecraft.

[Sp0nge]

I could prob quote our dev on another forum:

I'm using JVisualVM and the remote "debugger" on our linux server. Its a proper cpu profiling tool.

So i suggest you stop using/looking for some halfassed tools aswell and go for proper things that tell you the packages, classes, methods, cpu times, wait times, thread deadlocks, memory/heap graphs etc.

For the classes and methods you can use the sampler snapshot to see per thread hot spot calls trees to find the issue chains, just knowing a mod/class/method otherways tells nothing.