It all basically depends on the machine that you are running the server on. A reasonable gaming rig, even a bit dated (gaming computer years relate to human years at about 1:30), can support dozens (hundreds?) of chunk loaders without breaking a sweat, when running a server (and a client).
If, however, you are using a computer from the 1990s, and run both server and client on the same machine, and put down chunk loaders for a reason (to keep a complex processing plant going), then you are in trouble, yes. Although I'd guess that in this case the major contributor to the lag would be the electric/piping systems of the processing plant.