Odd Big Dig Server Issues

[xb2003]

I just started locally hosting a Big Dig server. I have a couple years of hosting a Bukkit server, but this is my first dealing with Technic. Anyways, I had the server setup in one place, but I moved it home because I have a much stronger internet connection there. (15 down, 2-3 up). And now, after moving it, I am getting terrible lag. But its just me, there can be 5 others on the server and none of them lag a bit. I can't hardly move. This problem kind of breaks off into two sections here:

First, while no one else is getting lag, I get the overload message on the server a ton. And every once in a while it kicks everyone off momentarily. The server is a 2.3GHz Core 2 Duo, with 2 GB of ram. I allocate 1.75GB to the server, Permgen at 256MB. Java uses merely half of that when with 5 people online. And CPU usage never spikes to more then 40%. With my Bukkit servers, 20 people with MultiWorld installed didn't lag a bit.

Me personally, my lag is so bad I cant hardly move. Which leads me to believe its my internet connection. My computer is an i5 3570, 8GB Ram, HD 7700. Not great but i get 100+ fps on Big Dig Single player(and when i hosted the server on that computer) easily with high settings. But my internet is more then capable of gaming. I've checked if I'm dropping packets or something, nope. I also have PermGen enabled and 2 or 3 GB (I don't remember) allocated to Technic.

If you can show me what I have done wrong, it would be greatly appreciated.

Thanks,

Jesse