I've found that anything more then 2GB is unnecessary and can actually degrade performance, this experience is from my private pack with 130+ mods on a server with about 15+ active members.
Really 1.5GB - 2GB seems to be the sweet spot even if you have 16GB+ memory in your system.
Enabling increase permgen also needs to be set server side as well or you will notice random crashing etc. on your server as you do with your client.
To enable this server side add "-XX:MaxPermSize=256M" (no quotes) to your startup batch file.