Actually your hardware is only the start of it, you need a good upload/download rate also.
I have a 20 man server with 3GB ram and a Dual core 2.4 ghz. My upload is 3mb and i use about 1/6 of that.
With about 12 plugins the server has been constantly running for the past 4 weeks, the only downtime being when people openly abused Nukes, which after removing was a simple matter.
Try getting Nolagg on the server to help make it run better and more sustainable.
As a minimum you want atleast 1.5Gb and a dual core of any speed +2.0 and atleast 1mb upload or your friends are going to lag horribly.