That version of Vista supports up to 16GB of RAM. The 4GB problem you've heard of is related to 32 bit OSes. With a 64 bit system you could theoretically use 16 exabytes, but Windows and modern hardware limitations get in the way.
That said, you should try to allocate physical RAM only. Readyboost is a fancy form of swap space that, while faster than your hard drive, takes ages to access from the CPU's perspective when compared to RAM. It can help if you're already hitting swap on your hard drive, but otherwise it's not worth it. Also keep in mind that Vista and any background processes will want their own RAM, so you shouldn't use all 4GB for Tekkit.
I'm running a small FTB server on an old laptop with 2GB of RAM with the major bottleneck being my upload speed. With a couple people online it uses around 1GB. You should be fine with 2GB for yours, depending on how many people you play with.
TL;DR Use 2GB for the server and 1-1.5GB for the client.