Generally the mods will give you instructions on how to install them so pay attention to their instructions. However, neither Tekkit nor Big Dig are based on Bukkit which is what provided the plugins folder. They are all now based on Forgecraft which has a mods folder and generally the same jar/zip you would use for single player is used for multiplayer. Just drop the jar/zip into the mods folder on both server and client and you are good to go.