If you are talking about the server you need to edit the lunch.bat.
change this line "java -Xmx3G -Xms2G -jar Tekkit.jar nogui"
To: java -Xmx(Amount)(G = Gigs, M= Megabit ) -Xms(Amount)(G = Gigs, M= Megabit) -jar Tekkit.jar nogui
Amount is what ever you want to set it to.
Now for the client you can set it under options with in the launcher.
[img width=600]