Yeah I figured that ninjakivi2, I must have been out of it when I replied the first time.
How much memory do you have allocated? I've read some things about memory, and found that increasing your memory allocation "too far" your FPS can actually drop again, and give you serious lag. and likewise, the vanilla allocation is probably not enough for the mods.
As the smart people above me have already said: Optifine is there to improve framerate, but if you want to still try and remove it you could try this: In theory, depending if your up to it, you could manually remove all of optifines '.class' files from the bin.jar. But the trick is, that Optifine (I'm pretty sure) modifies vanilla .class files. So you could do a side-by-side comparison (or find out which files it changes), then re-insert the vanilla files to re-overwrite the Optifine ones.
Hope that made sense, I could be out of it again... At any rate, if you give this a shot I have one thing to say... BACKUP YOUR .JAR FIRST! that is all.