It's most likely one of two things, you either have less than 4GB of RAM, or your default Java virtual machine isn't the 64-bit version. If you have 64-bit windows, you can fix this easily by uninstalling all JAVA instances, then reinstalling 32-bit Java, and then 64-bit Java. Doing this will allow IE11 to work properly with sites that use Java, and will make the last one you installed the system VM (64-bit.)
Alternatively, if you have 64-bit java, you can run it directly from "C:Program FilesJavajre#binjava.exe" replacing the # symbol with either 7 or 8 depending on which version you have installed, so you'd just replace the "java" part in the batch file with that entire string, including the quotes.
If you have less than 4GB of RAM, you need to edit the environment or check the Java control panel iin order to change the values you have entered. The environment can be edited through the System control panel icon, then (With Windows 7) Advanced System Settings..., and there's a button called environment variables. The VM options in the Java applet are under the "Java" tab, and then click on View... The column is labeled Runtime Parameters (At least with Java 8.)