It seems that even if you configure custom JVM arguments on the Launcher Options, they are not being used. It happens on all plataforms. Windows, Linux or OSX. This is my configuration: You can check it's not being applied in multiple ways:
1) FML Logs
-- System Details --
Details:
Minecraft Version: 1.7.10
Operating System: Linux (amd64) version 3.19.0-28-generic
Java Version: 1.8.0_60, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 681338400 bytes (649 MB) / 918552576 bytes (876 MB) up to 3817865216 bytes (3641 MB)
JVM Flags: 2 total; -Xmx4096m -XX:MaxPermSize=256m
2) jvisualvm
3) Even a "ps" on Linux showing the cmdline.
edgard@lynxarc:~/.technic/modpacks/bmt/logs$ ps -ef | grep -i java | grep bmt
edgard 29173 26603 80 22:13 ? 00:03:25 /usr/lib/jvm/java-8-oracle/jre/bin/java -Xmx4096m -XX:MaxPermSize=256m -Djava.library.path=/home/edgard/.technic/modpacks/bmt/bin/natives -Dfml.core.libraries.mirror=http://mirror.technicpack.net/Technic/lib/fml/%s -Dminecraft.applet.TargetDirectory=/home/edgard/.technic/modpacks/bmt -Djava.net.preferIPv4Stack=true -cp /home/edgard/.technic/modpacks/bmt/bin/modpack.jar:/home/edgard/.technic/cache/net/minecraft/launchwrapper/1.11/launchwrapper-1.11.jar:/home/edgard/.technic/cache/org/ow2/asm/asm-all/5.0.3/asm-all-5.0.3.jar:/home/edgard/.technic/cache/com/typesafe/akka/akka-actor_2.11/2.3.3/akka-actor_2.11-2.3.3.jar:/home/edgard/.technic/cache/com/typesafe/config/1.2.1/config-1.2.1.jar:/home/edgard/.technic/cache/org/scala-lang/scala-actors-migration_2.11/1.1.0/scala-actors-migration_2.11-1.1.0.jar:/home/edgard/.technic/cache/org/scala-lang/scala-compiler/2.11.1/scala-compiler-2.11.1.jar:/home/edgard/.technic/cache/org/scala-lang/plugins/scala-continuations-library_2.11/1.0.2/scala-continuations-library_2.11-1.0.2.jar...
