Hello,
I had exactly this problem last night, and only stumbled across a solution that may work for you by accident.
Check the server configuration file for an option to allow nether (true/false). If set to false this will prevent new dimensions (including Mystcraft's) from working.
Hope that helps.