To answer your first problem, your system ran out of RAM (and virtual memory) to run the server. How much RAM you got installed on your PC?
Secondly, I do hope you're running THIS server: http://mirror.technicpack.net/Technic/servers/bigdig/BigDigServer-v1.3.9.zip
You launch the server with launch.bat. By default, it uses between 2-3GB of memory.