I didn't find the cause of problem but fixed the effect, just installed mob spawn control that watches over number of mobs on the server and I guess it despawned them where ever they were spawned and now it's all right.
That might be some bug with dimensional doors because when I left the limbo I spawned 50 kilometres from home and I travelled all the way back eventually, they might get stuck somewhere on the way.