J1bb

So I play with my two other friends, mostly one of them though, and we already find this game not a challenge at all 5 days in. I set the difficulty to 3 in the server properties file, but it feels the same. We still take like 1 damage to everything that hits us. Am I missing something?