Why do my villagers keep disappearing?

[Voldemort_5]

Me and my friend have a world, in which a few villagers have their own houses. There's a fence surrounding them, so I know they're not escaping. They simply vanish into thin air when we're not paying any attention.

Are they despawning, and if so, why? If not, what's causing this?

Either way, I'd like to stop this from happening. Some have had good trades, like 3 fish for an emerald.

Thank you in advance!

EDIT: Also, this isn't necessarily a server. Rather, it's a LAN world.

[Richs_Yard]

Zombies. Did you put down any Iron golems?