I think The Doctor is the one that created the cracks, probably accidently, rewriting history and editing it. The cracks of sucking in things and erasing their existence from time entirely. Time travler's brains aren't effected in the same way because their timeline isn't linear. The TARDIS might have something to do with it too.