Novel!Kazuo was born with no emotions and no mother; manga!Kazuo lost both when he was about four. I think in both there's an implication that Kazuo was also illegitimate/not his father's child. And the family were very rich. So... there was probably a bit of a funny atmosphere. Whereas while Ken's parents are annoying as heck, at least they're both there. And Ken has his brain intact.