I hear his grandfather was a gambler/yakuza type.
Wandering gamblers often appear in period dramas.
Specifically a professional gambler; 渡世人 includes a broader sense of a rootless outsider.
Compound of 渡世 (tosei, 'making a living, getting by in the world') and 人 (nin, 'person'). The exact historical derivation is uncertain, but the term came to refer to those who lived by their wits outside conventional society.