dice, temple visit
賽 centers on dice and games of chance, and by extension a temple visit where one might seek divine favor or draw lots.
賽 originally combined 貝 (shell, used as money) with a phonetic element that also suggested competition or offering, relating to gambling with shells or presenting tribute at a temple.
Under a roof (宀), two hands (ハ) hold a shell (貝) as a die to roll, or as an offering at a temple visit.
For サイ, imagine rolling dice and shouting 'sai!' as they land: sai -> サイ, matching the gambling and chance meaning.
—