賽 originally combined 貝 (shell, used as money) with a phonetic element that also suggested competition or offering, relating to gambling with shells or presenting tribute at a temple.
Meaning mnemonic
Under a roof (宀), two hands (ハ) hold a shell (貝) as a die to roll, or as an offering at a temple visit.
Reading mnemonic
For サイ, imagine rolling dice and shouting 'sai!' as they land: sai -> サイ, matching the gambling and chance meaning.