ホラー小説 is a more modern, direct loanword for horror novels, whereas 怪奇小説 has a slightly older or more literary feel.
Etymology
Compound of 怪奇 (mysterious, bizarre) and 小説 (novel). The exact historical derivation is uncertain; the spelling is conventionally associated with mystery and horror fiction.