only when; particularly in the case of; of all times/people
Attaches to a noun and marks it as the exclusive case where something (usually negative or unexpected) happens. Often implies 'why now/this person of all times/people?' or 'it had to be this one.'
The bus is late only when I'm in a hurry.
He, of all people, would never say something so rude.
ばかり expresses 'nothing but' or a repeated occurrence, whereas に限って singles out one specific case, often implying 'why this one, of all times?'
Formed from the particle に and the te-form of the verb 限る (to limit). Literally 'limiting to', it grammaticalized into an expression meaning 'only in the case of' with an added nuance of unexpectedness.