My sister ... study at school on holidays.

The sentence expresses a lack of obligation. 'Doesn't have to' indicates that studying is not required. 'Mustn't' implies prohibition. 'Has to' indicates obligation. Therefore, 'doesn't have to' is the correct choice.

My sister doesn't have to study at school on holidays.