They didn't (to have) a birthday party 3 days ago.

The sentence is in the past tense, indicated by "didn't". The verb "to have" needs to be in its base form after "didn't". Therefore, the correct form is "have".
Answer: have