When is it safe to remove the head support in a baby seat?

My little boy is nearly 3 months old and supports his head really well. Recently he's been wriggling alot when he's in his car seat and he tries to look from side to side. I thought this might be because the head support is irritating him. At what age is it safe to remove the head support?