I'm so worried about hurting baby's head?

I have a 5 month old baby boy - and when ever I bathe him or carry him he tries to wriggle out of my grasp - especially when I'm bathing his head - so I have to hold his head tight to prevent it from fall backwards and also to prevent him from falling down when I carry him. I do my best to hold his head with my palm but sometimes I have to use my fingers also - can this be hurting his brain in anyway? or is the skull tougher than I think?