It's generally best not to because they are so delicate, however it is a myth that a mother rabbit will abandon her babies if a human touches them. This is actually not true of any species.
It generally isn't necessary to touch them though. In the earliest stages of their lives, their mother does everything for them, so it's best not to disturb her. It's only really necessary if there is some sort of danger (i.e. if there is a medical emergency with a baby and it needs to see a vet) or if of course the mother is no longer around and the newborns are being raised by a human. Although that last one's pretty self-explanatory!
