Go your profile and click on "fans".
You will see a list of icons and the names of the users you fanned. Next to their name you see a little red cross. When you click on the cross the user is deleted from your list of fans.
Note: The person will NOT be informed. But some users keep a close eye on their fancount and might research if they see it dropping. They will however NOT get a message saying they were dropped from someone else's fanlist.