How to remove a vote for an answer as "best answer"?

You can switch from voting for one answer to voting for another, simply by clicking on the Best Answer button for the second response. But I don't know if it's possible to just undo the vote without selecting another.

I don't think that you can.