Add additional sanity checks to RSA verification code.

Also, make algorithm unsigned int in most places.

BUG=chrome-os-partner:701
TEST=existing RSA verification tests still pass

Review URL: http://codereview.chromium.org/3136017
10 files changed