tpm2: avoid comparison between signed and unsigned ints
The marshaling/unmarshaling code is using integer values to represent
room left in the buffer, to be able to communicate three conditions:
positive number means there is room left in the buffer, zero means
that the exact amount of data in the buffer was unmarshaled and
negative value means that the result of the operation did not fit into
the buffer.
The implementation is wrong though, as it compares directly signed and
unsigned values, which is illegal, as signed values get promoted to
unsigned by the compiler.
This patch changes the marshaling code to use size_t for the size, and
use zero as marshaling failure indication - after all the buffer where
the data is marshaled to should definitely be large enough, and it is
reasonable to expect at least some room left in it after marshaling.
The unmarshaling situation is different: we sure want to communicate
errors to the caller, but do not want to propagate error return values
through multiple layers. This patch keeps the size value in int, but
checks if it is negative separately, before comparing with positive
values.
BRANCH=none
BUG=chrome-os-partner:50645
TEST=with the rest of the patches applied kevin successfully boots up.
Change-Id: Ibfbd1b351e35e37c8925a78d095e4e8492805bad
Signed-off-by: Martin Roth <martinroth@chromium.org>
Original-Commit-Id: b1e862c2a650fa5f6cb25a01fe61e848a696cf17
Original-Change-Id: Ie7552b333afaff9a1234c948caf9d9a64447b2e1
Original-Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Original-Reviewed-on: https://chromium-review.googlesource.com/358772
Original-Reviewed-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-on: https://review.coreboot.org/15610
Tested-by: build bot (Jenkins)
Reviewed-by: Patrick Georgi <pgeorgi@google.com>
2 files changed