tree c20ba4b3e0d0302b1670f7b527cfabccf7d757a2
parent d45d8933e61ccd8bd50fcf58f33a604627e6552c
author David Benjamin <davidben@google.com> 1659566185 -0400
committer Boringssl LUCI CQ <boringssl-scoped@luci-project-accounts.iam.gserviceaccount.com> 1660591585 +0000

Tighten up supported PSS combinations in X.509.

Matching Chromium, Go, and TLS 1.3, only allow SHA-256, SHA-384, and
SHA-512 RSA-PSS signatures, where MGF-1 and message hash match and salt
length is hash length. Sadly, we are stuck tolerating an explicit
trailerField for now. See the certificates in cl/362617931.

This also fixes an overflow bug in handling the salt length. On
platforms with 64-bit long and 32-bit int, we would misinterpret, e.g,
2^62 + 32 as 32. Also clean up the error-handling of maskHash. It was
previously handled in a very confusing way; syntax errors in maskHash
would succeed and only be noticed later, in rsa_mgf1_decode.

I haven't done it in this change, but as a followup, we can, like
Chromium, reduce X.509 signature algorithms down to a single enum.

Update-Note: Unusual RSA-PSS combinations in X.509 are no longer
accepted. This same change (actually a slightly stricter version) has
already landed in Chrome.

Bug: 489
Change-Id: I85ca3a4e14f76358cac13e66163887f6dade1ace
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/53865
Auto-Submit: David Benjamin <davidben@google.com>
Reviewed-by: Adam Langley <agl@google.com>
Commit-Queue: Adam Langley <agl@google.com>