Fix MSI DigitalSignature handling with MsiDigitalSignatureEx

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #507

Current Behavior

MsiDigitalSignatureEx is prepended to the MSI content digest, while the corresponding stream is skipped during stream hashing.

This produces an incorrect DigitalSignature digest when another MSI stream sorts before MsiDigitalSignatureEx.

New Behavior

MsiDigitalSignatureEx is included in the MSI content digest at its sorted stream position.

The same digest ordering is used for signing and verification. During verification, the recalculated MsiDigitalSignatureEx value is also checked against the stored value.

Scope of Changes

• Calculate MsiDigitalSignatureEx separately from the MSI content digest.
• Hash MsiDigitalSignatureEx at its sorted stream position.
• Use the same digest ordering for signing, verification, and detached-signature digest calculation.
• Verify the recalculated MsiDigitalSignatureEx value against the stored value.

Testing

• Existing tests
• New tests added
• Manual testing

Verified an affected MSI file successfully with both osslsigncode and Windows signtool.

Additional Notes

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Signing and verification now disagree on MSI digest ordering

msi.c:480 — Verification now hashes MsiDigitalSignatureEx at its sorted position, but signing still prepends it in msi_calc_MsiDigitalSignatureEx(). When another stream sorts before it, this branch successfully signs a file that it cannot subsequently verify. Previously generated signatures also stop
verifying.

Reproducer using the existing fixture, renaming its embedded-cabinet stream so it sorts before DSE:

python3 - <<'PY'
from pathlib import Path
b = bytearray(Path("tests/files/unsigned.msi").read_bytes())
b[7168] = 0
Path("early-stream.msi").write_bytes(b)
PY

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout key.pem -out cert.pem -subj /CN=Review \
  -days 1 -addext extendedKeyUsage=codeSigning

build/osslsigncode sign -certs cert.pem -key key.pem \
  -add-msi-dse -in early-stream.msi -out signed.msi
build/osslsigncode verify -CAfile cert.pem -in signed.msi

Result: base passes; this branch fails with Calculated DigitalSignature ... MISMATCH!!!, although the stored and calculated DSE values match. Both versions produce signatures accepted by base and rejected by this branch.

The unmodified fixture with DSE, and the modified fixture without DSE, pass on both versions.

Suggestion: make signing and verification use the same digest ordering, including the detached-signature digest path. Explicitly address compatibility with previously generated signatures and add an early-sorting-stream regression test.

[mtrojnar]

Also update README.md. Consider adding something like:

Compatibility note: MSI files previously signed with -add-msi-dse whose streams sort before MsiDigitalSignatureEx must be re-signed to verify with this version. Corrected signatures for these files will not verify with older osslsigncode versions. Other MSI files are unaffected.