Decoder accepts non-minimal integer encodings

[trackoor]

Summary

The MessagePack specification directs an encoder to use the shortest representation that holds the value, so a decoder that accepts a wider encoding admits a second encoding of a value its own encoder would never emit.

Specification: https://github.com/msgpack/msgpack/blob/master/spec.md

Problem

msgpack-javascript accepts an input that its own encoder would
never produce. Decoding the witness below succeeds, and re-encoding the
decoded value with the same library yields different bytes. The library is
therefore not the inverse of itself on this input, so two peers that disagree
about whether to accept it will disagree about the value's encoding, and any
system that treats the encoding as identifying (content addressing, signature
preimages, deduplication) can be made to see two encodings of one value.

Version tested: @msgpack/msgpack 3.1.3

Reproduction

import { decode, encode } from '@msgpack/msgpack';
const buf = Buffer.from('cc6c', 'hex');
const v = decode(buf);
const re = Buffer.from(encode(v));
console.log('input   :', buf.toString('hex'));
console.log('decoded :', JSON.stringify(v, (_k, x) => typeof x === 'bigint' ? x.toString() : x));
console.log('re-encode:', re.toString('hex'));
console.log(re.equals(buf) ? 'ROUNDTRIP-STABLE' : 'REENCODE-DIFFERS');

Observed output:

input   : cc6c
decoded : 108
re-encode: 6c
REENCODE-DIFFERS

The re-encoding differs from the input, which is the defect.

Expected behavior

The decoder should reject this input rather than accept it, because the
encoding is not the canonical one for the decoded value. A canonical encoding of the same value is 9334406d, which round-trips byte-identically.

Suggested fix

While decoding an integer argument, check that the value could not have been
written in a shorter form, and reject it if it could. Normalising at encode
time does not help: by then the non-canonical bytes have already been accepted.