avcodec/diracdsp: Fix integer anomaly in dequant_subband_*

Fixes: negation of -2147483648 cannot be represented in type 'int32_t' (aka 'int'); cast to an unsigned type to negate this value to itself
Fixes: 23760/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DIRAC_fuzzer-604209011412172

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Michael Niedermayer 2020-07-18 14:30:19 +02:00
parent 5dae33bb39
commit ca3c6c981a

View File

@ -198,9 +198,9 @@ static void dequant_subband_ ## PX ## _c(uint8_t *src, uint8_t *dst, ptrdiff_t s
PX c, sign, *src_r = (PX *)src, *dst_r = (PX *)dst; \
for (i = 0; i < tot_h; i++) { \
c = *src_r++; \
sign = FFSIGN(c)*(!!c); \
c = (FFABS(c)*(unsigned)qf + qs) >> 2; \
*dst_r++ = c*sign; \
if (c < 0) c = -((-(unsigned)c*qf + qs) >> 2); \
else if(c > 0) c = (( (unsigned)c*qf + qs) >> 2); \
*dst_r++ = c; \
} \
src += tot_h << (sizeof(PX) >> 1); \
dst += stride; \