This reverts commit 04aa09c4bcf2d5a634a35da3a3ae3fc1abe30ef8
and reintroduces 0ff5567a30be6d7c804e95997ae282d6bacd76c3 that
was temporarily reverted due to minor regressions.
It also reverts e5bce8b4ce7b1f3a83998febdfa86a3771df96ce that fixed FATE refs.
The fate-ffm change is caused by field_order now being set
on the output format because the first frame arrives earlier.
The fate-mxf change is assumed to be the same.