When media element is inactive, asking MDSM to seek is actually useless, because all the data we decode won't be showed to user. In addition, we have to store extra pending events for `seeking` and `seeked`, which might result in memory overflow if the inactive page is calling `seek()` all the time.
Therfore, we should delay all seeking tasks while media is inactive, and perform the latest seeking task when media becomes active.
Differential Revision: https://phabricator.services.mozilla.com/D58918
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
Only SeekToNextFrame cares about promises, but prior to this patch the common
method HTMLMediaElement::Seek() would always return a promise.
When the caller was not SeekToNextFrame (e.g., SetCurrentTime, FastSeek), the
promise would end up *not* being exposed. When later rejected, we would catch
this and write an error to the js console.
This patch lifts the handling of the promise out of Seek() and into
SeekToNextFrame() so that we avoid creating promises that would not get exposed
to js.
Differential Revision: https://phabricator.services.mozilla.com/D42511
--HG--
extra : moz-landing-system : lando
Only SeekToNextFrame cares about promises, but prior to this patch the common
method HTMLMediaElement::Seek() would always return a promise.
When the caller was not SeekToNextFrame (e.g., SetCurrentTime, FastSeek), the
promise would end up *not* being exposed. When later rejected, we would catch
this and write an error to the js console.
This patch lifts the handling of the promise out of Seek() and into
SeekToNextFrame() so that we avoid creating promises that would not get exposed
to js.
Differential Revision: https://phabricator.services.mozilla.com/D42511
--HG--
extra : moz-landing-system : lando
Create an event in MediaFormatReader the will signal to the HTMLMediaElement to initiate a new storing according to the latest VideoInfo. Also when the application is shutting down, trigger a new storing early enough, before all the events are disconnected.
Differential Revision: https://phabricator.services.mozilla.com/D38316
--HG--
extra : moz-landing-system : lando
This is inherently large, because modifying these bits of DOMMediaStream and
MediaStreamTrack affects all consumers and producers of all DOMMediaStreams and
MediaStreamTracks.
Things are generally much simpler now.
Producers of tracks now create a MediaStream in the graph, add it to a
MediaStreamTrackSource subclass that takes ownership of it, and add the source
to a MediaStreamTrack. Should the producer need a DOMMediaStream it is now much
simpler to create as the only thing needed is the current window. The stream is
a rather simple wrapper around an array of MediaStreamTracks.
HTMLMediaElement is still not as straight forward as other consumers since it
consumes the DOMMediaStream directly, as opposed to a set of tracks.
The new MediaStreamRenderer helper class helps bridge the gap between this fact
and the new track-based MediaStreamGraph interface, as it needs to juggle
registering multiple audio tracks for audio output. This hooks into existing
HTMLMediaElement logic and brings a welcome simplification to all the glue
previously needed there.
Differential Revision: https://phabricator.services.mozilla.com/D37934
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This removes DecodedStream's use of MediaStreamListener in favor of
MediaStreamTrackListener. This change has however rippled through to a lot
more cleanup, per below.
This moves the MediaStreamTrack lifetime ownership for captured
HTMLMediaElements from the media element to DecodedStream, where the
MediaStreamGraph-side tracks are already created and ended today.
This makes MediaStreamTrack creation explicit across the entire codebase and
lets us remove the MediaStreamTrackSourceGetter class and the infrastructure
of adding MediaStreamTracks after they've already been created in the graph
from DOMMediaStream.
With track ownership, and thus TrackID allocation ownership, happening
exclusively in DecodedStream for its output tracks, we also stop throwing
away and recreating the SourceMediaStream to which we feed data on seek.
This is one step closer to fixing bug 1172394 and spec compliance of
HTMLMediaElement.captureStream().
Differential Revision: https://phabricator.services.mozilla.com/D12273
--HG--
extra : moz-landing-system : lando
Nobody is mirroring to 'mLogicallySeeking', so we can just use Watchable instead.
Differential Revision: https://phabricator.services.mozilla.com/D9642
--HG--
extra : moz-landing-system : lando
Add some memory usage information to the Performance counters and make everything asynchronous.
Differential Revision: https://phabricator.services.mozilla.com/D7984
--HG--
extra : moz-landing-system : lando
Nobody is mirroring to 'mLogicallySeeking', so we can just use Watchable instead.
Differential Revision: https://phabricator.services.mozilla.com/D9642
--HG--
extra : moz-landing-system : lando
Nobody is mirroring to 'mLogicallySeeking', so we can just use Watchable instead.
Differential Revision: https://phabricator.services.mozilla.com/D9642
--HG--
extra : moz-landing-system : lando
