The #WebRTC #WorkingGroup published the "MediaStreamTrack Insertable Media Processing using Streams" #timetogiveinput and 🧵👇
w3.org/TR/2022/WD-mediacaptu…
This tweet is unavailable
Feb 10, 2022 · 5:03 PM UTC
1
This #JavaScript #API allows to process raw video, either before it gets encoded or after it gets decoded, e.g. to add #effects such as background blur to real-time #video.
1
2
It uses the notion of video frames developed in #WebCodecs to expose the bytes that a video processing program would operate on.
Today's selection on WebCodecs is about the existing and emerging Web #APIs for media production track, with suggestions for #developers and open questions from #browser vendors' perspectives: w3.org/2021/03/media-product…
1
An earlier version of the #API is available in #Chromium chromestatus.com/feature/549… with a demo at webrtc.github.io/samples/src…
1
#MachineLearning models are often used in this context, so special efforts are being put in figuring out what is needed to integrate #WebNN in this pipeline with maximal #performance.
github.com/webmachinelearnin…
1
This also ties to the conversation about minimizing memory copies (in general, and for media processing in particular).
2) "Memory copies & zero-copy operations on the Web" - to identify possible architectural updates to the #WebPlatform and reduce unneeded memory copies - youtube.com/lZW9zvdqnyw (see also slides at w3.org/2020/Talks/TPAC/uncon…)
1
Insertable Streams uses the #WHATWG @streamsstandard as a way to process video frames sequentially, while allowing potentially for optimized memory and processing semantics.
See this proposal to amend Streams to make them fit the need of raw video: github.com/whatwg/streams/bl…
1
So, still quite a bit of work to make this all fit together, but this is an important step to enable #RealTime video processing capabilities on the Web Platform.
Head to the #GitHub repo if you want to contribute!
github.com/w3c/mediacapture-…
1
1