Zaggy1024

A big nerd. Writing lots of video-related code for Ladybird, and previously Firefox, along with other various OSS work.

[Zaggy1024 @Zaggy1024]

@PhilAndrew61181 @awesomekling @ladybirdbrowser To implement audio playback rates, we'll probably end up integrating a library that does the signal processing for us. It also won't be designed to skip video frames, they may just not be presented if the display's refresh rate is too low.

[Zaggy1024 @Zaggy1024]

@PhilAndrew61181 @awesomekling @ladybirdbrowser By AV speedup, do you mean the playback rate? I'm not currently planning on making FFmpeg take care of playback, this is only replacing our home-grown VP9 codec with one that calls through to libavcodec.

[Zaggy1024 @Zaggy1024]

@jrmoulton3 @awesomekling @ladybirdbrowser Not yet, the painting system in Ladybird needs a lot of work to get to that point.

In master at the moment, all the painting commands are built up from scratch when the video frame changes, so it's very far from efficient. Even modifying the paint command in place will be big.

[Zaggy1024 @Zaggy1024]

@CEOofFuggy @awesomekling @ladybirdbrowser (Credit where it's due, a lot of Serenity's VP9 decoder's bitstream parsing existed before I took it over, thanks to @TheFalseHonesty)

[Zaggy1024 @Zaggy1024]

@CEOofFuggy @awesomekling @ladybirdbrowser The YouTube player is probably a long way off, but eventually that will happen.

You must have tested VP9 before, which used the codec I wrote from scratch for Serenity, and was already capable of decoding locally-downloaded YouTube videos.

[Zaggy1024 @Zaggy1024]

@_philn_ @awesomekling @ladybirdbrowser It's also worth noting that integrating FFmpeg is very easy in the current codebase, I've had to make a very few small changes to get this working so far.

I've got what might amount to a total rewrite of presentation to get A/V sync working, so we'll see how this holds up.

[Zaggy1024 @Zaggy1024]

@_philn_ @awesomekling @ladybirdbrowser I'd be open to arguments in favor of it, but from what I've read about WebKit's usage, it seems like it presents a fair amount of problems when it comes to MSE. Having full control over the playback system may end up being beneficial, but that remains to be seen.

[Andreas Kling @awesomekling]

First screenshot of WIP ffmpeg integration in @ladybirdbrowser 🤓📽️🐞

Very exciting work by @Zaggy1024!

[Zaggy1024 @Zaggy1024]

@Ramborambowski @changelog @awesomekling @adamstac @jerodsanto The Matroska reader error was a red herring, since we try to create decoders for both audio and video regardless of whether it will be used in a video or audio element.

After fixing a few bugs in our MP3 decoder, though, we can play it back perfectly!

https://t.co/IsJob8DifT

[Zaggy1024 @Zaggy1024]

@_philn_ @awesomekling Yep, although I get the feeling that with HTMLVideoElement shaping up we're getting fairly close to someone writing an implementation of MSE too!

If no one beats me to it, I may try my hand at it, but for now I'm mostly trying to achieve speed improvements in the video decoder.

[Zaggy1024 @Zaggy1024]

@awakecoding @awesomekling @mdprice @TheFalseHonesty I contemplated writing an implementation in Rust since it's well-suited to the task. For Serenity, though, there was already initial VP9 work in C++ and there isn't currently any Rust integrated into the codebase, so C++ was the path of least resistance. :^)

[Zaggy1024 @Zaggy1024]

@awesomekling @mdprice @TheFalseHonesty The decoder is also being fuzzed by Google's OSS-Fuzz, so we should be able to catch plenty of issues when that is an actual focus.

[Zaggy1024 @Zaggy1024]

@awesomekling @mdprice @TheFalseHonesty Also, a lot of decoders are written in C, whereas the Serenity VP9 decoder is written in C++ where bounds checks can be integrated into the buffers used throughout the decoder. In theory this should prevent some footguns, although it obviously doesn't cover every security bug.

[João Santos 🍂 @JonnyPSR]

Harmony practice 😬

[Zaggy1024 @Zaggy1024]

@awesomekling Thanks! It's been a lot of slow debugging but it's incredibly satisfying to be at this point after all that work.

At this point I've got too many ideas for features and performance improvements to choose from, hoping I can keep improving it for a while.