On the fly transcoding

[FluorescentHallucinogen]

Is it possible to use ffmpeg.js for realtime (on-the-fly) transcoding and playing of media stream (e.g. online radio or online TV) in the browser (without needing to download the entire media file before it can start playing)?

[Kagami]

What type of stream do you mean? In browser you have same-origin enforcement and basically can do only HTTP and WebSocket requests. In case of HTTP same-origin media stream it might be possible, though you'll need to patch ffmpeg in order to use Emscripten HTTP calls, see here.

In case of TCP, UDP, streams without CORS headers, etc. you will need some proxy and to wrap stream data into HTTP/WebSocket in order to read that in the browser. See discussion here.

You may also try Virtual XHR Backed FS: create lazy file for HTTP stream on the JS side and open it in ffmpeg as regular file. Though I'm not sure how good that would work.

[Kagami]

I think this was answered, feel free to ask additional questions. Transcoding might be possible on the fly (via patching or XHR FS) in some cases (e.g. same-origin HTTP), but I don't think there is a sane way to play intermediate results in main process. There is SharedArrayBuffer but it's experimental and it's a bad idea to start playing a file while ffmpeg transcodes it.

[ockulkarni]

Hey can you please tell how to patch or use xhr fs for transcoding on the fly. i want to encode file uploaded by my user from mp4 to .264 format for later using it for mp4box.js for segmenting it which is also going to be done on the fly . Any help will be appreciated thankyou!!

[mysterybear]

@Kagami Hello, a use case I'm interested in:

Raspberry Pi's raspivid to pipe H.264 across a websocat, transcoding live in the browser with ffmpeg.js. Is this reasonable/possible currently?

Edit: these streams would be continuous/infinite, the practical use case is surveillance camera footage viewing in the browser.

[SlugFiller]

In case someone finds this issue in the feature, I'd like to point out that it is possible to do this using Asyncify. However, I had to go through quite a bit of trouble while doing it myself, so I'd like to leave some tips for people who might want to do it.

First thing's first, TL;DR, here's a Dockerfile which will build an "async" ffmpeg.wasm. You will need to edit or create your own ffmpeg.js to use it, as the default would not do anything async.

FROM trzeci/emscripten-upstream

RUN git clone --depth 1 https://git.ffmpeg.org/ffmpeg.git ffmpeg && cd ffmpeg && emconfigure ./configure --disable-x86asm --disable-inline-asm --disable-doc --disable-stripping --disable-network --disable-pthreads --disable-sdl2 --disable-ffplay --disable-ffprobe \
 --nm="llvm-nm -g" --ar=emar --ranlib=emranlib --cc=emcc --extra-cflags="-s USE_PTHREADS=0 -s ASYNCIFY=1 -s ASYNCIFY_IMPORTS=[env.__syscall3,env.__syscall4,wasi_unstable.fd_write,wasi_unstable.fd_read,env.nanosleep]" --extra-ldflags="-s USE_PTHREADS=0 -s ASYNCIFY=1 -s ASYNCIFY_IMPORTS=[__syscall3,__syscall4,fd_write,fd_read,nanosleep]" --cxx=em++ --objcc=emcc --dep-cc=emcc && emmake make -j4 && \
 cp ffmpeg ffmpeg.bc && emcc -O3 ffmpeg.bc -o ffmpeg.js -s TOTAL_MEMORY=33554432 -s STANDALONE_WASM=1 -s WASM=1 -s EXIT_RUNTIME=1 -s ALLOW_MEMORY_GROWTH=1 -s ASYNCIFY=1 -s "ASYNCIFY_IMPORTS=[env.__syscall3,env.__syscall4,wasi_unstable.fd_write,wasi_unstable.fd_read,env.nanosleep]" && cd .. && cp ffmpeg/ffmpeg.wasm . && cp ffmpeg/ffmpeg.js . && rm -rf ffmpeg

CMD bash

You can use it like this:

docker build --rm -t ffmpegjs .
docker run --rm --name ffmpegjs -dt ffmpegjs
docker cp ffmpegjs:/src/ffmpeg.js .
docker cp ffmpegjs:/src/ffmpeg.wasm .
docker stop ffmpegjs

The important parts here:

• Using emscripten-upstream, to be able to use the WASM backend which supports Asyncify, since the fastcomp backend no longer does.
• Specifying ASYNCIFY for ALL files. Initially, I tried just for the final link phase, and it didn't quite work.
• Using STANDALONE_WASM so we can get exports with sane names.
• The functions we Asyncify are fd_write for writing to stderr, __syscall3 for reading from stdin, and __syscall4 for writing to stdout. I also added fd_read for completeness, even though I've never seen it being called. nanosleep has a default implementation that uses a busy spin loop, needlessly taking 100% CPU. While it's very rarely called during transcoding, I decided to reimplement it using nanotimer instead.
• The functions to Asyncify must have the module names specified. That was quite annoying to debug.
• I disable any I/O besides stdin/stdout, because it's easier for me to use ffmpeg only as a stream transcoder, than to reimplement all the necessary socket or filesystem functions with async support. All the I/O heavy lifting ffmpeg would normally do (including ICY support, see below), can be done in the main thread, and then streamed pre-digested into ffmpeg.
• While the JS can't be used as-is, it does work as a good base, as it implements necessary functions you don't really care about, like args_sizes_get and clock_gettime. It also gives a ready library for using Asyncify, which might not be perfect, but good enough if you're looking to save time.

After this, the JS needs to be created. As stated, beautifying the existing JS helps a lot. But it still leaves a lot of work left. Here's some landmines I stepped on:

• The Emscripten JS runtime is built be alone in the namespace, as it uses a lot of globals. This could probably be solved by adding -s MODULARIZE=1. But since I wouldn't want it blocking the main thread anyway, I just had it run in a worker. This, however, creates other problems.
• Since the default runtime assumes it is running alone, it simply opens the WASM file, and compiles it while instantiating. However, ffmpeg is pretty big, which means a 20-30 second compile time. This is solved by creating a WebAssembly.Module and passing it to the worker using postMessage. Much faster, but you have to rewrite the call to run within the JS, and also remove all the part where it tries to load the WASM itself.
• For the actual communication, I tried to pass Buffers over postMessage, but the result was bizarrely slow. Turns out that creating a new ArrayBuffer, copying the data into it, then passing it in the transfer list, is orders of magnitude faster. I would have assumed that's what postMessage does internally for non-transferred buffers, but looks like a no.
• While it's possible to send argv to a worker, I preferred just sending them in a run message, since I was already sending one to pass the precompiled WASM.
• console.log does not work very well out of a worker. Specifically, it only works when it reaches the event loop. As in, when it unwinds. When needing real time logs, it's better to use postMessage, and do the logging on the main thread.
• Even though there's a ready library for it, understanding how Asyncify unwind/rewind works helps. It could, for instance, help understand why the async function is called a second time at the end of the rewind process. The binaryen blog post about Asyncify is super helpful.
• If for any reason ffmpeg seems to run unusually slow, chances are the issue is with the I/O. Once everything was fixed on the main thread side, the ffmpeg side ran super smooth.
• Copying between Buffer and Int8Array is not as intuitive as I had initially expected. ffmpeg failing to read the data could be a symptom of doing it wrong.
• The exports field of a WASM instance is not a writable object. I found that out while trying to patch _start to detect a program end (it requires checking that Asyncify.state === Asyncify.State.Normal, to tell it apart from an unwind). Object.assign fixes that.
• Slightly not directly related to ffmpeg, but I also learned about an HTTP variant called ICY that is often used for streaming media, and why it causes node's default http library to crash. Good thing node-icy exists, and is transparent for regular HTTP/S. I probably would not have even noticed if I let ffmpeg do socket I/O, since it already supports it internally.

Hopefully this will help any people in the future to not be stuck on the same things I was stuck on.

[deepesh-agarwal]

@SlugFiller It would be nice if you could publish the async version and provide examples in a fork for others to use and improve upon.

[SlugFiller]

@deepesh-agarwal It's not really a fork. It's just the above Dockerfile, edited JS worker, and the calling app. The JS worker is somewhat of a hackjob made in a rush, which could certainly be made better by someone actually spending time on it from scratch. And the calling app uses a very my-style app-specific API, which isn't likely to make sense to a third party. If I was bothering to make something worth showing, I'd have at least subclassed node's Stream. And if I had the time, I'd reimplement the worker with 0 Emscripten code lines. What I ended up with is mostly the product of time constraints.

A person making those latter two from scratch using my above tips could probably get much better results. Someone going the extra mile might even asyncify other methods, like enabling net access.

[Kagami]

Thanks @SlugFiller, that's really interesting!

[Banou26]

Did anyone manage to reproduce/make a stream-able ffmpeg version based on @SlugFiller instructions/tips ?
I'm currently in need of something like this but i don't have any knowledge of WASM so it might be quite big challenge for me.

[dinedal]

@SlugFiller - trying to reproduce your effort and I have to say that even the code you're afraid about posting would help speed things up significantly, seeing as it's 10 months past now and a lot has changed.

[SlugFiller]

@dinedal - You're right that a lot has changed, namely in that the system that was using the code has been mostly phased out of use, due to underperforming. It was a lightweight SIP server that would stream audio from HTTP (or compatible) to the caller. The problem is that the RTP stream timing wasn't stable enough, and the stream would sound, while recognizable, quite distorted. Anyway, since the code is no longer in use, I figure it wouldn't harm to let others have a look, and maybe you can figure out what was wrong with the RTP timing.

I've temporarily put the code here:
https://ufile.io/1k3w1cn0

Grab it while it's hot. It's temporary hosting, which is probably a good thing, because pre-compiled ffmpeg is probably a violation of the license. This code will automatically CnD itself in 30 days. Compiling it yourself can be done using the above Dockerfile anyway.

[dinedal]

Grabbing now, thanks @SlugFiller !

[davedoesdev]

I have on-the-fly transcoding using asyncify working in #166