CEF audio

[deadbeef84]

No description provided.

[ronag]

Suggested change

	frame.set_audio_data(std::move(data));
	frame.audio_data() = std::move(data);

[niklaspandersson]

Obviously! Good catch 👍 I'm still not 100% used to the rvalue special functions. That just shows how long it's been since i actually coded c++ on a regular basis before this.

[deadbeef84]

@niklaspandersson this is waiting for a fix of the initial glitching samples

[Julusian]

Thanks for starting on this, this inspired me to take a look at this, and I have reworked this a bit in my own prototype implementation CasparCG#1517.
I have a couple of thoughts on the implementation here

---

I have a theory on the glitching audio, in some cases inside these bits of code https://github.com/nxtedition/casparcg/blob/cef-audio/src/modules/html/producer/html_producer.cpp#L236-L247 you will need to wrap the returned frame with core::draw_frame::still(...), so mute the audio when a frame is being repeated.

I encountered a similar thing, when pausing/resuming a low framerate video, fixed by adding those calls in https://github.com/CasparCG/server/blob/feat/cef-audio/src/modules/html/producer/html_producer.cpp#L263-L273

The code around here has diverged a bit by now, so may take some thought to translate across

---

I don't think this implementation is very efficient on memory copying.

By moving the mutable_frame to draw_frame to try_pop

casparcg/src/modules/html/producer/html_producer.cpp

Lines 491 to 492 in d655390

	result = (frames_.front().is_empty()) ? core::draw_frame::empty()
	: core::draw_frame(std::move(frames_.front().frame));

, that means that the copy from gl buffer to texture is being delayed until try_pop. This delays the copy to being done just before the mixer wants to composite with it, rather than 2-4 frames in advance.

Additionally, I count that for each frame CEF paints, you are copying into a gpu buffer (as before) and then copying from that gpu buffer into last_video_frame_.
Each time a frame is repeated with new audio, you are copying the cached buffer into a new gpu buffer.

I would suggest that instead of this, it would be more efficient to use the layering nature of draw_frame. so that each draw frame returned from the producer contains up to two other draw_frames, one which contains just audio, and one which contains just video. That means it will be kept at one memcpy per video frame produced by cef.