Playback cursor does not follow notes accurately

[mudcube]

Currently, the playback cursor does not accurately follow the notes being played back. For example, the following image shows the cursor straddling two notes, which makes it difficult to know which note is currently playing:

In the editor mode, you can hover over notes, and the exact note is highlighted, this is really neat:

I wonder, how would one go about refactoring the playback cursor to highlight the exact notes being played?

Additionally, it would be really nifty for each part to have it's own color associated with it—so the bass part could have notes highlighted in blue, and the alto could have notes highlighted in green, and so-forth.

Btws, this project is looking fantastic! 🤯

[AaronDavidNewman]

Hi Mudcube. Thanks, glad you are enjoying Smoosic.

The playback rectangle is a compromise for performance reasons (I borrowed the idea from MuseScore). The biggest limitation on the size of score right now is playback - the audio buffers need to be created and deleted in real time. If there are too many voices it stutters.

You could probably fix the horizontal positioning by just adding a horizontal position to SoundParams in player.ts, and populate it in AudioPlayer.getNoteSoundData. The note.svg.logicalBox, if it exists, will have the value. If there are multiple voices, they may not be vertically aligned, but you could pick the first x value, or the lowest one, etc. Then you can pass this information to SuiTracker.musicCursor. If you're interested in producing a PR, that would be a welcome fix.

To highlight the individual notes/parts, I think that would be hard to do in a performant way. Part of the issue is that a note plays one time, but sounds for a while. So you'd have to set timers to create and clear the highlights in SVG. And you'd be modifying the DOM with all the music in it, which can be kind of large, and these timers would be competing with the timers that create the audio buffers.

One of the improvements I need to make is to break up the SVG DOM, maybe on a per-page basis. I think updating SVG in real time would be faster if the DOM is smaller (although there's no way to know how much before trying it).

Another improvement would be using audio worklets or animation frames for audio. But that would require smoosic always to be running through a server, and possibly over https, which makes local development more of a hassle, and raises the entry price for possible contributors.

[AaronDavidNewman]

hi @mudcube, this is fixed. Tracking works pretty well now. Let me know and if I don't hear back I'll close this issue.

I've thought a bit about what you said about parts in colors. Maybe it would be nice if you could pass your own function in that is used for playback?

[mudcube]

Hi Aaron, thank you so much for your thoughtful response :) My apologies for the late response!

Here's a few thoughts:

• What if the cursor was in it's own SVG, that was overlaid atop the sheet-music? This way, all the static elements would be in their own SVG, and any elements that need updating would be in their own SVG, which would allow the browser to optimize performance better. Or, the active elements could even be a Canvas, which has less overhead.

• Audio playback of sheet music on it's own is a huge project, and it's incredibly difficult to capture all the nuance of MIDI, while maintaining performance. Do you think it'd make sense for Smoosic to allow for plugging in other audio engines? Then, splitting off the audio rendering portion of Smoosic into its own repository. For example, WebAudioFont has great support for SF2 audio rendering, and appears to be able to handle many channels: https://surikov.github.io/webaudiofont/examples/midiplayer.html

• I love the idea of passing in my own function, that receives NoteOn/NoteOff commands, along with a reference to the SVG Element of the specific note being triggered!

• Is the tracking update in the live demo? It looks the same to me as before, but, I really need to setup a build myself, so I can use the latest live code, as have only been playing around with the live demo up until now! Here's what I'm seeing in the demo though:

[AaronDavidNewman]

love the idea of passing in my own function, that receives NoteOn/NoteOff commands, along with a reference to the SVG Element of the specific note being triggered!

Yeah, same lets go with that approach.

It is updated (subtly) just now. I still use the blue rectangle as before, but it now tracks with the music, if possible. I guess rectangles around the individual notes isn't really a performance issue anymore now that I have different pages with their own SVG. It's more that the rectangle scales better for larger scores, and for just playing back the music you wrote it is sufficient. In other words, just not something I want to put too much effort into :)

But if we allowed a user to specify a their own function, that would be more in keeping with other customization in Smoosic.

(feel free to not read all this...it got very long!)

What if the cursor was in it's own SVG, that was overlaid atop the sheet-music?

Good idea. Also for text editing and the selection boxes. Is that possible though? I'm trying to think how that would work with scrolling and opacity to have them exactly overlay, since each page is in an SVG already. Each overlay would have to be absolutely positioned and have opacity of 0, in which case you wouldn't see it... SVG is painters model so stuff you add to the end of the DOM already is overlaid on stuff that appears earlier. And the 'static' content is actually redrawn pretty frequently if it changes, so there's a lot of work keeping them in sync. I've tried 'shadow-DOM' solutions before, and it didn't improve performance very much and it was crazy complicated. We already have to manage 'SVG' coordinate space and client space for events and scrolling.

Do you think it'd make sense for Smoosic to allow for plugging in other audio engines?

The project you mentioned does basically the same thing as Smoosic - playing samples in the browser. So the performance would be about the same. And there's really no other way to do it in the current model, where everything runs in the browser in javascript and there is no server-side component. I can't even use worker threads without running into CORS issues when running locally, but I really want the technology stack to be just javascript in the browser, until I completely hit a wall.

It is actually possible to create/use your own samples right now by just replacing the audio links in the 'smoosic.html' but it's not well-documented yet.

Thanks for letting me soundboard this stuff off you.

[mudcube]

But if we allowed a user to specify a their own function, that would be more in keeping with other customization in Smoosic.

Love it!

Good idea. Also for text editing and the selection boxes. Is that possible though?

That's true—definitely possible, but would require tedious synchronization of onScroll with positioning of the overlay. On second thought, it seems like the multiple pages solution you've already come up with (splitting the pages into their own individual SVGs) probably addresses the performance concerns. TBH, everything seems to be running very smoothly to me! :)

The project you mentioned does basically the same thing as Smoosic - playing samples in the browser.

Ok, gotcha. I had not compared the source-code. It sounds like both implementations creates queues of many notes into the future in batches. I think the main difference between WebAudioFont and Smoosic is its usage of linearRampToValueAtTime (or another similar WebAudio feature), to fade in/out notes, which allowed it to provide a more smooth playback (I was confusing that with a performance issue).

Btws, do you happen to have a stress test sheet music that you like to use, to test performance issues?

[AaronDavidNewman]

do you happen to have a stress test sheet music that you like to use, to test performance issues?

the big band charts are the biggest scores I have in Smoosic right now, Bilongo in particular. These are exports from Sibelius and need some work in general, but they are good benchmarks especially for a full render.

[mudcube]

Thanks :) I'm noticing in _setModifierBoxes (which takes ~3s on Bilongo) a lot of time is spent with Sizzle queries, it looks like in that method you could replace with native dom, e.g. el.classList.add('grace-note') and document.querySelectorAll('.' + modifier.attrs.id). I wonder if that would help with that method call?

[AaronDavidNewman]

Thanks. that's true. In fact I think I have the element here so I don't need to look anything up, I can just use it to get the bounding box.

[AaronDavidNewman]

@mudcube that made a big difference! I was able to remove the jquery DOM searches in a few cases and just use the elements or the element ID. The post-render mapping time for large scores is much lower.

Another change we should be able to make is changing the zoom level. In theory, we shouldn't have to re-render at all there since the only change is the viewport in the SVGs. The relative svg coordinates don't change. So I will work on that next. For big scores, changing zoom is a pretty common thing to have to do.

[mudcube]

@AaronDavidNewman Nice, glad to hear that helped :)

The zoom update sounds like a major win, I imagine that'll make it nearly instantaneous after the initial render 🚀

[mudcube]

It seems like changing viewport (resizing the window) should also not need a re-render?

[mudcube]

Another thought comes to mind (inspecting the performance tab on chrome)... is it possible to not add the SVG to the screen, until all the sub-elements (notes, staves, ect) have been added to it? Sometimes this can improve the performance, as if it's on the screen, the browser is actively trying to keep it updated with all the new things being added to it, whereas, if it's not appended to the dom until all the changes have been made, it can batch all those actions.

[AaronDavidNewman]

True about the viewport. It's a little harder to handle since it comes through as an event, but in theory no different than changing the zoom.

I don't think keeping the SVG out of the DOM would do anything. If it did, systems further down on the page would take longer and we don't really see that.

It seems like there is some issue with the mouse pointer on scores with many pages. The rectangle for clicking seems to be drift higher for each page, vs the rectangle of the note. You can really notice it on Bilongo which has a lot of pages and a small zoom level.

[mudcube]

Using DocumentFragment does improve performance, as it prevents page reflows:
https://blog.hao.dev/web-api-appendchild-v-s-createdocumentfragment

As far as how much it would improve performance, it's hard to say! But, if it's easy to replace the root document with a fragment, and then append the fragment to the main document at the end of the page build, it might be worth a shot! It looks like it might be about 25% faster for those operations :)