Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Here's an example to transfer timbre in browser [Docs] #482

Open
mepc36 opened this issue Mar 15, 2023 · 0 comments
Open

Here's an example to transfer timbre in browser [Docs] #482

mepc36 opened this issue Mar 15, 2023 · 0 comments

Comments

@mepc36
Copy link

mepc36 commented Mar 15, 2023

Pulled my hair out over this one, so leaving it here for anyone else who needs it:

                <IonButton onClick={async () => {
                  const Magenta = (await import('@magenta/music'))
                  const { DDSP, SPICE } = Magenta
                  const spice = new SPICE()
                  await spice.initialize()

                  // const audioUrl = 'https://s3-us-west-2.amazonaws.com/s.cdpn.io/123941/Yodel_Sound_Effect.mp3'; // yodel

                  const context = new AudioContext();

                  window.fetch(audioUrl)
                    .then(async response => {
                      const arrayBuffer = await response.arrayBuffer()
                      return arrayBuffer
                    })
                    .then(async arrayBuffer => {
                      const audioBuffer = await context.decodeAudioData(arrayBuffer)
                      return audioBuffer
                    })
                    .then(async audioBuffer => {
                      const audioFeatures = await spice.getAudioFeatures(audioBuffer)
                      const checkpointUrl = 'https://storage.googleapis.com/magentadata/js/checkpoints/ddsp/violin'
                      const ddsp = new DDSP(checkpointUrl)
                      await ddsp.initialize()
                      const synthesizedBuffer = await ddsp.synthesize(audioFeatures)

                      // Copied from ~/node_modules/@magenta/music/esm/ddsp/buffer_utils.js, which is
                      // called by the addReverb method in node_modules/@magenta/music/esm/ddsp/add_reverb.js,
                      // which is called by the .synthesize method in node_modules/@magenta/music/esm/ddsp/model.js
                      const arrayBufferToAudioBuffer = (audioCtx, arrayBuffer, sampleRate) => {
                        const newBuffer = audioCtx.createBuffer(1, arrayBuffer.length, sampleRate);
                        newBuffer.copyToChannel(arrayBuffer, 0);
                        return newBuffer;
                      };
                      console.log('synthesizedBuffer:', synthesizedBuffer)

                      const synthesizedAudioBuffer = arrayBufferToAudioBuffer(context, synthesizedBuffer, 48000)

                      function play(myBuffer) {
                        const source = context.createBufferSource();
                        source.buffer = myBuffer;
                        source.connect(context.destination);
                        source.start();
                      }

                      play(synthesizedAudioBuffer)
                    });
                }}>
                  <IonIcon icon={volumeHigh} />
                </IonButton>

Checkpoint info (for violin, flute, etc.) can be found here:

https://github.com/magenta/magenta-js/tree/master/music/checkpoints#table

In my experience, the checkpoint downloader script didn't really work. It also appears there are 2 generations of DDSP models: one that powers this site here...

https://magenta.github.io/magenta-js/music/demos/ddsp_tone_transfer.html

...and then one that powers the Google Colab they link to on the front page of the DDSP README:

https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/demos/timbre_transfer.ipynb

I've only been able to access the gen 1 checkpoints so far. Haven't really tried with gen 2, but my plan for that one is to run the Google Colab, and manually download the checkpoint files from disk on the left, and then host them at S3. That will also need to host the .pkl and .gin files too, AFAIK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant