Improve ComfyUI integration

[jbilcke-hf]

Context

ComfyUI is all the rage right now, and I would like to use it as one of the supported backends for Clapper.

Currently, Clapper only has ultra-basic support for ComfyUI cloud providers, such as Comfy.icu (see this screenshot, the ugly text field at the very bottom):

Goal

At minimum, it should be able to use an existing integration running locally or in the cloud

But what I have in mind is more something like this:

• a list of available workflows straight from from the global ComfyUI community, such as OpenArt.ai
• a way to easily pick them, eg. click "apply", "use", "install", or some as simple as a drag & drop
• a display of the workflow within the UI (eg. preview of the comfy graph)

Limitations / things to checks

I'm not a ComfyUI developer (or user), so it's gonna be a fun ticket!

How to implement

For now I've only added some basic code that doesn't really work (search for "comfy" and "ComfyIcu" - it's related to comfy.icu - in the codebase)

Acceptance criteria

Being able to run any ComfyUI workflow, for : image, video, and why no also sound, voice and music

[lalalune]

/boss $250

[boss-bounty]

Total Bounty: $1050

Bounties

Bounties paid automatically on close:

• @lalalune: $250
• @lalalune: $500
• @nha1012: $300

[sendn0des]

Flexibility to pull from openart.ai would present some wonk. There's too many workflows and many are meh - but it could lead to an issues with frontend design later attempting to accommodate.

I'll bring over some workflows I've considered or have that would play well and can design toolkit around them.

[lalalune]

/boss $500

[jbilcke-hf]

I'm making some changes to the codebase to prepare and facilitate work for this ticket.

These changes are not specific to ComfyUI, so not in the scope of this ticket, and not meant to claim the bounty:

1. I've added the concept of workflow to Clapper and the project file format (OpenClap):
   (import { ClapWorkflow } from "@aitube/clap"))

2. A .clap project (import { ClapProject } from "aitube/clap") can now embed an array of generic workflows (clap.workflows = [] // type ClapWorkflow[])

3. A timeline segment (import { ClapSegment } from "aitube/clap") can now optionally reference a workflow id (segment.workflowId = "..."). Right now this will do nothing, but the idea is that the workflow should be called to generate the corresponding segment's output value. (I advise to use this with parcimony, to avoid using +100 different workflows within your project (you could, but I'm not sure it would be wise). Instead I encourage to use a default workflow for image, another default workflow for video, another for sound, music etc..)

4. A workflow can have an array of parameters, and default values

5. there is now a empty WorkflowEditor, to be worked on later. The idea is to add a UI to be able to visualize, edit and set the default parameters of workflows (again it's not about ComfyUI but any AI workflow engine like Transformers.js, MediaPipe, Visual Blocks, Glif etc)

[sendn0des]

love the idea of specific workflows. will avoid the comfy ecosystem compatibility and deprecated node issues.

glif could be useful here since the workflows are managed internally and maintained

for later, comfydeploy looks interesting for rapid iteration. it has a nice desktop workflow -> deploy -> frontend setup

[jbilcke-hf]

I have made some improvements to the code to improve ComfyUI support (a real ComfyUI server that you run locally or on your own remote server)

I believe we are close to having a basic (static) demo soon, hopefully this week

[nha1012]

/boss $300