Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(unit3): Add example of img2img and inpainting implementation #91

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

metamath1
Copy link

Title: feat(unit3): Add example of img2img and inpainting implementation

Body:

This pull request adds example code of a manually implemented denoising loop for img2img and inpainting to the stable_diffusion_introduction.ipynb file in unit3.

Motivation

I was able to understand diffusion models through the tutorial provided in this repository, and I am extremely grateful for that. It was fascinating to see that diffusion models can be applied to tasks like img2img and inpainting. However, it was difficult to grasp the inner workings just by using the pipeline. Therefore, I added a direct implementation to help users understand it more easily.

Detailed Explanation of Implementation

  1. While the added code references the call function within the StableDiffusionPipeline of the diffusers library, it has been intentionally simplified to focus on the core aspects of the denoising loop for easier understanding by beginners. Unlike the original code that implements the entire loop, non-essential parts have been omitted to concentrate on the noise removal process, minimizing the code's length.
  2. This modification has been added to the "A DIY Img2Img Loop" and "A DIY Inpainting Loop" sections respectively. The code within these sections is written concisely and clearly, with comments added to each step to aid understanding.
  3. Since runwayml/stable-diffusion-inpainting is no longer available, the model for inpainting was replaced with stabilityai/stable-diffusion-2-inpainting.

Request

Please review the code for accuracy, efficiency, and readability. In particular, I would appreciate feedback on the implementation of the denoising loop and the masking process for inpainting.

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@metamath1
Copy link
Author

@johnowhitaker
I noticed the workflow is awaiting approval. Could you kindly review and approve it when you have a moment? Thank you!

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@metamath1
Copy link
Author

@johnowhitaker
Upon reviewing the documentation preview, I noticed two issues with the images:

  1. An existing image (inpaint_w_border.png) was broken in the preview despite being present in the original file.
  2. A newly added image (inpaint_from_scratch.png) that I introduced in this PR was also not rendering properly.

To fix both issues, I converted the images to data URIs and committed the changes to the same branch. Could you please review the updates and approve the workflow again so the latest changes are reflected in the documentation preview?

Thank you for your time and support!

@metamath1
Copy link
Author

@johnowhitaker
I noticed that both the existing image (inpaint_w_border.jpg) and the newly added image (inpaint_from_scratch.png) were not rendering properly in the live

To address this issue, I initially tried using Base64 encoding, but it seems to have caused problems with the PR build.

To resolve the issue for now:
For the new image (inpaint_from_scratch.png), I used the Raw URL from my forked repository.
![inpainting from_scratch](https://raw.githubusercontent.com/metamath1/diffusion-models-class/unit3/implement_img2img_inpaint/unit3/inpaint_from_scratch.png)

For the existing image (inpaint_w_border.jpg), I updated the link to reference the Raw URL from the main repository.
![inpainting from_scratch](https://raw.githubusercontent.com/huggingface/diffusion-models-class/main/unit3/inpaint_w_border.jpg)

I understand that using external image URLs (e.g., images from my forked repository) may not be ideal for long-term maintenance. If this PR is merged, I can update the link referencing my forked repository as needed.

Thank you for your guidance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants