Nodes Browser
ComfyDeploy: How ComfyUI_Aniportrait works in ComfyUI?
What is ComfyUI_Aniportrait?
implementation of [a/AniPortrait](https://github.com/Zejun-Yang/AniPortrait) generating of videos, includes self driven, face reenacment and audio driven with a reference image
How to install it in ComfyDeploy?
Head over to the machine page
- Click on the "Create a new machine" button
- Select the
Editbuild steps - Add a new step -> Custom Node
- Search for
ComfyUI_Aniportraitand select it - Close the build step dialig and then click on the "Save" button to rebuild the machine
Updates:
① Implement the frame_interpolation to speed up generation
② Modify the current code and support chain with the VHS nodes, i just found that comfyUI IMAGE type requires the torch float32 datatype, and AniPortrait heavily used numpy of image unit8 datatype,so i just changed my mind from my own image/video upload and generation nodes to the prevelance SOTA VHS image/video upload and video combined nodes,it WYSIWYG and inteactive well and instantly render the result
- ✅ [2024/04/09] raw video to pose video with reference image(aka self-driven)
- ✅ [2024/04/09] audio driven
- ✅ [2024/04/09] face reenacment
- ✅ [2024/04/22] implement audio2pose model and pre-trained weight for audio2video,the face reenacment and audio2video workflow has been modified, currently inference up to a maximum length of 10 seconds has been supported,you can experiment with the length hyperparameter.
U can contact me thr twitter
Weixin：GalaticKing
audio driven combined with reference image and reference video
<table class="center"> <tr> <td width=100% style="border: none"> <video controls autoplay loop src="https://github.com/frankchieng/ComfyUI_Aniportrait/assets/130369523/1ab454ef-13ca-4262-9206-383470a8408d" muted="false"></video> </td> </tr> </table>raw video to pose video with reference image
face reenacment
<table class="center"> <tr> <td width=100% style="border: none"> <video controls autoplay loop src="https://github.com/frankchieng/ComfyUI_Aniportrait/assets/130369523/0a9e46fb-33b8-4787-b8ca-b70301240752" muted="false"></video> </td> </tr> </table>This is unofficial implementation of AniPortrait in ComfyUI custom_node,cuz i have routine jobs,so i will update this project when i have time
you should run
git clone https://github.com/frankchieng/ComfyUI_Aniportrait.git
then run
pip install -r requirements.txt
download the pretrained models
download the weights:
denoising_unet.pth reference_unet.pth pose_guider.pth motion_module.pth audio2mesh.pt audio2pose.pt film_net_fp16.pt
./pretrained_model/
|-- image_encoder
| |-- config.json
| `-- pytorch_model.bin
|-- sd-vae-ft-mse
| |-- config.json
| |-- diffusion_pytorch_model.bin
| `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5
| |-- feature_extractor
| | `-- preprocessor_config.json
| |-- model_index.json
| |-- unet
| | |-- config.json
| | `-- diffusion_pytorch_model.bin
| `-- v1-inference.yaml
|-- wav2vec2-base-960h
| |-- config.json
| |-- feature_extractor_config.json
| |-- preprocessor_config.json
| |-- pytorch_model.bin
| |-- README.md
| |-- special_tokens_map.json
| |-- tokenizer_config.json
| `-- vocab.json
|-- audio2mesh.pt
|-- audio2pose.pt
|-- denoising_unet.pth
|-- motion_module.pth
|-- pose_guider.pth
|-- reference_unet.pth
|-- film_net_fp16.pt
Tips : The intermediate audio file will be generated and deleted,the raw video to pose video with audio and pose2video mp4 file will be located in the output directory of ComfyUI the original uploaded mp4 video requires square size like 512x512, otherwise the result will be weird