Due to the extreme diversity of video content like motion, camera panning, and length, the challenging part arises when working with video frames to attain depth estimation. Depth Crafter can make your life easier. It has been released by Tencent AI Lab,3ARC Lab, and The Hong Kong University of Science and Technology.
We have already experienced depth estimated generation using Controlnet models for images but this is for instant video estimation. The model works on a video depth estimation methodology with achieving great consistency. You can get more in-depth knowledge from their relevant research paper.
DepthCrafter: Generating Consistent Long Depth Sequences by Tencent 😻
— Stable Diffusion Tutorials (@SD_Tutorial) October 19, 2024
Now in comfyUI 😎
Original repohttps://t.co/n4iTX5HVPu
Setup in comfy https://t.co/zz0PELX6J0 pic.twitter.com/imLz4tNu7B
The most important point to note is this does not use any additional information like optical flow or camera-posing angles. Now you will be wondering, where you can apply this?
Well, this will be very powerful when you want to generate animated videos using AnimateDiff. There are multiple workflows that we have already explained Video-to-Video workflows like creating VFX with AnimateDiff using IC light, and animating objects with AnimateDiff for viral social media platforms.
Now let's see how to do the installation in ComfyUI.
Installation
1. You need to install ComfyUI first to get your work done.
2. Navigate to the "ComfyUI/custom_nodes" folder and open the command prompt by typing "cmd" on the folder address bar.
3. Clone the repository by typing the commands provided below:
git clone https://github.com/akatz-ai/ComfyUI-DepthCrafter-Nodes.git
4. Restart ComfyUI to take effect.
Workflow Explanation
1. Get the workflow file inside your "ComfyUI/custom_nodes/ComfyUI-DepthCrafter-Nodes/examples_workflow" folder. Just drag and drop it to ComfyUI.
2. Load your video into the "Load Video" node.
3. The Depth Crafter model node already loads the Depth Crafter model. No need to load the model.
4. Configure the relevant settings(maximum resolution, steps, CFG, etc.) from DepthCrafter node.
5. Get the output depth estimated video from the Video Combine node.
Conclusion
Whether you are creating anime videos or generating videos for your AI influencer, generating depth estimation has never been so easy.
Now, using Depth Crafter you can create consistent depth estimation for your required workflows.
