Imagine you are a business owner or professional creator managing a project based on Stable Diffusion, and due to some government policies any AI based startup instantly remove their models from the Hugging Face repository. Your business will collapse.
The AI community is so rapidly growing and all those government interventions impacted a lot in the past days.
A similar incident recently happened. There was the dubious situation in the Reddit community when RunwayML deleted their officially released Stable Diffusion 1.5 model from their Hugging Face account.
After doing quite in-depth research, we found a quick solution to deal with these types of uncertain situations. To do this, you can remotely clone any public Hugging Face repository to your personal repository. All the steps are described in an easy flow that a non-techy savvy can also grasp.
Here, we are using Google Colab so that you don't need to manage huge AI models locally and spend all your network bandwidth. This process can be followed in any cloud server as well.
Table of Contents (Jump to section):
Step-By-Step Process:
This process assumes you have access to both the source and destination repositories on Hugging Face and they are not listed as the private one.
1: Set Up Your Environment in Google Colab:
First, you need to open google colab and create a new notebook. Use this command to install the hugging face library to do the initial setup.
!pip install huggingface_hub
2. Creating Hugging Face Access Token
Check the box to grant relevant permission to your access token. There are multiple options you can choose from- read , write, delete, etc. Add a relevant name for your access token. Then, just click "Create" button available at the bottom.
Next, you will be prompted with your new access token. Click "Copy" to save it. Make sure you save the access token safely as it will not be accessed in the future. To use the Hugging Face, you need to create a new one again.
3. Login hugging face using Access token:
from huggingface_hub import login
login(token="<<your-access-token>>")
Here, replace <<your-access-token>> with your new access token.
After the code execution, a popup message will appear. You have to grant access by hitting the "Grant access" option.
After that, you will get a confirmation message "Login successful".
4. Cloning/Downloading the whole hugging face repo:
!git clone <<public-repository-link>>
This is useful if you want to download the entire repository. Make sure it's in the public domain and not set to private which means it can be accessed by anyone. For illustration, we are using the public "pt-sk/stable-diffusion-1.5" Hugging face repository.
To get the public Hugging face id, just copy the id from its repository like we shown above.
Coning entire repository is a cumbersome task and required enough disk space. Make sure you have enough disk space on Google Colab account.
You can use these function to download only specific file:
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="put-public-repo-id", filename="name-of-file", local_dir="file-location-path")
5. Create a new hugging face repository:
from huggingface_hub import create_repo
create_repo("<<your-HF-profile-name>>/<<your-HF-repo-name>>")
You can see we have the new created repository as "stable-diffusion-v1.5" in our Hugging face account.
6. Uploading the whole repository to the personal repository:
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder( folder_path="<<path-of-folder-location>>", repo_id="your-new-created-repo-id", repo_type="model" )
To copy the relevant path of any file select using right click option. Then, put the path and repository id beside the relevant parameters as we show above.
Parameters:
folder_path="<<path-of-folder-location>>"
repo_id="your-new-created-repo-id"
repo_type= "your-repository-type"
Now, you can see the whole repository has been cloned to our new Hugging Face repository.
Upload a single file to your personal account:
from huggingface_hub import HfApi
api = HfApi()
api.upload_file(path_or_fileobj="<<file-path>>", path_in_repo="README.md", repo_id="<<your-new-created-repo-id>>", repo_type="model")
Important points:
