fedem.utils package
Submodules
fedem.utils.huggingface module
- fedem.utils.huggingface.get_client_details(hf_token: str | None = None) tuple[source]
Get the client details from the Hugging Face API.
- Parameters:
hf_token (str, optional) – The Hugging Face API token. Defaults to None.
- Returns:
The Hugging Face API client and the user details.
- Return type:
tuple | None
- fedem.utils.huggingface.verify_user_with_org(client_details: dict, org_id: str, access_level: list = ['contributor']) dict[source]
Verify if the user is part of the organization.
- Parameters:
client_details (dict) – The client details.
org_id (str) – The organization id.
- Returns:
The org details if the user is part of the organization, else None.
- Return type:
dict | None
Module contents
- fedem.utils.get_checkpoint_model(model_name)[source]
Get the checkpoint model based on the model name and organization ID.
- Parameters:
model_name (str) – Name of the model.
- Returns:
Model ID if found, False otherwise.
- Return type:
str | bool
- fedem.utils.load_data(data_path)[source]
Load dataset from a given path and split it into train and validation sets.
- Parameters:
data_path (str) – Path to the dataset.
- Returns:
Dictionary containing train and validation datasets.
- Return type:
DatasetDict
- fedem.utils.load_json(json_path)[source]
Load JSON data from a file.
- Parameters:
json_path (str) – Path to the JSON file.
- Returns:
Loaded JSON data.
- Return type:
dict
- fedem.utils.load_model(config)[source]
Load a model based on the provided configuration.
- Parameters:
config – Model configuration.
- Returns:
Loaded model.
- Return type:
- fedem.utils.load_model_pretrained(config)[source]
Load a pre-trained model based on the provided configuration.
- Parameters:
config – Model configuration.
- Returns:
Loaded pre-trained model.
- Return type:
- fedem.utils.load_model_with_LoRA(model, target_modules, local_path)[source]
Load a model with LoRA (Low-Rank Adaptation) applied.
- Parameters:
model – Base model to apply LoRA to.
target_modules – List of target modules.
local_path (str) – Local path to save the adapter.
- Returns:
Model with LoRA applied.
- Return type:
- fedem.utils.load_tokenizer(path)[source]
Load tokenizer from a given path.
- Parameters:
path (str) – Path to the tokenizer.
- Returns:
Loaded tokenizer.
- Return type:
AutoTokenizer