{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Configuration basics"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/fostiropoulos/ablator/blob/v0.0.1-mp/docs/source/notebooks/Configuration-Basics.ipynb)\n",
"\n",
"Ablator embraces a versatile configuration system that configures every facet of the training process of machine learning models, covering not only the model architecture but also the training environment.\n",
"\n",
"Think of this configuration system as a blueprint for crafting experiments. By leveraging this system, Ablator orchestrates the creation and setup of experiments, seamlessly integrating the necessary configurations. Furthermore, Ablator offers the flexibility to dynamically construct a hierarchical configuration through composition.\n",
"\n",
"You have the choice to override settings either by using Python named arguments, by `yaml` configuration files, or by using dictionaries. In this tutorial, we will explain all configuration-related concepts in Ablator. We will also demonstrate all necessary steps to configure an experiment in Ablator (named arguments method). Delve into [this section](#alternatives-to-constructing-configuration-objects) of this tutorial to gain insights into implementing the latter two approaches."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration categories\n",
"\n",
"In ablator, configurations are divided into different categories, these include:\n",
"\n",
"- [Model configuration](#model-configuration) (or model config).\n",
"\n",
"- [Training configuration](#training-configuration) (or training config/ train config).\n",
"\n",
"- [Optimizer and Scheduler configuration](#optimizer-and-scheduler-configurations) (or optimizer config and scheduler config).\n",
"\n",
"- [Running configurations](#running-configurations) (or run configuration/ running config/ run config), either for training a single prototype model or training multiple models in parallel.\n",
"\n",
"These configuration classes will be used together to configure an experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Model Configuration"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Model configuration is required when creating the run configuration (`RunConfig.model_config` and `ParallelConfig.model_config`). This configuration class is used to define [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) specific to the model of interest. By default, it does not have any attribute, instead, we typically inherit from this and add custom attributes for each of the hyperparameters specific to our models. Later, you will use this configuration to construct the model.\n",
"\n",
"There are 2 steps that are required after defining a model config class for your model:\n",
"\n",
"- Pass the model config to the main model's constructor so you can construct the model using the attributes that's defined in the config.\n",
"\n",
"- Create a custom [running configuration class](#running-configurations) and update `model_config` class type to the newly created model config class.\n",
"\n",
"\n",
"\n",
"Note\n",
"\n",
"Ablator requires the model’s forward function to return two objects: one dictionary of model’s batched output (e.g. labels, predictions, logits, probabilities, etc.), and the other is the loss value. Notice that these values must be tensors. You also have the choice to return None for either of the values, depending on the use case.\n",
"\n",
"When writing forward method of the main model, ablator requires that you return 2 outputs in the following order:\n",
"\n",
"- dictionary of model’s batched output (e.g. labels, predictions, logits, probabilities, etc.), e.g. `{\"y_pred\": , \"y_true\": }`.\n",
"\n",
"- Loss value. Here the loss value will be considered an auxiliary metric that will be recorded for later analysis (e.g. tracking loss with `Tensorboard`).\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition, you can create a search space over these parameters for your ablation experiment. A sample use case for this is when you want to test different values for model size, number of layers, activation functions, etc. You can do this by creating a search space via `SearchSpace` class for the hyperparameters that you have defined in the model config. Refer to the [Ablation experiment](./HPO-tutorial.ipynb) tutorial for more details."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This configuration class defines the training setting (e.g., batch size, number of epochs, the optimizer to use, etc.). Two important attributes to metion are `optimizer_config` and `scheduler_config`. As the names suggest, they configure the optimizer and scheduler to be used in the training process."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"| Parameter | Usage |\n",
"|-------------------|-----------------------------------------------------------------------|\n",
"| dataset _(`str`)_ | dataset name. maybe used in custom dataset loader functions. |\n",
"| batch_size _(`int`)_ | batch size. |\n",
"| epochs _(`int`)_ | number of epochs to train. |\n",
"| optimizer_config _(`OptimizerConfig`)_ | optimizer configuration.|\n",
"| scheduler_config _(`Optional[SchedulerConfig]`)_ | scheduler configuration.|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Training configuration is required when creating the run configuration (`RunConfig.train_config` or `ParallelConfig.train_config`)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Optimizer and Scheduler Configurations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, Abaltor takes care of creating the optimizer (and optionally the scheduler) for training models. Thus, you also need to configure them so ablator knows which optimizer to pick.\n",
"\n",
"`OptimizerConfig` is used to configure the optimizer for the training process. Currently, we support `SGD` optimizer, `Adam` optimizer, and `AdamW` optimizer.\n",
"\n",
"`SchedulerConfig`, on the other hand, can be used to configure the learning rate scheduler for the training process. Currently, we support `StepLR` scheduler, `OneCycleLR` scheduler, and `ReduceLROnPlateau` scheduler."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Both of these config classes have similar arguments:\n",
"\n",
"| Parameter | Usage |\n",
"|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| name _(`str`)_ | The type of the optimizer or scheduler, this can be any in ``['sgd', 'adam', 'adamw']`` for optimizers and
in ``['none', 'step', 'cycle', 'plateau']`` for schedulers. |\n",
"| arguments _(`OptimizerArgs`)_ | The arguments for the scheduler or optimizer, specific to a certain type of scheduler or scheduler. For opimizer, **MUST** include an item for learning rate, e.g. `{\"lr\": 0.1}` |\n",
"\n",
"The table below shows possible arguments can be defined for each type of optimzer:\n",
"\n",
"| Optimizer type | Arguments |\n",
"|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| sgd | `weight_decay` _(defaults to 0.0)_: Weight decay rate
`momentum` _(defaults to 0.0)_: Momentum factor
|\n",
"| adam | `betas` _(defaults to (0.9, 0.999))_: Coefficients for computing running averages of gradient and its square.
`weight_decay` _(defaults to 0.0)_: Weight decay rate.
|\n",
"| adamw | `betas` _(defaults to (0.9, 0.999))_: Coefficients for computing running averages of gradient and its square.
`eps` _(defaults to 1e-8)_: Term added to the denominator to improve numerical stability.
`weight_decay` _(defaults to 0.01)_: Weight decay rate.
|\n",
"\n",
"The table below shows possible arguments can be defined for each type of scheduler:\n",
"\n",
"\n",
"| Scheduler type | Arguments |\n",
"|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| cycle | `max_lr` : Upper learning rate boundaries in the cycle.
`total_steps` : The total number of steps to run the scheduler in a cycle.
`step_when` _(defaults to `\"train\"`)_: The step type at which the `scheduler.step()` should be invoked: ``'train'``, ``'val'``, or ``'epoch'``. |\n",
"| plataeu | `patience` _(defaults to 10)_: Number of epochs with no improvement after which learning rate will be reduced.
`min_lr` _(defaults to 1e-5)_: A lower bound on the learning rate.
`mode` _(defaults to \"min\")_: One of ``'min'``, ``'max'``, or ``'auto'``, which defines the direction of optimization, so as to adjust the learning rate
accordingly, i.e when a certain metric ceases improving.
`factor` _(defaults to 0.0)_: Factor by which the learning rate will be reduced: ``new_lr = lr * factor``.
`threshold` _(defaults to 1e-4)_: Threshold for measuring the new optimum, to only focus on significant changes.
`verbose` _(defaults to False)_: If ``True``, prints a message to ``stdout`` for each update.
`step_when` _(defaults to \"val\")_: The step type at which the scheduler should be invoked: ``'train'``, ``'val'``, or ``'epoch'``.
|\n",
"| step | `step_size` _(defaults to 1)_: Period of learning rate decay.
`gamma` _(defaults to 0.99)_: Multiplicative factor of learning rate decay.99.
`step_when` _(defaults to \"epoch\")_: The step type at which the scheduler should be invoked: ``'train'``, ``'val'``, or ``'epoch'``.
|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Running Configurations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Running configurations define the environment of an experiment (experiment main directory, number of checkpoints to maintain, hardware device to use, etc.). There are 2 types of running configurations:\n",
"\n",
"- `RunConfig` for prototype experiments\n",
"- `ParallelConfig` for ablation experiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### `RunConfig` for prototype experiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The table below summarizes the parameters:\n",
"\n",
"| Parameter | Usage |\n",
"|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| experiment_dir _(`Stateless[Optional[str]]`, defaults to None)_ | location to store experiment artifacts. |\n",
"| random_seed _(`Optional[int]`, defaults to None)_ | random seed. |\n",
"| train_config _(`TrainConfig`)_ | training configuration. (check ``TrainConfig`` for more details) |\n",
"| model_config _(`ModelConfig`)_ | model configuration. (check ``ModelConfig`` for more details) |\n",
"| keep_n_checkpoints _(`Stateless[int]`, defaults to 3)_ | number of latest checkpoints to keep. |\n",
"| tensorboard _(`Stateless[bool]`, defaults to True)_ | whether to use tensorboardLogger. |\n",
"| amp _(`Stateless[bool]`, defaults to True)_ | whether to use automatic mixed precision when running on gpu. |\n",
"| device _(`Stateless[str]`, defaults to \"cuda\")_ | device to run on. |\n",
"| verbose _(`Stateless[Literal[\"console\", \"progress\", \"silent\"]]`, defaults to \"console\")_ | verbosity level. |\n",
"| eval_subsample _(`Stateless[float]`, defaults to 1)_ | fraction of the dataset to use for evaluation. |\n",
"| metrics_n_batches _(`Stateless[int]`, defaults to 32)_ | max number of batches stored in every tag(train, eval, test) for evaluation. |\n",
"| metrics_mb_limit _(`Stateless[int]`, defaults to 10_000)_ | max number of megabytes stored in every tag(train, eval, test) for evaluation. |\n",
"| early_stopping_iter _(`Stateless[Optional[int]]`, defaults to None)_ | The maximum allowed difference between the current iteration and the last
iteration with the best metric before applying early stopping. Early stopping
will be triggered if the difference ``(current_itr-best_itr)`` exceeds ``early_stopping_iter``.
If set to ``None``, early stopping will not be applied. |\n",
"| eval_epoch _(`Stateless[float]`, defaults to 1)_ | The epoch interval between two evaluations. |\n",
"| log_epoch _(`Stateless[float]`, defaults to 1)_ | The epoch interval between two logging. |\n",
"| init_chkpt _(`Stateless[Optional[str]]`, defaults to None)_ | path to a checkpoint to initialize the model with. |\n",
"| warm_up_epochs _(`Stateless[float]`, defaults to 1)_ | number of epochs marked as warm up epochs. |\n",
"| divergence_factor _(`Stateless[Optional[float]]`, defaults to 10)_ | if ``cur_loss > best_loss > divergence_factor``, the model is considered
to have diverged. |\n",
"| optim_metrics: _(`Stateless[Optional[Dict[Optim]]]`)_ | The optimization metric to use for meta-training procedures, such as for model saving and lr scheduling e.g. ``{\"val_loss\": \"min\"}`` |\n",
"| optim_metric_name: _(`Stateless[Optional[str]]`)_ | The name of the metric to be optimized. |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### `ParallelConfig` for ablation experiments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`ParallelConfig` is a subclass of `RunConfig`. Therefore, it has all attributes `RunConfig` has. Additionally, it introduces other attributes to configure the parallel experiment:\n",
"\n",
"| Parameters | Usage |\n",
"|----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|\n",
"| total_trials: _(`Optional[int]`)_ | total number of trials. |\n",
"| concurrent_trials: _(`Stateless[Optional[int]]`)_ | number of trials to run concurrently. |\n",
"| search_space: _(`Dict[SearchSpace]`)_ | search space for hyperparameter search, eg. ``{\"train_config.optimizer_config.arguments.lr\": SearchSpace(value_range=[0, 10], value_type=\"int\"),}`` |\n",
"| gpu_mb_per_experiment: _(`Stateless[Optional[int]]`)_ | CUDA memory requirement per experimental trial in MB. e.g. a value of 100 is equivalent to 100MB |\n",
"| search_algo: _(`Stateless[SearchAlgo]`, defaults to SearchAlgo.random)_ | type of search algorithm, SearchAlgo.random for ablation studies, SearchAlgo.tpe for HPO. |\n",
"| ignore_invalid_params: _(`Stateless[bool]`, defaults to False)_ | whether to ignore invalid parameters when sampling or raise an error. |\n",
"| remote_config: _(`Stateless[Optional[RemoteConfig]]`, defaults to None)_ | remote storage configuration. |\n",
"\n",
"`search_space` is used to define a set of continuous or categorical/ discrete values for a certain hyperparameter. Refer to [Search Space basics](./Search-space-tutorial.ipynb) to learn more about how to use it."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configure your experiments"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's combine everything to configure your experiment!\n",
"\n",
"\n",
"\n",
"Note\n",
"\n",
"For predefined config classes in ablator, the tables above summarize the list of attributes for each config class that you can include when creating config objects, you can also inspect these in the modules documentation [Configuration module](../config.rst), specifically in their attribute sections.\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" import ablator\n",
"except:\n",
" !pip install ablator\n",
" print(\"Stopping RUNTIME! Please run again\") # This script automatically restart runtime (if ablator is not found and installing is needed) so changes are applied\n",
" import os\n",
"\n",
" os.kill(os.getpid(), 9)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In most cases, also as a good practice, we first configure our model (or not configure it at all if you're not running ablation study on the model architecture).\n",
"\n",
"In this example, we create a configuration class `MyModelConfig` for a simple 1-layer neural network model with the following hyperparameters: input size (to be inferred); hidden layer dimension, activation function, and dropout rate (all of which are stateful - discussed below). This configuration then will be used to construct the neural network model `MyCustomModel`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ablator import RunConfig, ModelConfig, Derived, configclass\n",
"\n",
"import torch.nn as nn\n",
"import torch\n",
"\n",
"@configclass\n",
"class MyModelConfig(ModelConfig):\n",
" inp_size: Derived[int]\n",
" hidden_dim: int\n",
" activation: str\n",
" dropout: float\n",
"\n",
"# Construct the model using the configuration\n",
"class MyCustomModel(nn.Module):\n",
" def __init__(self, config: MyModelConfig) -> None:\n",
" super().__init__()\n",
" self.linear = nn.Linear(config.inp_size, config.hidden_dim)\n",
" self.dropout = nn.Dropout(config.dropout)\n",
" if config.activation == \"relu\":\n",
" self.activate = nn.ReLU()\n",
" elif config.activation == \"elu\":\n",
" self.activate = nn.ELU()\n",
" self.criterion = nn.CrossEntropyLoss()\n",
"\n",
" def forward(self, x: torch.Tensor, labels: None):\n",
" out = self.linear(x)\n",
" out = self.dropout(out)\n",
" out = self.activate(out)\n",
" \n",
" loss = self.criterion(out, labels)\n",
"\n",
" return {\"preds\": out, \"labels\": labels}, loss\n",
"\n",
"my_model_config = MyModelConfig(hidden_dim=100, activation=\"relu\", dropout=0.3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how we're returning a dictionary for model's predictions and labels, and loss value in the `forward` method."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We next create a training configuration, which requires an optimizer config and an optional scheduler config. Here we create `optimizer_config` that for an `SGD` optimizer, and `scheduler_config` that configs a `OneCycleLR` scheduler, and then use them in `train_config`:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from ablator import OptimizerConfig, SchedulerConfig\n",
"from ablator import TrainConfig\n",
"\n",
"optimizer_config = OptimizerConfig(name=\"sgd\", arguments={\"lr\": 0.1})\n",
"scheduler_config = SchedulerConfig(name=\"cycle\", arguments={\"max_lr\": 0.5, \"total_steps\": 50})\n",
"\n",
"train_config = TrainConfig(\n",
" dataset=\"test\",\n",
" batch_size=128,\n",
" epochs=2,\n",
" optimizer_config=optimizer_config,\n",
" scheduler_config=scheduler_config,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The last step is to create a `run_config` object. This object combines `train_config` and `my_model_config`, along with runtime settings like verbosity and device. However, we first need to redefine the run config class to update its `model_config` attribute from `ModelConfig` (by default) to `MyModelConfig`:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"CustomRunConfig(model_config={'inp_size': None, 'hidden_dim': 100, 'activation': 'relu', 'dropout': 0.3}, experiment_dir=None, random_seed=None, train_config={'dataset': 'test', 'batch_size': 128, 'epochs': 2, 'optimizer_config': {'name': 'sgd', 'arguments': {'weight_decay': 0.0, 'momentum': 0.0, 'lr': 0.1}}, 'scheduler_config': {'name': 'cycle', 'arguments': {'max_lr': 0.5, 'total_steps': 50, 'step_when': 'train'}}}, keep_n_checkpoints=3, tensorboard=True, amp=True, device='cpu', verbose='silent', eval_subsample=1.0, metrics_n_batches=32, metrics_mb_limit=10000, early_stopping_iter=None, eval_epoch=1.0, log_epoch=1.0, init_chkpt=None, warm_up_epochs=1.0, divergence_factor=10.0, optim_metrics=None, optim_metric_name=None)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@configclass\n",
"class CustomRunConfig(RunConfig):\n",
" model_config: MyModelConfig\n",
"\n",
"run_config = CustomRunConfig(\n",
" train_config=train_config,\n",
" model_config=my_model_config,\n",
" verbose=\"silent\",\n",
" device=\"cpu\",\n",
")\n",
"run_config"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"That's it, we have finished configuring our experiment! With this, we are half-way to launching an ablation experiment. Refer to [Prototyping models](./Prototyping-models.ipynb) and [Ablation experiment](./HPO-tutorial.ipynb) tutorials for the next steps after configuration to launch the experiment.\n",
"\n",
"\n",
"\n",
"Note\n",
"\n",
"All configuration classes (including custom ones that you may create) must inherit from `ConfigBase` and decorated with `@configclass` decorator, you can see this in `MyModelConfig` and `CustomRunConfig` classes in the example above.\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Ablator custom data types for stateful experiment design"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One key feature of Ablator is the ability to run stateful experiments. To do this, we created three special types: **Stateless**, **Stateful**, and **Derived**. These are custom annotations to define configuration attributes to which the experiment state is agnostic, aka does not have any impact on the experiment state (which can be Complete, Running, Pending, Pruned, etc. Read more about experiment state from our [paper](https://iordanis.me/data/ablator.pdf)).\n",
"\n",
"- **Stateless** attributes can take different values between trials or experiments. For example, learning rate should be stateless, as we can train models with different learning rates. Note that if you're declaring a variable to be Stateless, it must be assigned an initial value before launching the experiment.\n",
"\n",
"- **Stateful** attributes, opposite to **Stateless**, must have the same value between different experiments. For example, a binary classification model should always have output size of 2. Stateful variables, defined as a primitive datatype (no annotating needed), must be assigned with values before launching the experiment.\n",
"\n",
"- **Derived** attributes are **Stateful** and are un-decided at the start of the experiment. Their values are determined by internal experiment processes that can depend on other experimental attributes, e.g model input size that depends on the dataset.\n",
"\n",
"Later when annotating attributes, you can wrap these keywords around their data type. For example, `inp_size: Derived[int]` means that `inp_size` is a derived attribute of type `int`, and similarly for Stateless. For Stateful however, any data types that are not annotated with Derived nor Stateless are considered Stateful.\n",
"\n",
"We also defined other structural data types such as List, Dict, Enum, etc. You can find more information about all custom data types (including the three above types) in the [Data types configuration](../ablator.config.types.rst) module documentation.\n",
"\n",
"\n",
"\n",
"Note\n",
"\n",
"- The reason for creating these annotations is that for stateful experiment design, the configuration should be unambiguous at the initialization state. And the use of these annotations assures the unambiguity of the configuration.\n",
"- If you are interested to learn more about stateful experiment design, see our paper: [ABLATOR: Robust Horizontal-Scaling of Machine Learning Ablation Experiments](https://iordanis.me/data/ablator.pdf)\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Alternatives to constructing configuration objects"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"There are three methods to configure an experiment: named arguments, file-based, and dictionary-based. All previous code snippets are examples of the named-arguments method. Now let's look at how file based method and dictionary based method work."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### File-based"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"File based configuration is a way for you to create simple configuration files. You can use `.load(path/to/yaml/file)` method to create configuration with values provided in the config file.\n",
"\n",
"To write these config files, simply follow `yaml` syntax. Make sure that **the attributes and their hierarchy match with those in the config classes** (for both default config classes from ablator, or custom ones like `MyModelConfig`). The following example shows what a config yaml file looks like. We will name it `config.yaml`:\n",
"\n",
"```yaml\n",
"experiment_dir: \"/tmp/dir\"\n",
"train_config:\n",
" dataset: test\n",
" batch_size: 128\n",
" epochs: 2\n",
" optimizer_config:\n",
" name: sgd\n",
" arguments:\n",
" lr: 0.1\n",
" scheduler_config:\n",
" name: cycle\n",
" arguments:\n",
" max_lr: 0.5\n",
" total_steps: 50\n",
"model_config:\n",
" inp_size: 50\n",
" hidden_dim: 100\n",
" activation: \"relu\"\n",
" dropout: 0.15\n",
"verbose: \"silent\"\n",
"device: \"cpu\"\n",
"```\n",
"\n",
"Now in your code, load these values to create the config object:\n",
"\n",
"```python\n",
"config = CustomRunConfig.load(\"path/to/yaml/file\")\n",
"```\n",
"\n",
"Note that since we created a custom running configuration class `CustomRunConfig` that is tied to the custom model config in the previous sections, we used `CustomRunConfig.load(\"path/to/yaml/file\")` to load configuration from file. Otherwise, if you're not creating any subclasses, simply run `RunConfig.load(\"path\")` or `ParallelConfig.load(\"path\")`."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dictionary based\n",
"\n",
"Another alternative is similar to the file-based method, but it's defining configurations in a dictionary instead of a yaml file, and then the dictionary will be passed (as keyword arguments) to the configuration at initialization"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"configuration = {\n",
" \"experiment_dir\": \"/tmp/dir\",\n",
" \"train_config\": {\n",
" \"dataset\": \"test\",\n",
" \"batch_size\": 128,\n",
" \"epochs\": 2,\n",
" \"optimizer_config\":{\n",
" \"name\": \"sgd\",\n",
" \"arguments\": {\n",
" \"lr\": 0.1\n",
" }\n",
" },\n",
" \"scheduler_config\":{\n",
" \"name\": \"cycle\",\n",
" \"arguments\":{\n",
" \"max_lr\": 0.5,\n",
" \"total_steps\": 50\n",
" }\n",
" }\n",
" },\n",
" \"model_config\": {\n",
" \"inp_size\": 50,\n",
" \"hidden_dim\": 100,\n",
" \"activation\": \"relu\",\n",
" \"dropout\": 0.15\n",
" },\n",
" \"verbose\": \"silent\",\n",
" \"device\": \"cpu\"\n",
"}\n",
"\n",
"config = CustomRunConfig(\n",
" **configuration\n",
")\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you've learned how to configure experiments, you can start creating your own prototype. In the next chapter, we will learn how to write a prototype model, define necessary configurations and model interfaces and launch the experiment. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}