Experiment results reader

class ablator.analysis.results.Results(config: type[ablator.config.mp.ParallelConfig] | ParallelConfig, experiment_dir: str | Path, cache: bool = False, use_ray: bool = False)

Bases: object

Class for processing experiment results. You can use this class to read the results in an experiment output directory. This can be used in combination with PlotAnalysis to show the correlation between hyperparameters and metrics. Refer to Interpreting Results tutorial for more details on plotting and interpreting experiment results.

Parameters:

configtype[ParallelConfig] | ParallelConfig

The configuration class used

experiment_dirstr | Path

The path to the experiment directory.

cachebool

Whether to cache the results, by default False.

use_raybool

Whether to use ray for parallel processing, by default False.

Raises:

FileNotFoundError

If the experiment directory doesn’t exists.

ValueError

If RunConfig is provided instead of ParallelConfig.

Examples

Suppose you have an experiment output directory stored at <path to experiment output defined in config experiment_dir>. You can read the results from the directory as follows:

>>> directory_path = Path('<path to experiment output defined in config experiment_dir>')
>>> results = Results(config=ParallelConfig, experiment_dir=directory_path, use_ray=True)
>>> df = results.read_results(config_type=ParallelConfig, experiment_dir=directory_path)

Pass df to PlotAnalysis to create an analysis object for plotting the correlation between the hyperparameters and the metrics and save the plots to an output directory. For example, the following template generates plots for each of the numerical and categorical hyperparameters and saves them to ./plots directory. Here “Validation Accuracy” is the name of the main metric.

>>> analysis = PlotAnalysis(
...     df,
...     save_dir="./plots",
...     cache=True,
...     optim_metrics={"val_accuracy": Optim.max},
...     numerical_attributes=<numerical name remap keys names>,
...     categorical_attributes=<categorical name remap keys names>,
... )
>>> analysis.make_figures(
...     metric_name_remap={
...         "val_accuracy": "Validation Accuracy",
...     },
...     attribute_name_remap= attribute_name_remap
... )

Attributes:

experiment_dirPath

The path to the experiment directory.

configtype[ParallelConfig]

The configuration class used.

metric_mapdict[str, Optim]

A dictionary mapping metric names to their optimization direction.

data: pd.DataFrame

The processed results of the experiment. Refer to read_results for more details.

config_attrs: list[str]

The list of all the optimizable hyperparameter names

search_space: dict[str, ty.Any]

All the search space of the experiment.

numerical_attributes: list[str]

The list of all the numerical hyperparameter names

categorical_attributes: list[str]

The list of all the categorical hyperparameter names.

classmethod read_results(config_type: type[ablator.config.main.ConfigBase], experiment_dir: Path | str, num_cpus: float | None = None) → DataFrame

Read all experiment results from the experiment directory (with ray if specified when initializing Result).

Parameters:

config_typetype[ConfigBase]

The configuration class.

experiment_dirPath | str

The experiment directory.

num_cpusfloat | None

Number of CPUs to use for ray processing, by default None.

Returns:

pd.DataFrame

A data frame of all the results from all experiments in experiment_dir.

Raises:

RuntimeError

If no results are present in the experiment directory.

Examples

>>> results.read_results(config_type = ParallelConfig, experiment_dir = "/tmp/results/experiment_8925_9991/")
train_loss      val_loss        best_iteration  best_loss       current_epoch   current_iteration       epochs
13.3658738                      0                       inf                 1               100             5
2.277102967     0.277085876     100                 0.277085876         2                   200             5
2.277154112     0.27619998      200                 0.27619998          3                   300             5
2.276529543     0.286987235     200                 0.27619998          4                   400             5
2.279828385     0.274052692     400                 0.274052692         5                   500             5
11.91869608                     0                       inf                 1               100             5