Experiment result metrics

class ablator.modules.metrics.main.Metrics(*args, batch_limit=30, memory_limit=100000000.0, evaluation_functions: dict[str, collections.abc.Callable] | None = None, moving_average_limit=3000, static_aux_metrics: dict[str, Any] | None = None, moving_aux_metrics: Iterable[str] | None = None)

Bases: object

Stores and manages predictions and calculates metrics given some custom evaluation functions. This class makes batch-updates as metrics are calculated while training/evaluating a model. It takes into account the memory limits, applies evaluation functions, and provides cached or online updates on the metrics.

We can access all the metrics from the Metrics object using its to_dict() method. Refer to Prototyping Models tutorial for more details.

to_dict()

Get all metrics, i.e moving auxiliary metrics, moving evaluation metrics, and static auxiliary metrics. Note that moving attributes will be an averaged value of all previous batches. Metrics are set to np.nan if it’s never updated before

Examples

>>> from ablator.modules.metrics.main import Metrics
>>> train_metrics = Metrics(
...     batch_limit=30,
...     memory_limit=None,
...     evaluation_functions={"mean": lambda preds: np.mean(preds)},
...     moving_average_limit=100,
...     static_aux_metrics={"lr": 0.75},
...     moving_aux_metrics={"loss"},
... )
>>> train_metrics.append_batch(preds=np.array([100]))
>>> train_metrics.evaluate(reset=False, update=True)
>>> train_metrics.to_dict()
{
    'train_mean': np.nan, 'train_loss': np.nan,
    'val_mean': 100.0, 'val_loss': np.nan,
    'lr': 0.75
}
>>> train_metrics.append_batch(preds=np.array([0] * 3))
>>> train_metrics.evaluate(reset=True, update=True)
>>> train_metrics.to_dict()
{
    'train_mean': np.nan, 'train_loss': np.nan,
    'val_mean': 62.5, 'val_loss': np.nan,
    'lr': 0.75
}