ablator.analysis.plot package#

Submodules#

ablator.analysis.plot.cat_plot module#

class ablator.analysis.plot.cat_plot.Categorical(*args: Any, **kwargs: Any)[source]#

Bases: Plot

This class is for preparing the results that are associated with each categorical attribute to be studied (e.g., grouping metric results with each of the attributes). Its constructor takes in as input positional arguments or keyword arguments from the base class Plot. Possible arguments are listed in the Parameters section. The Attributes section lists its own attributes as well as those that are inherited.

Parameters:
metricpd.Series

The ablation study metric values to plot.

attributespd.Series

The ablation study attributes values to plot.

metric_obj_fnOptim

The metric optimization direction.

y_axisstr, optional

The y-axis label (metric name), by default None.

x_axisstr, optional

The x-axis label (attribute name), by default None.

x_tickslist[str], optional

The x-axis ticks, by default None.

axAxes, optional

The axes to plot on, by default None.

Attributes:
metricpd.Series

The ablation study metric values to plot (with null value removed).

attributespd.Series

The ablation study attributes values to plot (with null metric value removed).

metric_obj_fnOptim

The metric optimization direction.

y_axisstr

The y-axis label (metric name).

x_axisstr

The x-axis label (attribute name).

x_tickslist[str]

The x-axis ticks.

figureFigure

The figure to plot on. If None, a new figure of size (4,4) will be created.

axAxes

The axes to plot on. If None, a new axis will be created as the first subplot in the first cell and first column of a 1x1 grid.

DATA_TYPEstr

The attribute data type. In this case, it is "categorical".

attribute_metric_mapdict[str, pd.Series]

A dictionary mapping attribute values to metric values.

DATA_TYPE: str = 'categorical'#
class ablator.analysis.plot.cat_plot.ViolinPlot(*args: Any, **kwargs: Any)[source]#

Bases: Categorical

Class for constructing violinplots. Its constructor takes in as input positional arguments or keyword arguments from the base class Categorical. Possible arguments are listed in the Parameters section. The Attributes section lists its own attributes as well as those that are inherited.

Parameters:
metricpd.Series

The ablation study metric values to plot.

attributespd.Series

The ablation study attributes values to plot.

metric_obj_fnOptim

The metric optimization direction.

y_axisstr, optional

The y-axis label (metric name), by default None.

x_axisstr, optional

The x-axis label (attribute name), by default None.

x_tickslist[str], optional

The x-axis ticks, by default None.

axAxes, optional

The axes to plot on, by default None.

Attributes:
metricpd.Series

The ablation study metric values to plot (with null value removed).

attributespd.Series

The ablation study attributes values to plot (with null metric value removed).

metric_obj_fnOptim

The metric optimization direction.

y_axisstr

The y-axis label (metric name).

x_axisstr

The x-axis label (attribute name).

x_tickslist[str]

The x-axis ticks.

figureFigure

The figure to plot on. If None, a new figure of size (4,4) will be created.

axAxes

The axes to plot on. If None, a new axis will be created as the first subplot in the first cell and first column of a 1x1 grid.

DATA_TYPEstr

The attribute data type. In this case, it is “categorical”.

attribute_metric_mapdict[str, pd.Series]

A dictionary mapping attribute values to metric values.

figsize: tuple

A tuple representing the size of the figure in terms of axes (x, y).

ablator.analysis.plot.main module#

class ablator.analysis.plot.main.PlotAnalysis(results: DataFrame | Results, categorical_attributes: list[str] | None = None, numerical_attributes: list[str] | None = None, optim_metrics: dict[str, ablator.config.proto.Optim] | None = None, save_dir: str | None = None, cache: bool = False)[source]#

Bases: Analysis

Class for plotting experiment results. You can use this class and Results class to visualize the relationship between the result metrics and any hyperparameter you run ablation study on. This valuable insight offers an intuitive understanding of how these parameters may influence your model’s performance.

Plots supported are linear plots for numerical data and violin plots for categorical data.

Parameters:
resultspd.DataFrame | Results

The result dataframe.

categorical_attributeslist[str] | None

The list of all the categorical hyperparameter names

numerical_attributeslist[str] | None

The list of all the numerical hyperparameter names

optim_metricsdict[str, Optim] | None

A dictionary mapping metric names to optimization directions.

save_dirstr | None

The directory to save analysis results to.

cachebool

Whether to cache results.

Examples

  • Data frame to be used:

>>> df = pd.DataFrame({'val_accuracy': np.random.uniform(0.8,0.9,10),
...       'train_config.optimizer_config.arguments.lr': np.random.uniform(0.001, 0.1,10),
...       "index": range(10),
...       "path": range(10)})

  • Creating dictionaries that map the configuration parameters [categorical + numerical] to custom labels for plots:

>>> numerical_name_remap = {
...     "train_config.optimizer_config.arguments.lr": "Learning Rate",
... }
... categorical_name_remap = {}
... attribute_name_remap = {**categorical_name_remap, **numerical_name_remap}

  • Initalize the PlotAnalysis and plot the figures:

>>> analysis = PlotAnalysis(
...     df,
...     save_dir="./plots",
...     cache=True,
...     optim_metrics={"val_accuracy": Optim.max},
...     numerical_attributes=list(numerical_name_remap.keys()),
...     categorical_attributes=list(categorical_name_remap.keys()),
... )
>>> analysis.make_figures(
...    metric_name_remap={
...        "val_accuracy": "Validation Accuracy",
...    },
...    attribute_name_remap= attribute_name_remap
... )

The directory "plots" contains all the plots of the HPO experiments

make_figures(metric_name_remap: dict[str, str] | None = None, attribute_name_remap: dict[str, str] | None = None, save_dir: str | Path | None = None, **plt_kwargs: Any)[source]#

Generate violin plots for categorical values and linear plots for numerical values. Plots are created as metrics vs. attributes. Additional keyword arguments to pass to the plot method are ax and append:

Parameters:
metric_name_remapdict[str, str] | None

mappings for config’s metrics keys to user defined names, by default None.

attribute_name_remapdict[str, str] | None

mappings for config’s searchspace names to user defined names for attributes, by default None.

save_dirstr | Path | None

optional directory of where to save the results, when unspecified, it expects one set during class initialization, by default None.

axAxes | None

A matplotlib.axes.Axes object representing the axis to plot on),

appendbool

A boolean indicating whether to append plots to an existing axes object) and extra arguments for creating the plots.

make_linearplot(attribute_names: list[str], metrics: list[str], save_dir: Path | str, **plt_kwargs) → dict[source]#

To make linear plots for the given attribute names (data type: numerical) v.s. the metrics and save the plots to the save_dir directory.

Parameters:
attribute_nameslist[str]

list of attributes to plot against the metrics.

metricslist[str]

list of metrics to plot against the given attributes.

save_dirty.Union[Path, str]

directory to save results.

**plt_kwargs

Additional keyword arguments to pass to the plot method.

Returns:
dict
make_violinplot(attribute_names: list[str], metrics: list[str], save_dir: Path | str, **plt_kwargs)[source]#

Make violin plots for the given attribute names (data type: discrete) v.s. the metrics and save the plots to the save_dir directory.

Parameters:
attribute_nameslist[str]

list of attributes to plot against the metrics.

metricslist[str]

list of metrics to plot against the given attributes.

save_dirty.Union[Path, str]

directory to save results.

**plt_kwargs

Additional keyword arguments to pass to the plot.

ablator.analysis.plot.num_plot module#

class ablator.analysis.plot.num_plot.LinearPlot(metric: Series, attributes: Series, metric_obj_fn: Optim | str, y_axis: str | None = None, x_axis: str | None = None, x_ticks: list[str] | None = None, ax: Axes | None = None)[source]#

Bases: Numerical

Class for generating linear plots

Parameters:
ax: Axes

axes object of linear plot.

figure: Figure

Its corresponding figure object.

class ablator.analysis.plot.num_plot.Numerical(metric: Series, attributes: Series, metric_obj_fn: Optim | str, y_axis: str | None = None, x_axis: str | None = None, x_ticks: list[str] | None = None, ax: Axes | None = None)[source]#

Bases: Plot

Base class for numerical plots

Attributes:
DATA_TYPE: str

data_type for numerical plots.

DATA_TYPE: str = 'numerical'#

ablator.analysis.plot.utils module#

ablator.analysis.plot.utils.parse_name_remap(defaults: list[str] | None = None, name_map: dict[str, str] | None = None) → dict[str, str][source]#

Returns a dictionary mapping input attribute names to output attribute names, with optional remapping based on name_map.

Parameters:
defaultslist[str] | None

The default attribute names to use as keys in the output dictionary. If None, the output dictionary will be based on name_map only.

name_mapdict[str, str] | None

A dictionary mapping input attribute names to output attribute names. If None, the output dictionary will be based on defaults only.

Returns:
dict[str, str]

A dictionary mapping input attribute names to output attribute names.

Raises:
NotImplementedError

If defaults and name_map are both None.

Examples

>>> defaults = ["attr1", "attr2", "attr3"]
>>> name_map = {"attr2": "new_attr2", "attr4": "attr4_renamed"}
>>> name_remap = parse_name_remap(defaults, name_map)
>>> assert name_remap == {"attr1": "attr1", "attr2": "new_attr2", "attr3": "attr3"}
>>> name_remap = parse_name_remap(defaults)
>>> assert name_remap == {"attr1": "attr1", "attr2": "attr2", "attr3": "attr3"}
>>> name_remap = parse_name_remap(name_map=name_map)
>>> assert name_remap == {"attr2": "new_attr2", "attr4": "attr4_renamed"}

Module contents#

class ablator.analysis.plot.Plot(metric: Series, attributes: Series, metric_obj_fn: Optim | str, y_axis: str | None = None, x_axis: str | None = None, x_ticks: list[str] | None = None, ax: Axes | None = None)[source]#

Bases: ABC

A base class for parsing experiment results and plotting them with pandas and matplotlib.

Parameters:
metricpd.Series

The ablation study metric values to plot.

attributespd.Series

The ablation study attributes values to plot.

metric_obj_fnOptim | str

The metric optimization direction.

y_axisstr | None

The y-axis label (metric name), by default None.

x_axisstr | None

The x-axis label (attribute name), by default None.

x_tickslist[str] | None

The x-axis ticks, by default None.

axAxes | None

The axes to plot on, by default None.

Attributes:
metricpd.Series

The ablation study metric values to plot (with null value removed).

attributespd.Series

The ablation study attributes values to plot (with null metric value removed).

metrics_obj_fnOptim

The metric optimization direction.

y_axisstr

The y-axis label (metric name).

x_axisstr

The x-axis label (attribute name).

x_tickslist[str]

The x-axis ticks.

figureFigure

The figure to plot on. If None, new figure of size (4,4) will be created.

axAxes

The axes to plot on. If None, a new axis will be created as the first subplot in the first cell and first column of a 1x1 grid.

make(**kwargs: Any) → tuple[matplotlib.figure.Figure, matplotlib.axes._axes.Axes][source]#