{ "cells": [ { "cell_type": "markdown", "id": "d8f3a7e7", "metadata": {}, "source": [ "# Introduction to Ablator" ] }, { "cell_type": "markdown", "id": "2258d905", "metadata": {}, "source": [ "Welcome to the first chapter, where we're going to dive deep into the backstory of Ablator. We'll take a closer look at two important topics: Ablation Studies and Hyperparameter Optimization and learn why Ablator plays a crucial role in these areas. We'll also get to know some important modules that Ablator offers and take a peek behind the scenes to see what happens when Ablator kicks into action. " ] }, { "cell_type": "markdown", "id": "56cc5f72", "metadata": {}, "source": [ "## Understanding Ablation Studies and Hyperparameter Optimization\n", "\n", "### Ablation Studies\n", "\n", "In the context of Machine learning (ML) research, an [ablation study](https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence)) is an experimental analysis used to understand the impact of different components of a Machine Learning model. This involves removing specific parts of a neural network architecture or changing different aspects of the training process to examine their contributions to the model's performance.\n", "\n", "In fact, the process of removing or modifying components of a model corresponds to the process of choosing different values of **hyperparameters**, which are parameters that define the model architecture or training process. For example, the number of layers in a neural network, the number of neurons in each layer, the learning rate, the batch size, the number of epochs, etc. are all hyperparameters.\n", "\n", "By selectively removing or modifying components/parameters, aka varying the hyperparameters combination (which we also refer to as configuration), researchers observe how the changes affect the system's output, performance, or behavior. Typically, hyperparameters are randomly chosen from a set of possible values called search space, which can either be discrete or continuous.\n", "\n", "### Hyperparameter Optimization\n", "\n", "Given the above definition of hyperparameters, we know that they are not models' parameters which are learned from the training process, but rather set by the user before training begins. Different combinations of hyperparameters can significantly impact the performance and convergence of a machine learning model.\n", "\n", "[Hyperparameter Optimization (HPO)](https://en.wikipedia.org/wiki/Hyperparameter_optimization) involves systematically searching for the best set of hyperparameters that lead to optimal model performance. The objective is to find the hyperparameter combination that helps the model reach the best metrics (e.g minimum loss value, or maximum accuracy) among all possible training attempts. This is different than ablation study, where combinations of hyperparameters are chosen in a way (most of the time random) that lets us understand the impact of different components of a machine learning model.\n", "\n", "Similarly to ablation studies, users will also define a search space for each hyperparameter, which is the base for our \"searching\" of the best hyperparameters." ] }, { "cell_type": "markdown", "id": "4c9c29d9", "metadata": {}, "source": [ "## Ablator's Role: Enhancing Ablation Studies and Hyperparameter Optimization\n", "\n", "As machine learning models grow in complexity, the number of components that need to be ablated also increases. This consequently expands the search space of possible configurations, requiring an efficient approach to horizontally scale multiple parallel experimental trials. Ablator is a tool that aids in the horizontal scaling of experimental trials.\n", "\n", "Instead of manually configuring and conducting multiple experiments with various hyperparameter settings, Ablator automates this process. It initializes experiments based on different hyperparameter configurations, tracks the state of each experiment, and provides experiment persistence. \n", "\n", "Ablator employs a random search algorithm when conducting ablation studies, and a greedy search strategy known as Tree-structured Parzen Estimators (TPE) for HPO. TPE approach aims to efficiently optimize the hyperparameters by iteratively selecting values that maximize performance, leading to improved overall results. These searching paradigms terminates when the user-defined total number of trials is reached.\n", "\n", "A few advantages of using Ablator are:\n", "\n", "- It is a tool that simplifies the process of prototyping and training machine learning models. It streamlines model experimentation and evaluation.\n", "\n", "- It offers a flexible configuration system.\n", "\n", "- It facilitates result interpretation through visualization." ] }, { "cell_type": "markdown", "id": "ea7d5a28", "metadata": {}, "source": [ "## Inside Ablator: Exploring its Modules and How They Work Together\n", "\n", "This section introduces the core modules within Ablator. We'll cover the Configuration Module, Training Module, Experiment Result Metrics Module, and Analysis Module. These modules collectively contribute to the framework's seamless experiment management and insightful analysis capabilities. \n", "\n", "### [Configuration Module](../config.rst)\n", "\n", "In Ablator, the configuration system serves as a foundation for crafting experiments. It enables researchers to set up the fine details. These settings are neatly grouped into various categories for easy organization:\n", "\n", "- Model configuration\n", "\n", "- Training configuration\n", "\n", "- Optimizer and Scheduler configuration\n", "\n", "- Running configurations\n", "\n", "Flexibility is a big plus with this system. It lets researchers tailor their experiments to their specific aims and hypotheses.\n", "\n", "### [Training Module](../training.rst)\n", "\n", "This module handles the execution of the experiment, whether it is a single prototype experiment (using `ProtoTrainer`) or a parallel experiment with ablation study/ HPO (using `ParallelTrainer`). It has a training interface, `ModelWrapper`, that encapsulates common boilerplate code to abstract the repetitive tasks. The Configuration Module and the Trainer Module are blueprints for shaping the experiment.\n", "\n", "### [Experiment result metrics module](../results.rst)\n", "\n", "This class plays a pivotal role in capturing the performance of models. It receives predictions and outputs after the training of the ML model, and applies specific evaluation functions to calculate metrics.\n", "\n", "### [Analysis Module](../analysis.rst)\n", "\n", "The Analysis module in Ablator provides essential tools for visualizing and interpreting the experiment outcomes. It consists of two main classes: `PlotAnalysis` and `Results`. This module comes into play after training and evaluation are completed. It takes the experiment results, which include metrics and hyperparameters, and processes them using the `Results` class. The processed data is then visualized using the `PlotAnalysis` class, creating insightful graphs." ] }, { "cell_type": "markdown", "id": "62252634", "metadata": {}, "source": [ "## What happens in the background once Ablator is launched? \n", "\n", "\"Ablator\n", "\n", "1. Ablator initializes a Ray cluster for parallel execution. As specified, trials are sampled randomly from the search space in ablation studies, and greedily via TPE algorithm in HPO experiments. In both cases, Optuna is employed to collect and possibly analyze the trials performance to effectively guide the exploration for the next trials within the search space.\n", "\n", "2. Trials are scheduled to run in parallel on available resources in the Ray cluster. The experiment directory is synchronized after each trial is completed, regardless of success or failure. This ensures that trial-specific information and results are stored for analysis (**Scheduling** from the Figure).\n", "\n", "3. Suppose the user-specified total number of trials is not reached (aka it's greater than the sum of the trials that have been completed and those that are currently pending). In that case, the system initiates new trials. New trials are passed to initiate additional parallel training on the ray cluster. These are scheduled on available resources in the Ray cluster, potentially replacing completed trails. This way, utilization of the available resources is maximized.\n", "\n", "4. After all trials are completed, a final synchronization of the entire experiment directory is done. All trial data, metrics, logs, and other relevant information are transferred for further analysis and storage. From this result, additional post-processing, analysis, and result aggregation tasks can be performed by the users. Moreover, Ablator's Analysis module aids in crafting visual plots that help understand how different components affect the model's performance, going beyond just looking at individual trial results. (**Syncing** and **Analysis** from the Figure)\n", "\n", "The utilization of the Ray cluster's parallel processing capabilities, coupled with the effective search space exploration, provides an efficient way to perform ablation studies and elevate machine learning model performance." ] }, { "cell_type": "markdown", "id": "bab73b73", "metadata": {}, "source": [ "To give you an idea of how experiments with ablator are written, we provide you with [these illustrative examples](./GettingStarted-more-demos.ipynb)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.4" } }, "nbformat": 4, "nbformat_minor": 5 }