Automatic Parameter Tuning for Machine Learning

I like to introduce the MetaOpt optimization kit which targets black box optimization and parameter tuning of heuristic algorithms respectively.

In the case of supervised learning in machine learning, the task is to learn a certain pattern-label mapping, so that the empirical risk of misclassification is reduced. The supervised learning methods like for example k-nearest neighbors or support vector machines have hyper-parameters which are responsible for the quality of the classification. I am writing classification, but it is true for regression as well. It is common practice to score the quality of the classifier dependend on certain hyper-parameters with cross validation. However, the choice of hyper-parameters remains an open task.

Toy Example

For this example, we use the Iris flower data set and the infamous soft-margin SVM with an RBF kernel. In order to fit the classifier to the data set we have to set the free hyper-parameters C and Gamma with respect to the maximization of the accuracy score. Hence, we need to find the maximum in the fitness landscape, where the fitness equals the accuracy score dependent on the free hyper-parameters C and Gamma.

Fitness Landscape

A naive approach is grid search. Each parameter combination is tested and the best parameter combination regarding to quality of the classifier is chosen. In dependence on the amount of parameter combinations, the supervised learning method, the size of the data set and the chosen cross validation method, this approach does not scale well. Let us use the power of meta heuristics and parallelism (not to mention the accelerating change)!

What We Need: MetaOpt

@maximize("Score")
@param.float("C", interval=[1, 10], step=0.5)
@param.float("gamma", interval=[1, 10], step=0.5)
def f(C, gamma):
    iris = datasets.load_iris()
    X_train, X_test, y_train, y_test =\
        cross_validation.train_test_split(
        iris.data, iris.target, test_size=0.4, random_state=0)
    clf = svm.SVC(C=C, gamma=gamma)
    clf.fit(X_train, y_train)
    return clf.score(X_test, y_test)

It is nothing new to minimize the empirical risk with evolutionary algorithms (EAs) like evolution strategies or particle swarm optimization, see here. While EAs are successfully applied to optimize the hyper-parameters, a common practice is missing. This leads to task-specific over-specialized implementations, which are unsustainable and not useful for others practitioners. Clearly, an universal easy-to-use toolkit for parameter tuning with meta heuristics is required.

In my work as a evolutionary algorithms researcher, we faced this problem many times and took the opportunity to develop MetaOpt, an easy-to-use toolkit for parameter tuning with different meta heuristics and out-of-the-box parallelism. As always, a code example gives you a good impression of how to use it, see above and below.

Super Powers: Time Constraints and Plugins

There is another kid arround the block, namely HyperOpt, for the same use case. However, if MetaOpt would be a super hero in the field of parameter tuning, and in my opinion it is, its super powers are time constraints and an impressive plugin infrastructure (plugins like logging and visualization).

from metaopt.core.main import optimize
from metaopt.optimizer.cmaes import CMAESOptimizer
from metaopt.plugins.print import PrintPlugin

timeout = 60 # 60 seconds
optimizer = CMAESOptimizer()
plugins = [PrintPlugin()]

# spawns multiple worker processes for multi-cores
optimum = optimize(f=f, timeout=timeout, optimizer=optimizer,
                   plugins=plugins)

Imagine you want to tune your new approach to a set of problems and you have some computation time on a cluster. Just write the tuning script as similiar as the brief example above and specify a global time constraint after which the best parameter combination is presented. You can even constrain the time of an evaluation (train and test of the classifier or a run of an optimizer). The time constraint features are implemented as a plugin. The code example illustrates how you can optimize the function f from above with a certain time constraint.

Self-Optimization

MetaOpt is universal. If you have a numerical black box optimization problem you can use MetaOpt to find the best solution with different meta heuristics. As I already mentioned, machine learning is clearly just one big field of application. Another important field is the parameter tuning of heuristical algorithms such as the optimizer heuristic itself, see Organic computing.

Start Your Engine

The first steps to get in touch with MetaOpt: Install it over PyPI (package name: metaopt, read the documentation and understand the examples. The first stable version of MetaOpt was just released. Clearly, there is still a lot to accomplish to transform MetaOpt into a robust optimizer for a variety of real world problems. Nevertheless, I am really looking forward to this innovative approach of automatic parameter tuning and I hope that the usefulness of MetaOpt will motivate its users to spread the word in the machine learning as well as in the heuristics engineering community. We really appreciate contributions over GitHub. Especially, meta heuristics engineers are invited to add their heuristics as optimizers to MetaOpt.


One or two mails a month about the latest technology I'm hacking on.