Project Overview#
Title: Anderson Acceleration Algorithms for Nonlinear Systems and Optimization Problems#
This project explores the development and application of Anderson acceleration algorithms to address challenges in optimization problems, which are fundamental in various fields such as computational science, engineering, operations research, statistics, data analysis, machine learning, and information processing.
Optimization problems often involve the minimization or maximization of an objective function subject to constraints. Efficiently solving these problems is crucial for applications ranging from training machine learning models to tuning parameters in climate models. Traditional optimization methods, including gradient descent and Newton’s methods, rely on the availability of (approximate) first and second derivatives of the objective function. However, the large scale and stochastic nature of contemporary optimization problems can render these methods computationally expensive or ineffective.
In response to these challenges, acceleration methods, particularly Anderson acceleration, have emerged as powerful tools to enhance convergence rates. These methods can be applied to systems of nonlinear equations and optimization problems, providing a generic approach to accelerate existing algorithms.
Avenues of Investigation#
The project will investigate:
Theoretical aspects of Anderson acceleration, including its application to stochastic optimization problems.
Practical applications in climate models, specifically aimed at speeding up the convergence of ocean model integrations.
The exploration of finding all solutions to a problem, which may necessitate the repeated application of Anderson acceleration.
Current Status#
This project is a work in progress, and the findings will continue to evolve as the research develops.
So far we have evaluated the performance of vanilla anderson on various simple stochastic problems, worked on tweaking the algorithm for improved performance and tested performance on simple 2D ocean simulations.
We are using Oceananigans.jl for our wind-forced ocean flows. Some visualisations can be found here.
Some details of our tweaked algorithm APCI (Adaptive Probability Coordinate Importance) Anderson and figures showing performance can be found here.
The model