Expensive gradient evaluation for multivariate functions

[jeanchristopheruel]

Multivariate gradient evaluation is very expensive since the function is evaluated as many times as the number of variables.

Minimal example:

uint32_t mock{0};
auto f = [&mock](Eigen::Ref<const Eigen::Matrix<autodiff::real, 1, 8>> const& x) -> autodiff::real
{
    ++mock;
    //  **very expensive computations**
    return x * x.transpose();
};

Eigen::Matrix<autodiff::real, 1, 8> x{1, 2, 3, 4, 5, 6, 7, 8};

autodiff::real u;
Eigen::VectorXd gx = gradient(f, wrt(x), at(x), u);
std::cout << "mock: " << mock << "\n"; 
// mock: 8

[allanleal]

Hi @jeanchristopheruel , this happens by design, and it is not an issue - this is how a forward mode automatic differentiation algorithm works. For gradient of a scalar function with many variables, a reverse algorithm, using var, computes all of them in a single pass (and it is more suitable for gradient of scalar functions). A forward algorithm is often more suitable for Jacobian computations of vector functions.

[jeanchristopheruel]

As always, thanks Allan for you quick reply ! Your answer makes a lot of sense. I have just read on the subject and I now understand much better the mechanics of forward and reverse modes.