Autodiff and GPUs

Before presenting the inner workings of KeOps, we recap briefly the main rules of GPU programming and automatic differentiation. This short introduction will, hopefully, allow readers to understand the capabilities and limitations of these (remarkable) tools.

Note that this section is extracted from the Chapter 2.1 of Jean Feydy’s PhD thesis.

Autodiff and GPUs

• The winning combination
• What is a GPU?
  • Parallel computing
  • Memory management
  • The CUDA development toolkit
  • How much is this going to cost?
  • KeOps is portable and free
• Backpropagation
  • Definitions
  • Computing gradients
  • Metric structure, chain rule
  • Backpropagation
• High-level frameworks
  • A minimal working example
  • Encoding formulas as tree-like objects
  • A well-packaged backropagation engine
  • Custom operators, higher-order differentiation