Introduction#
Welcome to the online supplement to the tutorial on “Introduction to Differentiable Audio Synthesiser Programming”, presented at the 24th International Society for Music Information Retrieval Conference at Politecnico di Milano, November 5th — 9th 2023. This web book contains all content presented at the tutorial, including code examples and references. It also contains further material, going into greater depth on the topics covered.
Motivation & Aims#
The field of differentiable digital signal processing (DDSP) emerged with the incorporation of components, such as linear synthesis filters [JBYA19] and spectral modelling synthesisers [EHGR20], into the end-to-end training of neural networks for speech and musical instrument synthesis. It has since grown to encompass applications including pitch estimation [ESH+20], source separation [SFRK+23], physical parameter estimation [SudholtCXR23], synthesiser sound matching [MS23], and beyond. By introducing a strong inductive bias, DDSP methods can often lead to a reduction in model complexity and data requirements, and provide an intermediate representation that is inherently interpretable in terms of familiar parametric signal processors.
Yet despite the growing popularity of such methods in research, the implementation of differentiable audio synthesizers is not always clearly documented, and the simple formulation of many synthesizers can obscure what often turns out to be complex optimization behaviour. This tutorial aims to address this gap through an introduction to the fundamentals of differentiable synthesizer programming. In particular, we hope that researchers in adjacent fields may find applications for these techniques in their work.
Who is this for?#
This book and the accompanying tutorial are aimed at music and audio researchers and engineers who wish to gain a practical understanding of how differentiable digital signal processors can be implemented and used in audio synthesis. The content is targeted at those with a grounding in the fundamentals of signal processing and machine learning, although references to educational resources are provided where relevant for those who wish to refresh or supplement their existing knowledge. We assume prior knowledge of Python 3. The tutorial content is written using the PyTorch machine learning framework. We do not assume prior experience with PyTorch, and attempt to provide sufficient background to allow the tutorial content to be understood. Nonetheless, we encourage those who are totally new to PyTorch to explore the official tutorials to provide further context.
A note on scope: we’re not training neural networks#
The majority of applications of DDSP involve its use in combination with a neural network, which typically produces the parameters of the signal processing operation (e.g. the cutoff frequency of a filter, or harmonic amplitudes). In this tutorial and web book, however, we focus only on the components of this system after the parameters are given — that is, we concern ourselves only with the differentiable signal processing. Nonetheless, all of the techniques we present can be (and mostly have been) composed with a neural network, as a direct consequence of their implementation using differentiable operations in a machine learning framework.
Getting Started#
This web book consists of a series of Jupyter notebooks, which can be explored statically on this page. To run the notebooks yourself, you will need to clone the Git repository:
git clone https://github.com/intro2ddsp/intro2ddsp.github.io.git && cd intro2ddsp.github.io
Then, you should create a Python virtual environment, using virtualenv, conda, or similar:
python -m venv venv && source venv/bin/activate
or:
conda create --name intro2ddsp python==3.10 && conda activate intro2ddsp
Next, you should install the dependencies:
pip install -r requirements.txt
And finally, you can launch the Jupyter notebook server:
jupyter notebook