Yee-King M.J. Build AI-Enhanced Audio Plugins with C++

Routledge, 2024. – 362 p. – ISBN: 978-1-032-43042-3.Build AI-Enhanced Audio Plugins with C++ explains how to embed Artificial Intelligence (AI) technology inside tools that can be used by audio and music professionals, through work examples using Python, C++, and audio APIs which demonstrate how to combine technologies to produce professional, AI-enhanced creative tools.Alongside a freely accessible source code repository created by the author that accompanies the book for readers to reference, each chapter is supported by complete example applications and projects, including an autonomous music improviser, a neural network-based synthesizer meta-programmer, and a neural audio effects processor. Detailed instructions on how to build each example are also provided, including source code extracts, diagrams, and background theory.The first aim of this book is to show you how to build AI-enhanced music software. However, I will take a different approach to many other AI and signal processing books. I have carefully developed this approach to solve specific problems and to address particular challenges you will face as a developer of AI-enhanced music software.My second aim is to show you how to construct the technology in a way that makes it accessible to musicians and audio professionals. Knowing how to build and run AI-enhanced music systems on your machine is one thing. A quirky Python script hacked together to work on your setup is acceptable for research and experimentation but you will have trouble getting musicians to use it. What is the ideal method for sharing software in a form that musicians and other audio professionals can use? The next aim of this book is to answer that question and to apply the answer to the design of the examples in the book. For many years I have worked as a researcher/engineer on research projects where one of the aims has been to get new technology into the hands of users so we can evaluate and improve that technology. There are many approaches to achieving this aim: running workshops with pre-configured equipment, making the technology run in the web browser, and so on. All of these approaches have their merits and are appropriate in different circumstances. But none of them is quite suitable for our purposes here. Here we are aiming to write software that can be used by musicians, producers, and sound engineers with minimal effort on their part.If you are a student programmer at the undergraduate or postgraduate level, you will find new knowledge in all areas of the book. You might be assigned all or part of this book as a textbook for a course you are studying, or you might have discovered the book independently. The book will teach you some C++ programming along the way, as well as help you to understand how to use an IDE and associated tools to develop software. You will probably find the setup section of the book very helpful as, in my experience, students spend a lot of time struggling to get their software development environment set up and working correctly. The book also provides information about general audio processing techniques and how they can be adapted using AI technology.If you are a Machine Learning engineer or AI scientist wishing to apply your skills in the music software domain, the book will help you do that. You will be familiar with the concepts of machine learning models, training, and inference, but you will likely need to become more familiar with digital signal processing techniques. You may be unfamiliar with common music technology such as plugins, MIDI data, etc. The book will explain exactly how to work with those technologies. As a Machine Learning engineer, you likely work primarily in Python or a specialized language. The book covers some Python but is mainly focused on C++ programming. You should find the information about setting up a C++ development toolchain useful.In Chapter 6, I will provide instructions on how to set up your Python environ­ment for the work in later chapters. We will only use Python for training neural networks and some analytical work. We will use C++ to build applications. When Python code is needed, I will not explain it in such great detail as the C++ code because Machine Learning in Python is very well covered in other books, unlike the C++ content. I will explain the principles behind the code and provide work­ ing example code in the code repository. Only some of the examples in the book require Python code.This is an essential guide for software developers and programmers of all levels looking to integrate AI into their systems, as well as educators and students of audio programming, Machine Learning, and software development.I Getting started.
Introduction to the book.
Setting up your development environment.
Installing JUCE.
Installing and using CMake.
Set up libtorch.
Python setup instructions.
Common development environment setup problems.
Basic plugin development.
FM synthesizer plugin.
II ML-powered plugin control: the meta-controller.
Using regression for synthesizer control.
Experiment with regression and libtorch.
The meta-controller.
Linear interpolating superknob.
Untrained torchknob.
Training the torchknob.
Plugin meta-controller.
Placing plugins in an AudioProcessGraph structure.
Show a plugin’s user interface.
From plugin host to meta-controller.
III The autonomous music improviser.
Background: all about sequencers.
Programming with Markov models.
Starting the Improviser plugin.
Modeling note onset times.
Modeling note duration.
Polyphonic Markov model.
IV Neural audio effects.
Welcome to neural effects.
Finite Impulse Responses, signals and systems.
Convolution.
Infinite Impulse Response filters.
Waveshapers.
Introduction to neural guitar amplifier emulation.
Neural FX: LSTM network.
JUCE LSTM plugin.
Training the amp emulator: dataset.
Data shapes, LSTM models and loss functions.
The LSTM training loop.
Operationalising the model in a plugin.
Faster LSTM using RTNeural.
Guide to the projects in the repository.