PROCESS
Our system will take a .wav audio file as an input. We focused on two types of audio - dog barks and a recording of the original “We Apologise.” Our software will then cut off everything past the first 5 seconds and slow the input file down to 10x the length using the PaulStretch algorithm and take the constant-q transform (CQT) of the result using the libROSA library. The CQT will then be remapped to the frequencies matching the 88 piano keys. It will then use a user specified instrument to build the spectrum of the input by using pyFluidSynth to create a MIDI note of matching fundamental frequency for each note. Using the user specified magnitude threshold, magnitude results from the CQT will be converted to note velocities. Since MIDI notes at higher pitches are quieter than those of lower pitches, the velocities are lowered linearly per key, descending. 88 pyFluidSynth channels are created with initial volume 0, with a maximum velocity note on event in each. For each CQT time frame, the channel’s velocity is set for each note to the correct volume using the calculated velocities, and recorded in an array. The MIDI notes will be created using a user specified S2F soundfont. Finally, PaulStretch is used to bring the reconstructed audio up to the playback speed chosen by the user (from 10x slower than the original to the speed of the original). To test results, the cross similarity matrix between the input audio and the MIDI audio will be computed. In addition, users can make qualitative comparisons between the input and output audio. The original “We Apologise” audio and choral reconstruction are used as a baseline to make these qualitative comparisons.
