Real-time Audio Processing with FfdShow

[MysteryX]

I want to change the audio pitch from 440hz to 432hz in real-time. So far, the best approach I found is this: creating an AVS script and opening that script in the player.

This is the script that works best.

PluginPath=""
LoadPlugin(PluginPath+"LSMASHSource.dll")
LoadPlugin(PluginPath+"TimeStretch.dll")
Import(PluginPath+"UUSize4.avsi")
SetMTMode(3,4)
file = "Video.mp4"
LWLibavVideoSource(file, cache=false, threads=1)
AudioDub(LWLibavAudioSource(file, cache=false))
SetMTMode(2)
UUSize4(mod=4)
ResampleAudio(48000)
TimeStretchPlugin(pitch = 100.0 * 0.98181819915771484)

Both FFmpegSource2 and LSMASHVideoSource crash regularly; LWLibavVideoSource is the only method that is reliable "most of the time". It still crashes (causing the player to reset the video) but rarely. I'm adding this into a software that automatically generates the script and plays the videos so the fact that the script must be edited every time isn't an issue.

Still this approach has several issues.

First, it adds instability. It crashes now and then which causes the video playback to reset.

Second, when loading videos, it doesn't always load the frames fast enough and the playback lags.

Perhaps another approach would be to plug into FfdShow like SVP is doing. When SVP plugs into it, it processes only the video and it has no access to alter the audio.

Is it possible to plug into the FfdShow Audio chain to implement this? If so how?

I'm looking into a more stable implementation of the 432hz auto-pitch.

[pandy]

Resample native 48000 to (48000*440)/432 then force sample rate back to 48000.

Simplest way to use ffdshow is:
create preset in ffdshow then use preset in avs by:

ffdshowAudio ("preset name")

[MysteryX]

Where did you find about ffdshowaudio? I don't find anything at all about this in Google.

Just trying to use it gives "there is no such function"

[StainlessS]

try google for better hits

"ffdshowaudio" site:forum.doom9.org

[MysteryX]

try google for better hits

"ffdshowaudio" site:forum.doom9.org

Still not finding anything clear. I'll need some kind of DLL because it's not a built-in function, and there is no official page to download that DLL, nor any online documentation about it.

[StainlessS]

Here:- http://www.avisynth.nl/users/fizick/docs/english/externalfilters/ffavisynth.htm

EDIT:

download: http://sourceforge.net/projects/ffdshow-tryout/ (plugin is included in ffdshow package)

EDIT: From top Link

Description

ffavisynth - A plugin which lets you directly use ffdshow image processing filters from AviSynth scripts.

Just install ffdshow (with Avisynth support option checked), and the plugin ffavisynth.dll will be installed in your avisynth plugin folder.

In ffdshow-20051124.exe and more recent versions, ffavisynth filter uses the AviSynth C interface, and thus should be loaded using
LoadCPlugin (or Load_StdCall_Plugin).
Since 2007-10-30 ffavisynth.dll is loaded automatically by AviSynth (with ffavisynth.avsi).

Syntax

video: ffdshow (clip, string "preset", string "options")
audio: ffdshowAudio (clip, string "preset", string "options")


EDIT: There are several mentions of 'ffdshow' and 'ffavisynth' in Avisynth builtin Docs,
Might be a little bit difficult to find if you dont use the MS Windows file indexing stuff (I dont),
I use Avisynth Compressed HTML help file for searching docs (*.chm file).
In MediaFire link in my sig below. In DATA Folder:- AvisynthEngHelp+SDK26_FINAL-2015-05-31.zip
EDIT: To show search options, click on "Show" in top left corner of helpfile window.

EDIT: Google below scans Avisynth.nl

"ffdshow" | "ffavisynth" site:avisynth.nl

[raffriff42]

http://www.avisynth.nl/users/fizick/docs/english/externalfilters/ffavisynth.htm
This page has never been ported to the regular wiki, probably because it's old and not maintained.
Nevertheless, I just ported it (http://avisynth.nl/index.php/FFAvisynth), FWIW.

BTW, I have ffdshow rev4532, Jul 17 2014 (the latest version on sourceforge (https://sourceforge.net/projects/ffdshow-tryout/)) and there is no support for pitch shifting.

AFAIK, there are no tools that do high quality pitch shifting in real time that don't require the purchase a a professional digital audio workstation (for example, Waves Soundshifter (http://www.waves.com/plugins/soundshifter), and even they don't promise the moon - from the manual: "This type of processing is not a purist process, but in modest ratios we claim relative transparency"). Mediocre quality, sure, you have lots of choices.

Even non-real time, the choices are limited. There's an interesting one (that I have not tested yet) called RubberBand (http://breakfastquay.com/rubberband/), which has a free command line tool available, and a nonfree but inexpensive GUI. (rubberband appears in the FFMpeg docs (https://ffmpeg.org/ffmpeg-filters.html#rubberband), but is not included in the binaries at this time). Take a look at these tuning options; there are similar options on all high quality pitch shifters, because pitch-shifting requires splicing short audio segments and there will always be tradeoffs. If a pitch shifter does not have these options, then it can't be optimized for the program material - therefore at least some material is gonna sound a little nasty.Rubber Band
An audio time-stretching and pitch-shifting library and utility program.
Copyright 2007-2012 Particular Programs Ltd.

Usage: rubberband [options] <infile.wav> <outfile.wav>

You must specify at least one of the following time and pitch ratio options.

-t<X>, --time <X> Stretch to X times original duration, or
-T<X>, --tempo <X> Change tempo by multiple X (same as --time 1/X), or
-T<X>, --tempo <X>:<Y> Change tempo from X to Y (same as --time X/Y), or
-D<X>, --duration <X> Stretch or squash to make output file X seconds long

-p<X>, --pitch <X> Raise pitch by X semitones, or
-f<X>, --frequency <X> Change frequency by multiple X

-M<F>, --timemap <F> Use file F as the source for key frame map

[...]

The following options provide a simple way to adjust the sound. See below
for more details.

-c<N>, --crisp <N> Crispness (N = 0,1,2,3,4,5,6); default 5 (see below)
-F, --formant Enable formant preservation when pitch shifting

The remaining options fine-tune the processing mode and stretch algorithm.
These are mostly included for test purposes; the default settings and standard
crispness parameter are intended to provide the best sounding set of options
for most situations. The default is to use none of these options.

-L, --loose Relax timing in hope of better transient preservation
-P, --precise Ignored: The opposite of -L, this is default from 1.6
-R, --realtime Select realtime mode (implies --no-threads)
--no-threads No extra threads regardless of CPU and channel count
--threads Assume multi-CPU even if only one CPU is identified
--no-transients Disable phase resynchronisation at transients
--bl-transients Band-limit phase resync to extreme frequencies
--no-lamination Disable phase lamination
--window-long Use longer processing window (actual size may vary)
--window-short Use shorter processing window
--smoothing Apply window presum and time-domain smoothing
--detector-perc Use percussive transient detector (as in pre-1.5)
--detector-soft Use soft transient detector
--pitch-hq In RT mode, use a slower, higher quality pitch shift
--centre-focus Preserve focus of centre material in stereo
(at a cost in width and individual channel quality)

-d<N>, --debug <N> Select debug level (N = 0,1,2,3); default 0, full 3
(N.B. debug level 3 includes audible ticks in output)
-q, --quiet Suppress progress output

-V, --version Show version number and exit
-h, --help Show this help

"Crispness" levels:
-c 0 equivalent to --no-transients --no-lamination --window-long
-c 1 equivalent to --detector-soft --no-lamination --window-long (for piano)
-c 2 equivalent to --no-transients --no-lamination
-c 3 equivalent to --no-transients
-c 4 equivalent to --bl-transients
-c 5 default processing options
-c 6 equivalent to --no-lamination --window-short (may be good for drums)

(What's RT mode? Real Time? I invite you to experiment)

[MysteryX]

I experimented with Rubberband.

According to my tests, the built-in TimeStretch causes severe distortion. Rubberband works quite well but in some cases also cause distortion.

TimeStretchPlugin works well, and works fine where Rubberband would cause distortion.

To avoid quality loss, it works better by first converting to 48000hz and sending the output as 32-bit float; anything less has an audible difference.

Resample native 48000 to (48000*440)/432 then force sample rate back to 48000.
Whether this gives better results, that would have to be tested.

[Motenai Yoda]

Whether this gives better results, that would have to be tested.
that change both tempo and pitch, as it simply slow down things
timestretch pitch actually IIRC works speedupping and perform tempo stuff, maybe using it to resample at 48000*432/440, stretch tempo to 440/432 and resamplerate at 48000 can do better.

also try playing with timestretch params...

btw I compiled an Soundstretch (timestretch cli) with vs2015, enabling mt and putting all optimization on fast, maybe some speed improvements can be done recompiling the avs plugin too.

ps:
- as I see there are many libs which rubberbands uses as there is one GPL and one only with licence, so that cli work with?
FFT libraries supported
-----------------------

Name Flags required Notes
---- -------------- -----

FFTW3 -DHAVE_FFTW GPL.

Accelerate -DHAVE_VDSP Platform library on OS/X and iOS.

Intel IPP -DHAVE_IPP Proprietary library, can only be used with
Rubber Band commercial licence. Define
USE_IPP_STATIC as well to build with static
IPP libraries.

KissFFT -DUSE_KISSFFT Bundled, can be used with GPL or commercial
licence. Single-precision. Slower than the
above options.
- to avoid stereo artifacts you have to add --centre-focus
- with media autobuild suite ffmpeg is compiled with it

[raffriff42]

I tried rubberband and it sounds pretty darn good!

Note it is set to "crispness 5" by default, which has artifacts on complete music mixes. I would suggest a test - listen to both rubberband -c 0 -f 0.9818181818 infile.wav outfile.wav
andrubberband -c 6 -f 0.9818181818 infile.wav outfile.wav
so you know what artifacts to listen for, then use - c 3 ±1 for the final product.

Here are all the crispness presets: -c 0 == "Mushy" (their term: transients smeared, a bit like reverb, but not)
-c 1 == "Piano"
-c 2 == "Smooth"
-c 3 == "Balanced multitimbral mixture"
-c 4 == "Unpitched percussion with stable notes"
-c 5 == "Crisp monophonic instrumental" (default)
-c 6 == "Unpitched solo percussion" ("gurgling" artifacts on music)

.....and make sure your source is clean, not distorted, and has some headroom (https://en.wikipedia.org/wiki/Headroom_%28audio_signal_processing%29) for signal processing overshoot; max peaks should probably be no more than -3dB.

[MysteryX]

Honestly, with TimeStretch.dll (not the internal function), by first converting to 48000hz and then having an output in 32-bit float, I listened carefully and couldn't hear any loss of quality between the original and the processed audio. There's a slight loss for sure, but it's insignificant.

One audio that is challenging is the electronic bass at the beginning of this song
https://www.youtube.com/watch?v=C2-AZ51NoQU

[raffriff42]

Checking your source, TimeStretchPlugin does sound better, especially since it holds a more stable stereo image. TimeStretchPlugin also has useful tuning options - I found a slight improvement to my ears with more smoothing:
TimeStretchPlugin(pitch=0.9818181818 * 100.0, sequence=256, seekwindow=100, overlap=48)

[pandy]

that change both tempo and pitch, as it simply slow down things

I want to change the audio pitch from 440hz to 432hz in real-time.

Well... i assumed transparent processing (no quality loss except resampling - can be ignored unless you can hear something at -160dBFS).

There is no good quality (perceptually lossless) algorithm for transposing pitch/tempo independently - there are various algorithms tailored for different types of signals.

[MysteryX]

OK I did a clean re-install of AviSynth 2.6, re-installed FfdShow and saw the option to add AviSynth plugins.

Now how do I get the audio processed?

In MPC-HC, I supposed I have to add either "ffdshow Audio Decoder" or "ffdshow Audio Processor" to External Filters. Which one?

Then for presets, can I use preset "defaut"?

And then... how do I actually get the AviSynth code to run and plug into the Ffdshow chain? Do I have to somehow run it in a separate process than MPC-HC, and it will update the audio of the player for as long as that process is running?

[pandy]

Then for presets, can I use preset "defaut"?


You can but better is to create dedicated profile (you can use default as source) - select filters in ffdshow, set all options as required, save and later call/use inside AVS.

Alternatively you can call directly filters with parameters - save preset (i would advise use name different than default and manually analyze to find proper options names and values) - this method is recommended if you need to modify values.

[MysteryX]

The most important question still hasn't been answered.

How am I supposed to run the AVS script while the video is playing in a different player? And how do I have to configure MPC-HC so that the audio passes through ffdshow?

[raffriff42]

And then... how do I actually get the AviSynth code to run and plug into the Ffdshow chain?You can't. The ffdshow audio decoder does not support Avisynth audio processing. Only the video decoder does.

The ffdshow audio decoder does support Winamp2-compatible plugins. I have used Pacemaker version 1.32 (http://www.surina.net/pacemaker/old/index.html) for low-fi purposes (speed listening of speech) both within Winamp5 and as an ffdshow plugin within MPC-HC, and it works pretty well. Unfortunately, Pacemaker cannot be tuned more finely than ±0.01% - actually that is optimistic due to its finicky slider control; realistically, it's ±1%.

How am I supposed to run the AVS script while the video is playing in a different playerIn a different player? Why?

To be honest, my actual advice to you is to run a script like the following. Yes, this will increase running time (by approximately 1.1 seconds per minute). Live with it. At least you will have no annoying artifacts in your audio. If you really want to, you can convert the video framerate using SVP or whatever. V=AssumeFPS(432.0/440.0*FrameRate)
A=AssumeSampleRate(432.0/440.0*AudioRate)
AudioDub(V, A)

[MysteryX]

Then the best approach still is to open the video file the way I'm doing.

10-bit videos fail, obviously.

I know there's a 'beta' plugin that opens 10-bit videos in Stack16 format, and is there a way to just open the video file and send it to the player without processing it?