Time Length Adjustment Tools Quality Comparison

[bond]

ok finally i found the time to do a quality comparison of various tools able to do timelength adjustments on audio streams

Test Setup
for my comparison i used the great ABC/HR (http://ff123.net/abchr/abchr.html) audio comparison tool (simply a must when doing "ear-based" quality comparisons :) )

i tested the following tools:
- Audacity 1.2.1
- Adobe Audition 1.0
- BeSweet v1.5b27 - OTA
- BeSweet v1.5b28 - Soundtouch
- Sonic Foundry Sound Forge 6.0
- Prosoniq Timefactory 1.6.1

i tested a 12.721 secs long stereo sample with 44.1khz samplerate
the original sample was a (not good quality) recording in mp3 format i had sync problems with when combining with a 25fps video stream
with 24.996fps the sync problems were gone, therefore i decided to use this situation for doing a 24996/25000 (99.984%) audio conversion, together with this quality comparison to be able to get sync at 25fps at the maximum quality possible


First of all, as most tools offer different options, i did some internal tests to find out which setting would bring the best result:

1) Audacity
i noticed that doing a length change from the source mp3 directly in audacity will lead to less quality than if feeding audacity with a .wav -> therefore i decoded the .mp3 to .wav in besweet first
i compared a tempo change only vs. tempo->pitch change sample => tempo-change only provided better quality (especially regarding the background noise which the pitch change somehow seemed to filter out)
2) Audition
i compared a tempo change only vs. tempo+pitch change sample => tempo+pitch provided better quality (the background noise sounded more than the one in the source)
3) Besweet Soundtouch
i compared a sample using the "rate" option (tempo+pitch change) vs. a tempo only sample => the range sample was better (the difference was not that big tough, when compared to the other pretty clear results)
4) Sound Forge
i compared a sample using the "mode 1" option vs. the "mode 3" option offered => mode 1 was better
5) Timefactory
i compared with vs. without "preserve formants" => without was better (the difference was not that big here too)

this means that finally i tested the following settings:
- BeSweet v1.5b27: OTA - Change Frame Rate from 24496 to 25000
- BeSweet v1.5b28: -soundtouch( -rate 0.99984 )
- Audacity 1.2.1: Change Tempo: 0,99984
- Adobe Audition 1.0: Time/Pitch -> Stretch: Ratio: 99.984, Precision: High, Stretching Mode: Resample (preserves neither)
- Sonic Foundry Sound Forge 6.0: Time Stretch: Mode 1 99,98%
- Prosoniq Timefactory 1.6.1: Duration 99.984%, Algorithm: Polyphonic (best)


Results
first of all some statistical results:

Tool Length (secs) Size (bytes)
Source 12.721 2.244.140
Timefactory 12.719 2.243.780
Audition 12.719 2.243.780
Sound Forge 12.719 2.243.800
BS (OTA) 12.691 2.238.824
Audacity 12.599 2.222.508
BS (Soundtouch) 12.595 2.221.808note that the correct length of the output should be 12.719 secs (12.721x0.99984)

now the quality comparison results from the double blind listening test in abc/hr (max. 5.0/min. 1.0 points possible) and my comments on the different tools:
Prosoniq Timefactory 5.0 (no difference to the source hearable)
Adobe Audition 5.0
Sound Forge 4.3
Besweet (Soundtouch) 2.7
Audacity 1.5
Besweet (OTA) 1.0ABC/HR Version 0.9b, 30 August 2002
Testname: Time Shorten Test ( 99.984% )

1L = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundforge-mode1.wav
2L = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_ota.wav
3R = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_prosoniq-P0 0 T1.00 1.wav
4L = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_audacity.wav
5R = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundtouch-rate.wav
6R = C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_audition-tempo-pitch.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundforge-mode1.wav
1L Rating: 4.3
1L Comment: 11.6 - 12.7 secs -> very small detail missing
---------------------------------------
2L File: C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_ota.wav
2L Rating: 1.0
2L Comment: horrible
---------------------------------------
4L File: C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_audacity.wav
4L Rating: 1.5
4L Comment: seems to have a stereo problem - after ~2 secs the right channel becomes stronger
---------------------------------------
5R File: C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundtouch-rate.wav
5R Rating: 2.7
5R Comment: 0.0 -1.5 secs -> the background noise seems to be at a lower volume
---------------------------------------
ABX Results:
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundforge-mode1.wav
61 out of 92, pval = 0.001
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_ota.wav
9 out of 9, pval = 0.002
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_prosoniq-P0 0 T1.00 1.wav
14 out of 29, pval = 0.644
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_audacity.wav
9 out of 9, pval = 0.002
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_soundtouch-rate.wav
24 out of 31, pval = 0.002
Original vs C:\Eigene Dateien\Eigene Videos\Test\Audio\timestretch\bond_audition-tempo-pitch.wav
42 out of 85, pval = 0.586Conclusion
Sad but true, the opensource time adjustment tools cant keep up qualitywise with the commercial tools in this test
I only tested a small time length adjustment but it was already enough to sort a whole bunch of tools out qualitywise in this test

As a result of this test i will use prosoniq timefactory or adobe audition for the conversion i tested here, as i couldnt hear a difference to the source with them in this test, for other conversions (and if you really cant do it with a simple and 100% lossless changing of the video framerate) i would recommend you to do your own tests to be absolute on the safe side :)


note: seems i made a mistake with the correct settings for audacity and soundtouch - soundtouch is able to hit the target length exactly!
read more below

[DSPguru]

very nice, bond!
appreciated.

Originally posted by bond
Conclusion
Sad but true, the opensource time adjustment tools cant keep up qualitywise with the commercial tools at the moment
I only tested a small time length adjustment but it was already enough to sort a whole bunch of tools out qualitywise, not to speak of that the opensource tools also werent able to hit the desired exact length

If you really need to change the length of your audio stream and you cant solve the problem with simply changing the framerate of your video, i would recommend you to use Prosoniq Timefactory or Adobe Audition as i couldnt hear a difference to the source when using these two tools, and believe me i really tried tonow, here i'm afraid your conclusion is to rash.
i mean, why do you consider a stretch of 99.9984% to be archetypical ?
wouldn't it be better to test more modes in order to be able to come up with such a generalized conclusion ?

i have no personal problem with your results (i'm not SoundTouch's author, after all..).
though, i think it would be more useful if you make another blind-test, testing the popular conversion modes (you know.. ntsc2pal, pal2ntsc).


Cheers,
Dg.

[bond]

Originally posted by DSPguru
i mean, why do you consider a stretch of 99.9984% to be archetypical ?
wouldn't it be better to test more modes in order to be able to come up with such a generalized conclusion ?

i have no personal problem with your results (i'm not SoundTouch's author, after all..).
though, i think it would be more useful if you make another blind-test, testing the popular conversion modes (you know.. ntsc2pal, pal2ntsc).yeah exactly
still basically there are two reasons why i wrote as conclusion that i recommend timefactory and audition (and sound forge, which only showed a small difference, but therefore wasnt transparent):
1) the output time lenght, which besweet ota, soundtouch and audacity didnt match correctly
2) the quality, which was clearly worse than the commercial tools ones

of course doing 24.996->25.000 fps is in no way representative, but still i have my doubts that both the output lenght precision AND quality would get better with standard ntsc<->pal conversions
of course it could be possible

still we dont know it (and i am too lazy to conduce such a test) and therefore to be on the "safest" side i would recommend using the mentioned commercial tools, i think you simply risk less with them

i rewrote my conclusion a little bit to sound not that "overall valid" :)

[DSPguru]

i have no problems with your recommendation, but with your conclusion :
"I only tested a small time length adjustment but it was already enough to sort a whole bunch of tools out qualitywise"

come on, bond, i'm really sorry to tell you - man doesn't have to be a logician to see that this is non-logic you're talking.
i've noticed this property several times in the past... you have this tendency of jumping into conclusions.

don't take it as personal offence, but with this kind of logical deduction system ("pick an absurd representative, prove/refute according to it, deduce a generalized conclusion"), the "four color" theorem (graph theory) would have been proven one hundered years ago, Eucluidean's "proof" of the fundemental theorem of arithmetic (cf. Elements IX, 14) would be considered as a proof, Suslin's hypothesis would become "truth" (instead of consistently true), and who knows, perhaps cryptographers had start regarding Carmichael numbers as primes (due to the Fermat property they satisfy).

problem is that Godel's completeness like-theorem would become inconsistent for this formal-logic system (even the soundness is lost!), and a more surprising result would be the fact that any adequate axiomatizable theory becomes "complete" (contradicting Godel's first incompletness theorem).
now, we don't want to work with those kinds of logical systems, do we ?

so.. please.. i beg you to be more cautious when formulating your future conclusions.


on a more particular note,
- i would suggest you to mail soundtouch's author with a link to here, he might have interesting things to say.
- about the drift in the signal's playback length, i'm interested to know what is missing - the beginning or the end or is it that the stretch factor was incorrect ?
i believe that a loss of couple of miliseconds at the end of a 4mins song (or a 100mins track) is negligible, but a drift in the beginning is obviously harmful.

[rjamorim]

don't take it as personal offence, but with this kind of logical deduction system ("pick an absurd representative, prove/refute according to it, deduce a generalized conclusion"), the "four color" theorem (graph theory) would have been proven one hundered years ago, Eucluidean's "proof" of the fundemental theorem of arithmetic (cf. Elements IX, 14) would be considered as a proof, Suslin's hypothesis would become "truth" (instead of consistently true), and who knows, perhaps cryptographers had start regarding Carmichael numbers as primes (due to the Fermat property they satisfy).

From this point of view, taking it to extremes, my listening test results would only be valid if testing all bitrates, with all music samples, on the entirety of mankind.

Take a chill pill, dude. This is not drug testing :)

[stephanV]

Originally posted by bond
this means that finally i tested the following settings:
- BeSweet v1.5b27: OTA - Change Frame Rate from 24496 to 25000
- BeSweet v1.5b28: -soundtouch( -rate 0.99984 )
- Audacity 1.2.1: Change Tempo: 0,99984
- Adobe Audition 1.0: Time/Pitch -> Stretch: Ratio: 99.984, Precision: High, Stretching Mode: Resample (preserves neither)
- Sonic Foundry Sound Forge 6.0: Time Stretch: Mode 1 99,98%
- Prosoniq Timefactory 1.6.1: Duration 99.984%, Algorithm: Polyphonic (best)


I believe the setting for Audacity is incorrect. Filling in 0.99984 as percentage would mean +/- a 1% speed up.

(12.721-12.599)/12.721 * 100% = 0.95%

The 'change tempo'-dialog box in Audacity shows 0% for the original value and the value must be filled in according to that. The correct value to fill in would be 25*100/24.996 - 100 = 0.016%.

When using 'change speed' with this value for a 30 second generated tone the result is 29.9952 seconds which is correct. Using 'change tempo' with this value results in a length of 29.96 seconds (not correct). However change tempo uses a horrible stretching algorithm to keep the pitch of the sound, which is for such a small speed change not necessary. Change speed would just do fine in this case IMO as the pitch change is very small.

I also tried 'change speed' with a 7.596 seconds MP3 track, this resulted in a length of aprrox. 7.5948 seconds (+/- 0.0156% speed up).

Anyway, this was my contribution to this test. I only can speak for Audacity, I've never used the other ones. :)

spelling

[Kika]

I have done a lot of Time Lenght Adjustments in the Past (23.976 -> 25) by using WaveLab3 (Highest Quality) and TimeFactory. I think, a test should be done on a longer File with Speech- and Music-Parts.
My first Test was an Anime with an orchestral Soundtrack. Done in WaveLab Fast Mode. Speech-Parts and some Effects like Explosions and so on came out good, but the Music-Parts where horrible.

So my suggestions for a propper Test:
Choose a Soundfile with a longer Playtime and Music- and Speech-Parts
Try different Lenght-Changes
Try different Source (converted WAV from MP3 or AC3-Source)

[bond]

Originally posted by DSPguru
i have no problems with your recommendation, but with your conclusion :
"I only tested a small time length adjustment but it was already enough to sort a whole bunch of tools out qualitywise"

come on, bond, i'm really sorry to tell you - man doesn't have to be a logician to see that this is non-logic you're talking.
i've noticed this property several times in the past... you have this tendency of jumping into conclusions. hum, first of all i have to say that my initial statement on its own was surely ambiguously, someone might see it as overall conclusion, someone else as only valid for this test (which is the obvious conclusion anyways imo)
but still you seem to have missed that i already (tried to) made this clear after your first comment with adding "in this test" at the end of the first sentence, but to wipe out all possible misunderstandings i will add "in this test" to all sentences now ;)

So my suggestions for a propper Test:
Choose a Soundfile with a longer Playtime and Music- and Speech-Parts
Try different Lenght-Changes
Try different Source (converted WAV from MP3 or AC3-Source)well again: i did this test because i wanted to find out what tool i should use in my specific case and posted my results because i thought they might be interesting for someone. i didnt do it to proove anything...
so anyone who wants to do a more representative test feel free to simply do it, i will surely help where i can

The 'change tempo'-dialog box in Audacity shows 0% for the original value and the value must be filled in according to that. The correct value to fill in would be 25*100/24.996 - 100 = 0.016%.
damn, seems i made this dumb mistake because most tools had 100% as default, which means 99.984 was the correct value :B

i tested my sample with the new value against the old one:
audacity:
0.99984 - 12.599
0.016 - 12.740 :confused:

soundtouch:
0.99984 - 12.595
0.016 - 12.719

the correct length would be 12.719, which means soundtouch hits the target correctly with the correct value :)

[DSPguru]

Originally posted by rjamorim
From this point of view, taking it to extremes, my listening test results would only be valid if testing all bitrates, with all music samples, on the entirety of mankind.may i remind you of the way the "four color" theorem had been eventually proved ?
no need for mankind's eternity. (even the infinite version of the theorem is valid, and this proof also does not require eternity - only a simple compactness argument).

seriously, roberto, i wonder why you are conducting those serious and long pre-test discussions if not to be able to avoid a mega-test ("all music samples, on the entirety of mankind"), while doing your best to ensure that you end up with a reasonable conclusion at the end of the test.

[DSPguru]

Originally posted by bond
the correct length would be 12.719, which means soundtouch hits the target correctly with the correct value :)good to hear :)

well again: i did this test because i wanted to find out what tool i should use in my specific case and posted my results because i thought they might be interesting for someone. i didnt do it to proove anything...very well. as said - i appreciate it !
it just that i'm afraid that some users, whom english isn't that sharP, might understand things differently, and i wanted to avoid that - therefore asked for clarifications.
so anyone who wants to do a more representative test feel free to simply do it, i will surely help where i can again, please don't take it personally. it's just that the reply posts you get now, are of the kind that should had been posted prior to the test (if a discussion thread existed then, announcing an upcoming test).

[bond]

Originally posted by DSPguru
it just that i'm afraid that some users, whom english isn't that sharP, might understand things differently, and i wanted to avoid that - therefore asked for clarifications.

again, please don't take it personally.i dont :)
productive criticism is always welcome, but guys, dont overestimate this test!
as i said i did it to find out what tool would be the best for my task and only after that posted the results/method aso, not the other way round, meaning a pre-test discussion would have been some sort of overkill, as the test wasnt made with the goal to be able to make an overall conclusion on which tools are better than others

still some people might find it useful as a guideline, "better than nothing", thats why i posted it :)