MutliLingual Vorbis (imported from vorbis-dev@xiph.org)

[Jean-Marc]

Hi,


We would like to announce the first release of the Speex project. Speex
(http://speex.sourceforge.net) is an open-source (LGPL), patent-free
compression format allowing an alternative to expensive proprietary
codecs. Unlike Ogg Vorbis which compresses general audio, Speex is
designed especially for speech. For that reason, Speex is meant to be a
complement to Vorbis. Since it is specialized for voice communications,
it is possible to attain lower (compared to Ogg Vorbis/MP3) bit-rates in
the 8-32 kbps/channel range. Possible applications include Voice over IP
(VoIP) applications, Internet audio streaming at low bit-rate and
archiving of speech data (e.g. voice mail).


This first version of Speex supports fixed bit-rate CELP coding at 14.5
kbps for speech sampled at 8 kHz (narrowband) and at 28.5 kbps for 16
kHz (wideband) speech. Future releases will likely provide a wider
choice of bit-rates, better quality, as well as variable bit-rate (VBR)
and discontinuous transmission (DTX).


Disclaimer:


Note that this is a preliminary release and that bit-rates, quality and
bitstream definition will change in the future. This means that the
format used in this release will not be compatible with future releases.
Also note that this software has so far received only a minimum amount
of testing, so it may break or do unexpected things.

[DSPguru]

Greetings developers, Jean-Marc,


Great to hear about your work. i find speech encoding pretty useful.


these days we think about MultiLingual vorbis defenitions, and the best
compression we could acheive were if we could save the common track on
two channels, and then save the delta between the common track to the
specifiec language (english speech/italian speech/etc'..) on a low
bandwidth track. a vocoder track for this task could be very
interesting. don't you think ?


another point,
I wanted to ask why you had decided to develop CELP, which is based on
linear prediction (LPC), while there are multi-band excitation (MBE)
speech coders (IMBE/AMBE) that have better compression-gain (down to 2kbps),
and consumes less MIPS. i heard some samples of DVSI's vocoders, and i
must admit that their quality is very very good.
LPC is abit old, isn't it ?-)



Good luck with your work !
Dg. http://DSPguru.doom9.net

[Jean-Marc]

> these days we think about MultiLingual vorbis defenitions, and the best
> compression we could acheive were if we could save the common track on
> two channels, and then save the delta between the common track to the
> specifiec language (english speech/italian speech/etc'..) on a low
> bandwidth track. a vocoder track for this task could be very
> interesting. don't you think ?


Probably, what kind of bandwidth (sampling rate) and bit-rate are you
thinking about?


> another point,
> I wanted to ask why you had decided to develop CELP, which is based on
> linear prediction (LPC), while there are multi-band excitation (MBE)
> speech coders (IMBE/AMBE) that have better compression-gain (down to 2kbps),
> and consumes less MIPS. i heard some samples of DVSI's vocoders, and i
> must admit that their quality is very very good.
> LPC is abit old, isn't it ?-)


Well CELP (mostly CELP variants) is still the most widely used technique
for speech coding. If you look at the latest standards (like G.729 and
AMR wideband), most use ACELP (which we cannot use because of patents,
but that's another thing). Note that CELP has nothing to do with LPC
vocoders (aside from the fact that is uses LPC analysis). There are
other techniques, like WI (waveform interpolation) and sinusoidal coding
(not sure about MBE), but most of them are for low bit-rate coding (in
the 4 kbps range). Our current goals focus more on high quality that low
bit-rate.


That being said, the Speex is an open-source project and the code is
designed so that it's really easy to change any part of the codec (LSP
quantization, pitch, fixed codebooks, ...). So you can play with it if
you like and if you find something that works well, let us know.


Jean-Marc

[DSPguru]

Hi Jean-Marc,


> > these days we think about MultiLingual vorbis defenitions, and the best
> > compression we could acheive were if we could save the common track on
> > two channels, and then save the delta between the common track to the
> > specifiec language (english speech/italian speech/etc'..) on a low
> > bandwidth track. a vocoder track for this task could be very
> > interesting. don't you think ?
>
> Probably, what kind of bandwidth (sampling rate) and bit-rate are you
> thinking about?


sampling-rate should be high (48khz), but bandwidth should be less than
16khz (after "extracting" speech-only from the lingual track).


about bitrate, let me describe something :
up until vorbis came, people used to encode their soundtrack of movies
at 128kbps to 192kbps MP3. now, with Ogg, we can encode the "common"
track at around 100kbps vorbis, and encode each speech track at less
than 30kbps with speex. this gives us about 180kbps for a movie with
three soundtracks (english/italian/francis, for instance).
that could make a small revolution :).



> Well CELP (mostly CELP variants) is still the most widely used technique
> for speech coding. If you look at the latest standards (like G.729 and
> AMR wideband), most use ACELP (which we cannot use because of patents,
> but that's another thing). Note that CELP has nothing to do with LPC
> vocoders (aside from the fact that is uses LPC analysis). There are
> other techniques, like WI (waveform interpolation) and sinusoidal coding
> (not sure about MBE), but most of them are for low bit-rate coding (in
> the 4 kbps range). Our current goals focus more on high quality that low
> bit-rate.


you're right, VoIP applications uses variations of CELP. AMBE (& MELP)
vocoders are mostly used in military applications (Digital voice over HF).
but in some way, MBE complements Speex, the same way Speex complements vorbis :).
anyway, it was only a thought..


just to mention, that it been some years since i studied vocoders
principles, but i still remember (correct me if i'm wrong here :>),
that CELP bitstream mostly (i believe that in LD-CELP it's even ONLY)
includes codes to exciters codebook, but both the encoder and decoder
includes LPC analysis.


you can find some info about MBE over at :
http://www.dvsinc.com/papers/mbe.htm


these days, some people are testing the difference between tracks within mutlilingual movies.
results would be published here :
http://forum.doom9.org/forumdisplay.php?s=&forumid=11


>
> That being said, the Speex is an open-source project and the code is
> designed so that it's really easy to change any part of the codec (LSP
> quantization, pitch, fixed codebooks, ...). So you can play with it if
> you like and if you find something that works well, let us know.
>
> Jean-Marc


10x,
Dg. http://DSPguru.doom9.net

[Jean-Marc]

> sampling-rate should be high (48khz), but bandwidth should be less than
> 16khz (after "extracting" speech-only from the lingual track).


Currently, Speex only supports sampling at 8 kHz and 16 kHz, so it would
need to be adapted to work at 32 kHz (and then up-sample to 48 kHz). I'd
say it's quite feasible.


> about bitrate, let me describe something :
> up until vorbis came, people used to encode their soundtrack of movies
> at 128kbps to 192kbps MP3. now, with Ogg, we can encode the "common"
> track at around 100kbps vorbis, and encode each speech track at less
> than 30kbps with speex. this gives us about 180kbps for a movie with
> three soundtracks (english/italian/francis, for instance).
> that could make a small revolution :).


I think 30 kbps is realistic. When we add VBR, the average could easily
drop to ~16 kbps/track.


> you can find some info about MBE over at :
> http://www.dvsinc.com/papers/mbe.htm


This info seems very biased to me...


So I'd say the first step would be to build a prototype that downsamples
the 48 kHz stream to 16 kHz and encodes it with the current Speex
version. Once that works, we can try making Speex work at 32/48 kHz.
Actually, that *might* not even be necessary, as most of the energy in
speech is in the 0-8 kHz band - and even the 4-8 kHz band can in some
cases (speech only) be severely distorted before the ear can tell the
difference.


Jean-Marc

[DSPguru]

Hi Jean-Marc,


> > sampling-rate should be high (48khz), but bandwidth should be less than
> > 16khz (after "extracting" speech-only from the lingual track).
>
> Currently, Speex only supports sampling at 8 kHz and 16 kHz, so it would
> need to be adapted to work at 32 kHz (and then up-sample to 48 kHz). I'd
> say it's quite feasible.


Speex shouldn't bother dealing with non-standard speech sampling-rates.
Encoding tools like mine would downsample the signal before delivering
it to Speex, and decoding tools like Tobias' DirectShowFilter should
take care of the upsampling and summing the different tracks (common + speech).
There's a brilliant, open-source, HQ sample-rate convertor, called
SSRC. it's under LPGL, and i even made a dll release of this fine tool.


>
> > about bitrate, let me describe something :
> > up until vorbis came, people used to encode their soundtrack of movies
> > at 128kbps to 192kbps MP3. now, with Ogg, we can encode the "common"
> > track at around 100kbps vorbis, and encode each speech track at less
> > than 30kbps with speex. this gives us about 180kbps for a movie with
> > three soundtracks (english/italian/francis, for instance).
> > that could make a small revolution :).
>
> I think 30 kbps is realistic. When we add VBR, the average could easily
> drop to ~16 kbps/track.


sure thing! the speech track suppose to have lots of silent moments, so
DTX (AD/CFI) would help to drop the bitrate.
problem is - can CELP handle multiple spokesmen (ie, when two ppl are
talking at the same time), and will sound quality differ when compressing
English track compared to encoding Russian track ?


>
> > you can find some info about MBE over at :
> > http://www.dvsinc.com/papers/mbe.htm
>
> This info seems very biased to me...


most probably.
still, i know of some satelite applications where AMBE is succesfuly
used. i also know of a few projects where MELP is used over HF.


>
> So I'd say the first step would be to build a prototype that downsamples
> the 48 kHz stream to 16 kHz and encodes it with the current Speex
> version. Once that works, we can try making Speex work at 32/48 kHz.
> Actually, that *might* not even be necessary, as most of the energy in
> speech is in the 0-8 kHz band - and even the 4-8 kHz band can in some
> cases (speech only) be severely distorted before the ear can tell the
> difference.


the first step is :
- decide how we extract the 'common' track
- define Speex integration in ogg
- start testing - taking a multilingual title, creating the 'common'
track, downsampling the 'lingual' tracks using ssrc.dll and muxing
everything to ogg stream. then doing the reversed process.., and
comparing quality.



Jean-Marc,
you have a lot of knowledge regarding speech models. can you point out
some useful sites/tools which i should check in order to implement the
first stage of 'extracting the common track' ?


keep in mind that i should take advantage of the fact that i have
multiple soundtracks that mostly (only?) differs in the speech content.


Best Regards,
Dg. http://DSPguru.doom9.net

[Jean-Marc]

> Speex shouldn't bother dealing with non-standard speech sampling-rates.
> Encoding tools like mine would downsample the signal before delivering
> it to Speex, and decoding tools like Tobias' DirectShowFilter should
> take care of the upsampling and summing the different tracks (common + speech).
> There's a brilliant, open-source, HQ sample-rate convertor, called
> SSRC. it's under LPGL, and i even made a dll release of this fine tool.


Well, it all depends on whether 8 kHz bandwidth (16 kHz sampling) is OK
for you...


> sure thing! the speech track suppose to have lots of silent moments, so
> DTX (AD/CFI) would help to drop the bitrate.
> problem is - can CELP handle multiple spokesmen (ie, when two ppl are
> talking at the same time), and will sound quality differ when compressing English track compared to encoding Russian track ?


The only potential problem is the one where two people are talking at
the same time. In this case, the solution could be to just boost the
bit-rate for a couple frames.


> the first step is :
> - decide how we extract the 'common' track


I'll leave that one to you. I have no idea about the properties of the
different tracks.


> - define Speex integration in ogg


Could anyone tell me how to do that?


> Jean-Marc,
> you have a lot of knowledge regarding speech models. can you point out
> some useful sites/tools which i should check in order to implement the
> first stage of 'extracting the common track' ?


Sorry, I have no idea. I guess it first depends on whether that "common"
part is *exactly* the same in all tracks (ie english track - common
track = only english and nothing else).


Jean-Marc

[DSPguru]

Hi Jean-Marc,

> The only potential problem is the one where two people are talking at
> the same time. In this case, the solution could be to just boost the
> bit-rate for a couple frames.

are you sure this will give us good results for both voiced and unvoiced sounds ?
i'm asking because this could raise an alternative solution.
assuming with have two speech signals, s1,s2. if Speex can encode
s1+s2, it should be able to encode s1-s2, as well. right ?
in this case, our "common" track could be the original english (s1)
soundtrack encoded at ~100kbps vorbis, and the german track (s2) would
be encoded as speex of s1-s2 at ~30kbps.
that way we have better compression-gain, and better quality for the
default soundtrack.
Beware, developers, we would need to define some comments strategy to
the ogg, to be able to hold info about languages.
so in the future, each user could choose its prefered langauge, and the
player should be able to supply it by default (if available).


>
> > the first step is :
> > - decide how we extract the 'common' track
>
> I'll leave that one to you. I have no idea about the properties of the
> different tracks.

ok. maybe anyone else in the list have ideas ?


Best Regards,
Dg. http://DSPguru.doom9.net

[Jean-Marc]

> > The only potential problem is the one where two people are talking at
> > the same time. In this case, the solution could be to just boost the
> > bit-rate for a couple frames.
>
> are you sure this will give us good results for both voiced and unvoiced sounds ?

I believe unvoiced sounds won't be a problem, but voiced sounds could
(especially two simultaneous vowels at different pitch and very
different LPC), but increasing bit-rate could still work.

> i'm asking because this could raise an alternative solution.
> assuming with have two speech signals, s1,s2. if Speex can encode
> s1+s2, it should be able to encode s1-s2, as well. right ?
> in this case, our "common" track could be the original english (s1)
> soundtrack encoded at ~100kbps vorbis, and the german track (s2) would
> be encoded as speex of s1-s2.

I don't think this would work. Encoding one track with occasional
double-talk would make sense (even if we need to triple bit-rate for the
few double-talk instances), but continuous double-talk would cause too
much problems.

Plus there's another, more important problem. Consider s1 (english
track) is encoded with Vorbis and s2-s1 is encoded with Speex. Unless
the compression is lossless, adding the two signals back won't remove
the english track completely (because the Speex and Vorbis won't have
the same "error signal").

> > > the first step is :
> > > - decide how we extract the 'common' track
> >
> > I'll leave that one to you. I have no idea about the properties of the
> > different tracks.
>
> ok. maybe anyone else in the list have ideas ?

Just a thought: ICA (Independent Component Analysis) might be able to do
it. Not sure whether it's good enough though. It has to do a perfect job
if you don't want to end up with the problem I described above. So I'm
not sure whether the project is feasible at all without access to the
"original" common track.

Jean-Marc

[DSPguru]

> I believe unvoiced sounds won't be a problem, but voiced sounds could
> (especially two simultaneous vowels at different pitch and very
> different LPC), but increasing bit-rate could still work.

that's exactly what i was afraid of. two voiced sounds with different
pitches.. :(
we need to test Speex behavior under this.

> I don't think this would work. Encoding one track with occasional
> double-talk would make sense (even if we need to triple bit-rate for the
> few double-talk instances), but continuous double-talk would cause too
> much problems.

okay, back to original plan.
btw, when you say "triple bit-rate", how much would that be.. ?


> Just a thought: ICA (Independent Component Analysis) might be able to do
> it. Not sure whether it's good enough though. It has to do a perfect job
> if you don't want to end up with the problem I described above. So I'm
> not sure whether the project is feasible at all without access to the
> "original" common track.

there's a lot we can do.
we should take advantage of the following facts :
- we have several soundtracks that mostly differ in the speech, or in
other words (generally speaking) : spectral view mostly differs in 3 to
6 formants.
- BEFORE we downmix the 6ch (5.1) soundtrack to 2ch, we can focus our
analysis on the _center_ channel on each soundtrack, which its nature
is speech content.

above all that, we should try to run ICA, kareoke, echo cancelling, etc'..


Best Regards,
Dg. http://DSPguru.doom9.net

[MaTTeR]

Doom9 made a statement today that got me thinking. He said that some movies don't necessarily have all dialog on the center channel. I went and pulled some random DVD's and cranked the volume up on my sub. Sure enough, some people's voice's that were deep bled through to both the sub and L/R.

So I guess I'm wondering how the dialog/vocals on movies like this would be handled. How would an encoder application be smart enough to extract the vocals from multiple channels?

Beware, developers, we would need to define some comments strategy to the ogg, to be able to hold info about languages.
so in the future, each user could choose its prefered langauge, and the player should be able to supply it by default (if available).
This could be quite tricky I think, if I'm understanding you correctly. AFAIK the comments tags in Ogg are only human readable.

I don't really see a major obstacle in the way if you wanted to take a Speex track and multiplex it into an Ogg with another Vorbis file. However, are current players and/or filters capable of playing to audio streams simultaneously? Furthermore, would this require modifying the Ogg Multiplex filter as well to except a Speex stream on one of the pins?

I'm just thinking aloud, hope I'm to far off base here:) This a very interesting idea you guys are tossing around, I'm hoping it will materialize.

[TheHeap]

Hi there..
first Thanks Jean-Marc for your efforts.
second: @dspGuru I think some of your calculations are wrong.
How much bitrate do you (Jean-Marc) think a 5.1 channel speex will take (considering that most times it will only contain 2 or 3 channels) ?
From what i have already heared in this thread it will be around lets say. 60k(80k) ??.(2*30 or 3*20 ...) This would make the movie have 100k vorbis + 60k (80K) for each language.
Let's compare this to standard vorbis encoding:
(I don't have that much experience with vorbis, so correct me if my numbers are wrong)
1. One Language full vorbis track = 160 k
2. one common vorbis + one language speex track = 160k (180k)
3. Two language full vorbis track = 320k
4. one common vorbis + two language speex track = 220k (260k)


that makes a difference of 100(60) kbps.
Indeed its better BUT only if speex can do 5.1 speech at lower then 100/80 k.
The speex codec maybe a good solution for our problem but it aint a wonder anyway.(->NO 180k with 3 languages)

gr33tz

TheHeap

[Antimon]

If you will pardon interjection from somone enterly lacking your qualifications.


Would not such a low samplerate file compared/combined with a higher samplerate fiel not only cus mathamaticle problems and difficulty there but also how would you maintian proper sync? I cna;t get a 22 khz mp3 and a 48 kh/z ac3 to coexist happily as 2 multiplexed independent streams, how woudl you get them locked together for comparison subroutines?

ALso somthign striked me as curiose, you were talking about havignthe main track be higher bitrate with the main laugage and the secondlaungage as a lower bitrate you cold then overlay into the main...if you do that the question becomes not how do you add this new information...but how do you remove the main laugage form yoru bitstream?

It would have to be soem sort of kareoke type filter which they have come a logn way but it would still suffer.

A solution would be to have a comon track with NO langage *near impossible from an encoding point of view* and then 2 laungage tracks where one or the other would be mixed in based on user choice, which starts introducing some pretty good overhead inless you could interface with the multi chanal dsp's of soundcards much like games do prolly through direct x drivers i'd sumize.


I also question how much time this new format would live...how long it would be nessicary, cus racing along with yoru possible develpment you have the 99 minute cdr, you have the popularity of dvd-r increasing slowly but surely, and you have every increasing hard drives where burning the movie to cd becomes mroe about back up then cleanign space off the drive to make room for suture projects.

Also there is a movement marching twords mp4 divx/xvid streams muxed with aac/oog/mp3/ac3 compliant devices to allow playback off disk on a set top dvd player. This is a step backward from that requierign a pc to play for the forseable future.

As a proof of concept and programing problem solvign thsi is very interestign and should be persued from a challenge point of view for sure. I have seriose reservations about the practicality of such a feature and design though.

It seems it would be an imposable comprise of audio quality to save possible signifigant space....to coverign all your basses and ahvign flawless audio but at the expence of the very bits you're tryrign to trim off to the point of negative returns on time investment and ocompatability.

Storage sapce is gettign larger whiel compresion codecs get better.

Does a technology such as this a kind of interactive multistream file have much of a future?

espeially since you could get to 80 kbs ogg full stream effects and dialog and stll only be up to 160 kb/s with 2 full laungags and mp3 128+ quality, where you are asured no cross bleading and no processing penalty, where your proposale so you have an 80 kb/s "main" chanal with just effects and music and 2 30 kb/s speach files, you're up to 140..... you've saved 20 kb/s which isn;t that small, but nothing thst going to make a dramatic difernce in an encode, and the penalty has been multichanal speach down to mono center only speach....is it worth it?

To me, it would not be, to others maybe it would, i'm not sure.

[Antimon]

Somthign else that was raised in the other thread and one mor ethign i just thought of reading some aditional posts here are these.

Many times the langage tracks are not identicle in type bitrate nor content.

Say for instance the main laungage is 448 kb's 5.1 and the 2nd laungage is as it so often is a 224 k dolby suround 2 channal mix. Now oyu have no dedicated "center" chanal to base your calculations under so you'd get mroe cross bleeding.

Also AS you pointed out dialog ia mostly dedicated to the center chanal, but the reverse is not true, the center is not dedicated msotly to speach. The center is the most used track of any including music screen effects paning information across tracks and laungage, it's quite dence and removing the voice only would be quite difficult to do cleanly...you'd ether clip into teh speach or blead the effects in. Then you have the matter of ocne you extract the speach what becomes of all the non speach material disgarded, will that be recovered and mixed into the left/right chanals...will you need a common center as well now, making it enstead 2 tracks plus 2 speach, 3 tracks plus 2 speach?

you would also have to be very careful about how they are mixed in that you dotn get any phase cancelation or amplification of any redundent information during the suming

[TheHeap]

Antimon
i think u are right.
This codec may be fine for something like Voice over Ip but for our needs the implementation would be too difficult, and resulting in only small advantages over vorbis.
I think the devs should concentrate on making vorbis/xvid burnable on an MCF-like cd giving us a lot more bitrate for additional quality or additional languages.
My hope goes to a hq dvd-rip with two 5.1 language streams on an 90 or 99 minute cd burned in mode 2.



gr33tz

TheHeap

p.s. antimon.. when addressing people you should prefer saying his name at least once instead of only using you *G*. just to make things clear.

[Antimon]

this is true :-)
i'm most tired, plus i figured people remeber when you're adressign a point they made :-)

Well if you thinka bout it we're already craming 4.7 gb of data onto a 700 meg cd, not identicle, but close, and i think thats damn impressive considerign where we came from in terms of audio and video compresion.

I'm happy where the development is now, if it gets better so much the better but you have to keep in mind too as our outpt devices become higher quality our low bitrae cheats will no longer be aceptable.

I dont think a 80 kb/s oog and 512 divx will look so great on a 1080I 60" tv and a 7.1 hifi system :-)

Which is where dvd-r enters the picture, this si still several years off mind you but time flies.

I agree with you Heap :-) that more tiem should be put into the mcf cd and improving existing technology more then coming up with new methods though there are plenty of programers to go around and i think this is a cool hobby project just to say it can be done and prove it. But i dont see it havigna futue in main stream "backing up"