QuEnc 0.54 released

[dragongodz]

this is the official QuEnc 0.54 release.

its released under the GPL so if you dont like that... dont use it. :devil:

changes 0.52 -
.some more tweaking to settings.
.added pass text on main window.
.notch matrix completely replaced by QLB(Quenc Lower Bitrate) matrix.
.the kvcd cli commands replaced with QLB ones.

changes 0.53 -
.properly replace notch matrix.
.minor tweaks.

changes 0.54 -
.aspect ratio should be working properly now ;)
.bad parameter window tells you what the bad one was now
.version of exe changed to 0,0,5,4 so jdobbs can pick it in DVD-RB :)
.dll version is cvs 12th july as reccomended over 0.49pre1, found on ffmpeg homepage

.QLB commands changed to cmatrix and nocmatrix (custom matrix) in the -? but not actual command line yet.

0.54 is compatible with DVD-RB 0.57.

i do ask that if you do any modified releases please identify them as such and not up QuEncs number. meaning please dont change a few things and release QuEnc 0.55 etc. i got permission from Nic to bump the version and it would save a lot of confusion if others didnt do so without asking him aswell first.
also if you change the matrix please say it in your release. one of the reasons notch matrix was removed was because it has big problems with trellis being used. i have no interest in answering questions about problems caused by other peoples changes.

i put a text in with the source giving a quick guide on how to compile it. Nic has already warned before that neither he nor i will have the time to go in to detail on compiling it but that should get you well started.

neither Nic or i will be working on this for awhile since avcodec really needs the rate control fixed for it to work 100% properly and we do have other things we want to do.

downloads found at
http://www.pcpages.com/dragongodz2/

mirrors are of course welcome. :D

[Doom9]

here's another download link: http://www.doom9.org/Soft21/Encoders/MPEG2/QuEnc052.zip

[glänzend]

notch matrix completely replaced by QLB(Quenc Lower Bitrate) matrix.

Please correct me if I'm wrong, but after looking at a small encoded clip we just made with your new version of QuEnc, we confirm that the resulting file was created with the KVCD notch matrix tables. :confused: This was also re confirmed by looking at YOUR source code. :confused: :confused:


The following code segment shows clearly the KVCD notch matrix tables:


// dragongodz - brand new custom lower bitrate matrix especially for
QuEnc :)
if ( bUseQLB )
{
CString csQLBIntra = CString("8 9 12 22 26 27 29 34 ")
+("9 10 14 26 27 29 34 37 ")
+("12 14 18 27 29 34 37 38 ")
+("22 26 27 31 36 37 38 40 ")
+("26 27 29 36 39 38 40 48 ")
+("27 29 34 37 38 40 48 58 ")
+("29 34 37 38 40 48 58 69 ")
+("34 37 38 40 48 58 69 79 ");

CString csQLBNonIntra=CString("16 18 20 22 24 26 28 30 ")
+("18 20 22 24 26 28 30 32 ")
+("20 22 24 26 28 30 32 34 ")
+("22 24 26 30 32 32 34 36 ")
+("24 26 28 32 34 34 36 38 ")
+("26 28 30 32 34 36 38 40 ")
+("28 30 32 34 36 38 42 42 ")
+("30 32 34 36 38 40 42 44 ");


pIntraMatrix = (unsigned short*) av_malloc(1000);
pNonIntraMatrix = (unsigned short*) av_malloc(1000);

parse_matrix_coeffs(pIntraMatrix, csQLBIntra);
parse_matrix_coeffs(pNonIntraMatrix, csQLBNonIntra);

c->intra_matrix = pIntraMatrix;
c->inter_matrix = pNonIntraMatrix;
}

Please either give the proper credit due, (there is no problem with that? is there? ;) ) or just remove it and create your own.
ciao
glänzend

[audioslave]

Yep, glänzend, you're right. The so called QLB Matrix IS the KVCD Notch Matrix. :rolleyes:

glänzend wrote:
Please either give the proper credit due, (there is no problem with that? is there? ) or just remove it and create your own.

I totally agree!

Other than that... Thanks for another great release of QuEnc! :)

[dragongodz]

jeez and why dont you guys try giving some credit in return ?

it was an honest mistake when i was cleaning it all up for a clean release. that is i went back over the 0.51 files and made the changes and the matrix accidently didnt get replaced.

i have no interest in taking credit or "stealing" the notch matrix. the fact the source code is there for anyone to check and the fact i have remedied the problem as soon as i have found out(its morning here, couldnt do it in my sleep :) ) should be the proof anyone needs.

[Paced]

Thanks for the new release + hard work dragongodz / Nic, I'm sure it's appreciated by many.

[dragongodz]

thanks Paced. its nice to get a thank you every now and then. :)

Nics working on mpegdecoder.dll(in avisynth forum) and i am working on some other things. so this is the last version for awhile.

[dragongodz]

a quick note about first frames and aspect ratio.

using MediaPlayerClassic's internal mpeg2 decoding aspect ratio is show correct. first frames with high quant(produced by 1 pass and not 2 pass) can show some blocks.
using intervideos mpeg2 decoding through MPC however the aspect ratio was not shown(did 1:1 no matter what aspect the video was) but no blocks for those first frames were obvious either.
using Windvd to play the files had aspect ratio shown correctly and showed no obvious blocks.

fun with pcs. :D

[vmesquita]

About the "keeps increasing the memory usage till there's no memory left yet" thing we were discussing at the FreeEnc thread.
I tried with no filters (just resizing) and still the same result:

Avisource("movie.avi",false)
bicubicresize(704,480)

It's really funny how it behaves differently in different machines... Same thing with FreeEnc. :D

[dragongodz]

hmm ok could you try the same using directshowsource() and see if that does it aswell ?

actually i do know there is 1 problem with using avisource() and QuEnc, no 2 pass encoding. first pass works but second pass gives an error(no yv12 data being sent or such).

hmm i did also turn on the mem align hack for that dll compile so its possible that could have an effect. it shouldnt really as its actually meant to be good for windows.

[vmesquita]

Hi dragongodz,

I tried with DirectShowSource and a very interesting thing happened. I use ffdshow, and it puts that small icon on the taskbar. Well, many icons started popping up, like if QuEnc were opening the scripts in many instances. :confused: And of course the memory usage increased, and increased...
I have a Windows2000 VMWare virtual machine that I set up with only the basic stuff to test FreeEnc crashes, I'll give it a try and let you know.

EDIT:
It worked fine in my w2000 virtual machine. It must be something wrong here.
Also it's good to know you used the hack, I am confident that will fix the crashes in FreeEnc. :D

[dragongodz]

wow that is strange. QuEnc only opens the avs once but maybe they are popping up with each get of a new frame. that sounds more like an avisynth or ffdshow problem and since its ffdshow that keeps reappearing i suspect thats the problem. maybe trying a different version would fix that or changing some of its settings. actually from memory it does have a setting to allow only 1 instance of ffdshow in the codecs settings(thats ffdshows codecs settings and not windows).

as for the mem align hack well i wouldnt count on that fixing the problem with FreeEnc and P4's since Nic didnt use it and from what you said his 0.51 dll works with FreeEnc. worth a try though. :)

[vmesquita]

Originally posted by dragongodz
wow that is strange. QuEnc only opens the avs once but maybe they are popping up with each get of a new frame. that sounds more like an avisynth or ffdshow problem and since its ffdshow that keeps reappearing i suspect thats the problem.

Dont forget that the same problem happens with AVISource, so it's not related to FFDShow. But it maybe avisynth. (I am using 2.54) Looks like something is being loaded again and again in every frame. I still could not test in my machine at work to report back.


maybe trying a different version would fix that or changing some of its settings. actually from memory it does have a setting to allow only 1 instance of ffdshow in the codecs settings(thats ffdshows codecs settings and not windows).]

I'll look for it.


as for the mem align hack well i wouldnt count on that fixing the problem with FreeEnc and P4's since Nic didnt use it and from what you said his 0.51 dll works with FreeEnc. worth a try though. :)
Yes but:
1) QuEnc avcodec parameters are different from FreeEnc default parameters. Maybe some parameters trigger the problem.
2) QuEnc was done in VC++, while FreeEnc was done in MinGW+WxWindows. It shouldn't make difference since the code that crashes is inside the DLL, but...

So far, Peter1234 (one of the happy P4 owners that reported the problem) has had no crashes. I'll keep my fingers crossed.:D

[dragongodz]

same problem happens with AVISource, so it's not related to FFDShow
depends if ffdshow is still the decoder. ok the taskbar icon may not show up but that doesnt completly rule it out as being used. when you change the "only allow 1 instance" in ffdshows settings can you try both ways again ?

as for the mem align hack, well all i was saying is that you have an example of a dll that doesnt use it that works(Nics). so if it then does work with it in your compile then it is probably because of something the ffmpeg guys have changed since Nics compile. the only way to know though is to test that since as you point out QuEnc and FreeEnc are not done the same. :)

[lamster]

QuEnc 0.53 doesn't work with Rebuilder 0.56, since QuEnc dropped support for the -kvcd option, and Rebuilder always passes either "-kvcd" or "-nokvcd". Either of these options results in a pop-up saying "Bad Parameter!"

A couple of suggestions, for whenever this gets updated again:

First, if you're going to complain about a bad parameter, it would be nice if the pop-up specified which one was bad.

Second, if you're dropping a parameter, it might be nice to place it on a separate list, and if the options "-auto -close" are also present, just display a warning and otherwise ignore it, so that people invoking QuEnc via a front-end program won't be left dead in the water.

Thanks.

[dragongodz]

QuEnc 0.53 doesn't work with Rebuilder 0.56
yes and jdobbs has already said that will be fixed with the next version of DVD-Rebuilder.

whenever this gets updated again
wont be for awhile as both Nic and i have other things we are doing. your suggestions are noted though.

[Trahald]

I would also like to thank you for working on this software.

[EpheMeroN]

I've been testing QuEnc v0.53 on a 4:3 source that I captured at 560x420 resolution and am trying to convert it via AviSynth and QuEnc to a 1/2 D1 DVD Compliant MPEG-2 Video. I'm also trying to keep it intact as interlaced for it's pure NTSC 29.97fps and has no FILM content whatsoever.

The quality is fine, but the AR is off. If I choose 4:3 as the output Aspect Ratio QuEnc randomly will spill out either 1:1 Square AR, or 16:9 AR. Why I have no idea. I did not have this issue w/ QuEnc v0.51. Any ideas?

Here's the script I'm using for the video:

LoadPlugin("C:\Video Apps\AviSynth v2.5.4\Plugins\SubtitleEx.dll")
LoadPlugin("C:\Video Apps\AviSynth v2.5.4\Plugins\PeachSmoother.dll")
LoadPlugin("C:\Video Apps\AviSynth v2.5.4\Plugins\UnDot.dll")
LoadPlugin("C:\Video Apps\AviSynth v2.5.4\Plugins\UnFilter.dll")
LoadPlugin("C:\Video Apps\AviSynth v2.5.4\Plugins\SimpleResize.dll")

Clip_01 = Trim(AVISource("C:\Windows\Desktop\noname76.avi"),388,824).FadeIn(25).FadeOut(12)
Clip_02 = Trim(AVISource("C:\Windows\Desktop\noname77.avi"),0,3536).FadeIn(25).FadeOut(12)
Clip_03 = Trim(AVISource("C:\Windows\Desktop\noname78.avi"),0,5745).FadeIn(25).FadeOut(45)
Blank_1 = BlankClip(Clip_01,length=60,color=$000000)
AlignedSplice(Blank_1,Clip_01,Blank_1,Clip_02,Blank_1,Clip_03,Blank_1)

DelayAudio(-0.2)
SSRC(48000)

AssumeTFF()
SeparateFields()
PeachSmoother(NoiseReduction=52,Stability=25,Spatial=60)
UnDot()
UnFilter(15,15)
Weave()
InterlacedResize(352,480)
ConvertToYV12(Interlaced=True)

[jdobbs]

Originally posted by EpheMeroN
The quality is fine, but the AR is off. If I choose 4:3 as the output Aspect Ratio QuEnc randomly will spill out either 1:1 Square AR, or 16:9 AR. Why I have no idea. I did not have this issue w/ QuEnc v0.51. Any ideas?
I'm having the same problem. My 4:3 sources are coming out 16:9. It seems to have started recently, as I believe 0.51 was working.

[p200002]

Originally posted by dragongodz
yes and jdobbs has already said that will be fixed with the next version of DVD-Rebuilder.


wont be for awhile as both Nic and i have other things we are doing. your suggestions are noted though.

Which one will you suggest to downgrade? dvd-rb to .55 or QuEnc to .52?

[lamster]

Originally posted by p200002
Which one will you suggest to downgrade? dvd-rb to .55 or QuEnc to .52?

Well, personally, I'd suggest patching Rebuilder as I described in the Rebuilder bugs forum http://forum.doom9.org/showthread.php?s=&threadid=73050 (posted August 13) rather than downgrading either...

[p200002]

Can you email me the hacked rb? I have no experience editting hex.

[dragongodz]

hmm with 720x756 it works fine. just tried half d1 and ntsc and yep its wrong.

hmm visual studio has been acting wierd lately so i think i will uninstall everything and reinstall and see if that makes any difference.

i am changing isp, today is the last day with my current one. so i will not be online for a week or two. that should give me some time to try and sort this out. :)

[Peter Cheat]

I was trying to compile the source for QuEnc 0.53 but it came up with a tonne of errors (from the ffmpeg headers). I can easily fix the invalid type errors (long followed by long), but the others I don't know how to fix. This is the compiler output:


Compiling...
AVSEnc.cpp
C:\QuEnc\config.h(11) : warning C4005: 'CONFIG_WIN32' : macro redefinition
C:\QuEnc\common.h(10) : see previous definition of 'CONFIG_WIN32'
C:\QuEnc\bswap.h(16) : error C2400: inline assembler syntax error in
'opcode'; found '('
C:\QuEnc\bswap.h(17) : error C2064: term does not evaluate to a function
C:\QuEnc\bswap.h(17) : error C2143: syntax error : missing ';' before ':'
C:\QuEnc\bswap.h(18) : error C2059: syntax error : ')'
C:\QuEnc\bswap.h(26) : error C2400: inline assembler syntax error in
'opcode'; found 'no token'
C:\QuEnc\bswap.h(27) : error C2064: term does not evaluate to a function
C:\QuEnc\bswap.h(27) : error C2143: syntax error : missing ';' before ':'
C:\QuEnc\bswap.h(34) : error C2059: syntax error : ')'
C:\QuEnc\bswap.h(39) : error C2632: 'long' followed by 'long' is illegal
C:\QuEnc\bswap.h(39) : error C2632: 'long' followed by 'long' is illegal
C:\QuEnc\bswap.h(41) : error C2146: syntax error : missing ';' before
identifier 'uint64_t'
C:\QuEnc\bswap.h(41) : error C2501: '__extension__' : missing
storage-class or type specifiers
C:\QuEnc\bswap.h(43) : error C2143: syntax error : missing ')' before ':'
C:\QuEnc\bswap.h(43) : error C2290: C++ 'asm' syntax ignored. Use __asm.
C:\QuEnc\bswap.h(45) : error C2059: syntax error : ')'
C:\QuEnc\bswap.h(46) : warning C4244: 'return' : conversion from
'unsigned __int64' to 'unsigned long', possible loss of data
C:\QuEnc\common.h(225) : warning C4005: 'NDEBUG' : macro redefinition
unknown(0) : see previous definition of 'NDEBUG'
C:\QuEnc\common.h(239) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(239) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(239) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(239) : error C2010: ')' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(278) : error C2143: syntax error : missing ')' before ':'
C:\QuEnc\common.h(278) : error C2290: C++ 'asm' syntax ignored. Use __asm.
C:\QuEnc\common.h(280) : error C2059: syntax error : ')'
C:\QuEnc\common.h(285) : error C2143: syntax error : missing ')' before ':'
C:\QuEnc\common.h(285) : error C2290: C++ 'asm' syntax ignored. Use __asm.
C:\QuEnc\common.h(287) : error C2059: syntax error : ')'
C:\QuEnc\common.h(800) : warning C4146: unary minus operator applied to
unsigned type, result still unsigned
C:\QuEnc\common.h(1077) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(1077) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(1077) : error C2010: '.' : unexpected in macro formal
parameter list
C:\QuEnc\common.h(1256) : error C2632: 'long' followed by 'long' is illegal
C:\QuEnc\common.h(1258) : error C2632: 'long' followed by 'long' is illegal
C:\QuEnc\common.h(1259) : error C2629: unexpected 'volatile int ('
C:\QuEnc\common.h(1263) : error C2143: syntax error : missing ')' before '}'
C:\QuEnc\common.h(1263) : error C2143: syntax error : missing ';' before ')'
C:\QuEnc\avformat.h(48) : error C2059: syntax error : 'bad suffix on number'
C:\QuEnc\avformat.h(48) : error C2146: syntax error : missing ')' before
identifier 'L'
C:\QuEnc\avformat.h(48) : error C2059: syntax error : ')'
C:\QuEnc\avformat.h(49) : error C2059: syntax error : 'bad suffix on number'
C:\QuEnc\avformat.h(49) : error C2146: syntax error : missing ')' before
identifier 'L'
C:\QuEnc\avformat.h(49) : error C2059: syntax error : ')'
C:\QuEnc\AVSEnc.cpp(309) : warning C4305: '=' : truncation from 'const
double' to 'float'
C:\QuEnc\AVSEnc.cpp(334) : warning C4305: '=' : truncation from 'const
double' to 'float'
C:\QuEnc\AVSEnc.cpp(557) : warning C4244: '=' : conversion from 'double'
to 'unsigned long', possible loss of data
C:\QuEnc\AVSEnc.cpp(605) : warning C4244: '=' : conversion from 'double'
to 'unsigned __int64', possible loss of data
Error executing cl.exe.

QuEnc.exe - 39 error(s), 8 warning(s)


(Are all these headers required???)
Am I doing something completely wrong, or am I just a retard???
I have a little experience with C/C++, I know Java and other languages better. I was expecting a pretty straigt forward compile but alas no.

You may be wondering why I want to compile the sources, well because I have "FIXED" the ratecontrol in ffmpeg (the code is a little messy at the moment, but works nicely). No bufferunderflows, quantiser is modulated with frame complexity, and if I get it compiled for QuEnc, should beat CCE or TMPEnc in quality (very good results with mencoder so far...).

For those that encode 2 hours of film onto 1 CD (SVCD 480x480, 480x576) at 850kps, this is what you've been looking for...

FFMPEG was compiled with MingW32 (maybe a problem?!?!?!?!?)
Tried to compile QuEnc with VC++ 6.0 (SP6) with latest Platform SDK etc etc.

Can you help me Dragongodz??? Anyone???

(btw. sorry for the long post)

[mean]

I'd be interested to have a look at the patch

[mean]

and "long long" means 64bits integer in Ansi/Gnu term

[Peter Cheat]

I'll clean op the ratecontrol code a bit, remove the debug messages I put in and post in on a website for those interested.

I guess I'll have to go through and fix the errors one by one. Damn I wish I knew X86 asm...

[mean]

Just out of curiosity, you are still using the VBV model for ratecontrol or a different way of enforcing max bitrate ?

[Nic]

Hi Peter,

Well done on fixing the ratecontrol :) (If indeed you have, I'd love to look at your source, if available please send to nic@nic.dnsalias.com )

As for the errors you were having...ffmpeg code is very gcc biased (it's a shame it wouldn't take much of a patch to make it (at least the headers) more VC compatible (although there is little interest in doing so)).

I've lost my notes, so I can't remember how I used to solve this problem. Try sticking a #undef ARCH_X86 in bswap.h after the header files. That would help some of those errors.

Shout me if you need more help. :)

-Nic

[vmesquita]

Hi Peter Cheat,

Great Job, I also would like to check the mod you did to the rate control. You don't need to recompile QuEnc: just recompile ffmpeg using mingw, so you'll get a new avcodec.dll with your fix. ;) Unless of course you changed something in avcodec.h.

[Peter Cheat]

@Nic
Thanks I'll try that when I get home from Uni

@mean
The VBV model is still being used, but to implement really strict max bitrate (ie frame size = max bitrate / frame rate) would take 3-4 extra lines of code max, and an extra parameter to enable this strict enforcement. I concentrated on fixing buffer underflows as my hardware player would jerk like crazy when they occurred.
I will write the code for strict max bitrate when I get home. I rewrote the frame size prediction, which was the problem with FFMPEG.

@vmesquita
Clean compiles are always best :)
Also, I would like to add extra options to improve functionality of QuEnc.

[mean]

Doig max framesize= bitrate/fps will give you a loss of quality

The following example is a bit extreme, but you 'll get the meaning

Assuming I frame size is 10
P frame size is 6
B frame size is 4
with a gop size of 12, you will have

1xI frame = 10
3xP frames= 18
8xB frames= 32

Total = 60
So it would give 60/12 frames~ 5 as upper limit for the I & P frames
which will negatively impact the overall quality

[Peter Cheat]

Originally posted by mean
Doig max framesize= bitrate/fps will give you a loss of quality


Yes, this is obviously the case.
I just realised this is pointless as if you want to keep the bitrate at a strict maximum, just make the VBV buffer really small (ie 1) :)
However there might be a need for strict max bitrate enforcement. I don't know but I'll leave it as it is for now.

btw
I managed to compile QuEnc, but, it crashes when you try to encode (Access Violation 0xC00000005) and for some reason, it compiles with the DLL built in (avcodec.dll not needed). My modified (improved)avcodec.dll will be uploaded sometime tomorrow for thorough testing.

[vmesquita]

Peter Cheat,

If possible, also post the source of your modified DLL so I can compile with my encoder FreeEnc. ;)

[RobertR]

I would also ask to post those changes at ffmpeg mailing list.
This would also help people that are trying encoding DVDs under Linux.
(there was someone there that fixed some problems with MPEG2 encoding, was that you?)
Can't wait till i get my hands on it
Thank You very much for your effort :D

[dragongodz]

However there might be a need for strict max bitrate enforcement
there is. vcd/svcd/dvd all do have a max bitrate because of hardware players. to ignore the max bitrate means that they may not play back smoothly or in extreme cases not at all.

I guess I'll have to go through and fix the errors one by one
yes the exact same thing as i have had to do. Nic and i did warn about this though so nobody should be surprised. as is i still cant get aspect ratio working properlly and it is no doubt 1 of those things that i havent changed right or something i need to change but havent. bloody frustrating.

[Peter Cheat]

Originally posted by dragongodz
there is. vcd/svcd/dvd all do have a max bitrate because of hardware players. to ignore the max bitrate means that they may not play back smoothly or in extreme cases not at all.


This surely can't mean that, for an SVCD for example, the maximaum frame size is 13KB at 25fps (2600kbits/s / 25 * 8 =13).
Hardware players have a buffer (112KB for SVCD). As I understand, as long as there is data in the buffer the player will be happy. Therefore a frame (perhaps I frame) can be 80KB, as long as the following frames do not chew up the rest of the buffer before it can recover. That is how I have implemented the code anyway, and my hardware player stopped jerking.

I have finally uploaded avcodec.dll (FFMPEG) and Mecoder, compiled with Ming32 to Modified Libavcodec Site (http://petercheat.host.sk/libav/). Try it out, and tell me what you think about it. The DLL should work with QuEnc and FreeEnc.

[Nic]

Hi Peter,

Well done on the work :) Being LGPL and everything, the source should really be released as well. Is there any chance of you uploading it soon? I'm sure myself, dgz, vmesquita would love to see the improvements you've made.

Cheers,
-Nic

[vmesquita]

Hi Peter Cheat,

Unfortunatelly, It didn't work properly in my test with FreeEnc... I asked for max 2500 and got spikes over 14000! :( Maybe I should tweak some parameter?

[mean]

In the original lavcodec 2 pass code, the max bitrate stuff was computed on the pre-pass2 stage but not in the actual pass2 .
Did you fix that too ?

[dragongodz]

Hardware players have a buffer (112KB for SVCD). As I understand, as long as there is data in the buffer the player will be happy. Therefore a frame (perhaps I frame) can be 80KB, as long as the following frames do not chew up the rest of the buffer before it can recover.
the max bitrate is usually averaged over the GOP. for reasonable quality there needs to be a limiting factor aswell to provide consistancy over the GOP aswell. in the case of CBR strict frame size limiting becomes even more important.

[Paced]

Originally posted by vmesquita
Hi Peter Cheat,

Unfortunatelly, It didn't work properly in my test with FreeEnc... I asked for max 2500 and got spikes over 14000! :( Maybe I should tweak some parameter?

Same thing happened to me; I managed to get a max bitrate of 23800 (with FreeEnc). On the other hand, when I use QuEnc (dragongodz' version), it exits/crashes just as it starts the second pass. With Nic's 0.51, the program doesn't start - it just comes up with a typical Windows 2000 crash popup window...this is all with Peter Cheat's avcodec.dll, by the way. :confused: But, I'm glad, and thankful, that someone is working on making the rate control better :D

[dragongodz]

Paced - with mine i have had some problems when avisource() is used for 2 pass, meaning second pass throws an error. directshowsource and mpeg2source(dvd2avidg 1.10) 2 pass works fine.

just did a quick 1500 frame test. not good.

target bitrate was 2000 with max 3000 for 352x576(hald D1). bitrateviewer shows average bitrate 3573 and max bitrate of 7682. those are waaaay over whats desired.

[mean]

The vbv buffer starts full
It is filled up with max bitrate maximum speed
It is emptied depending on the picture size and relation with previous pictures (i&p cannot be discarded right away).

If the buffer gets empty it means that max bitrate is not enough to fill it up quickly enough ->buffer underflow

The maximum bitrate is just one factor in the formula.
The VBV buffer size is as much or even more important.
The way the frames are removed from VBV buffer is not trivial either, especially for interlaced encoding.

What usually happens it that the ratecontrol reset itself at each gop and assumes it is full again. That(s why having variable gop size is not always a good idea.

Said in other words, the maxbitrate reported by bitrate view should be taken with care as it is not a real view of what happen in term of buffer fullness, which is what really matters, but a partial
view that is correlated to it but not equal.

The main issue with libavodec code is that the VBV buffer is used as very short term modifier, and in case of quick decrease of buffer fullness it not quick enough to compensate. Moreover the VBV model is a bit simple.

Now depending on the hardware, the buffer can be greater than the size indicated in the seq header.
It means that if the mpeg stream says it needs a buffer of 112K, the dvd player may use its full buffer anyway which may be > 240 Kb
Such player are very tolerant toward out of spec streams

Others apply strictly what's in the header and will use 112k period.

Finally the CD/DVD rom max speed should be x2 for SVCD, but some players happilly make them move faster, which makes them more tolerant toward out of spec stream.

A good example is pioneer old model (non MKT based), ST based.
They apply strictly the spec.
2x is 2x speed. Period.
112KB is 112K. Period.

[dragongodz]

mean - i agree but the max bitrate is usually a good indicator of if the VBV is working or not. a small spike in a GOP means nothing but when the total average and max are so high above target its easy to see its not.

also the point about different hardware being able to handle footage different is a good point. the target is and should always be to work on standards following hardware.

[Peter Cheat]

Originally posted by Nic
Hi Peter,

Well done on the work :) Being LGPL and everything, the source should really be released as well. Is there any chance of you uploading it soon? I'm sure myself, dgz, vmesquita would love to see the improvements you've made.

Cheers,
-Nic

I sent you an email with my modified source last night (my time). Source is available by request at the moment (don't want dodgy, almost completely undocumented code to spread). It has taken me longer to clean up than I expected.

Originally posted by mean

In the original lavcodec 2 pass code, the max bitrate stuff was computed on the pre-pass2 stage but not in the actual pass2 .
Did you fix that too ?


Ok, the two pass code is _entirely_ written from scratch. First pass is now done with constant quatiser 2, and max bitrate is ignored (first pass is useless to keep). Second pass takes the VBV buffer into account (rather than the max bitrate). The code works by using the average 'compressibility' of the 5 previous frames, and looks at the current and 9 future frames for possible buffer underflows. If a bufferunderflow looks imminent, the quantiser for the offending frames is increased by one and checked again. It will continue increasing the quantiser by one until there will be no buffer underflow. This creates a sort of averaging of degradation of frames, instead of a visible quatiser 31 (like in original code) for one frame, 10 frames can be averaged to a quant of about 7-10, which isn't noticeable. So far, I haven't been able to generate any buffer underflows, even with normally hard to encode material (such as the start of 'X-Men' movie, which is what TMPGEnc had a lot of trouble with). This is why I consider it fixed.


Originally posted by vmesquita

Hi Peter Cheat,

Unfortunatelly, It didn't work properly in my test with FreeEnc... I asked for max 2500 and got spikes over 14000! Maybe I should tweak some parameter?


But was there a buffer underflow? I don't think that there will be. Try encoding with Mencoder, or FFMPEG as I just realised that I put the version of Mencoder compiled for my CPU an Athlon XP. If you want to keep a strict maximum bitrate, set the buffer size to 0 and try that. 14000 / 8 / 25 = 70KB, which is less than the buffer size of 112KB, assuming you are using an SVCD profile.

I didn't expect it to work properly with QuEnc, or FreeEnc as I have changed code. QuEnc and FreeEnc was compiled with a avcodec.lib corresponding to a different DLL file. This is why I wanted to get QuEnc to compile (and work), so I could do some testing. On another note, what is the quality of the output like? Better than original libavcodec? As I said, if you cant get it to work, try with FFMPEG. If I can get QuEnc to compile, I can do more thorough testing, and even allow for a very strict bitrate enforcement, as it seems that this is what people are requesting (although I believe this is due to lack of understading of how hardware players work).

I'll upload some more tools soon, such as Mplex 2.0, which is great for SVCD multiplexing, some programs I have written, such as an audio analyser used to compare the difference in SNR and MSE for different codecs such as MP3, OGG, AC3, WMA etc so people can actually see that WMA sucks compared to MP3 :).

Back on topic, bitrate spikes are ok, PROVIDED BUFFER UNDERLOWS DO NOT OCCUR. A bitrate spike does not mean the codec stuffed up, it means the frame was harder to compress compared the the frames around it :) Buffer underflows are the problem I attempted to fix.

I am also considering the option of a third pass, which will marginally improve quality and further ensure no buffer underflows. But this is _after_ the 1 pass code has been done.

I'm out
Peter Cheat

[dragongodz]

bitrate spikes are ok, PROVIDED BUFFER UNDERLOWS DO NOT OCCUR.
sorry but thats not strictly true. small bitrate spikes will generally be fine true. spikes to 14000 such as vmesquita quoted(dvd target output i am assuming) is definatly not. most modern hardware dvd players may well play it but there will be others that stick strictly to the standard which may choke or stutter at that point. thats also why i said a more consistant GOP.

so actually there is 2 problems. VBV and max bitrate. both need to be considered. as i also said strict max bitrate is also very important for CBR.

actually i do remember an old open source encoder that used to do a GOP to memory and if it underflowed or overflowed then redid the whole GOP so it was consistant quality and able to catch major spikes in all frames. any spikes could then be dispearsed to the other frames in the GOP aswell.

[Peter Cheat]

Originally posted by dragongodz
sorry but thats not strictly true. small bitrate spikes will generally be fine true. spikes to 14000 such as vmesquita quoted(dvd target output i am assuming) is definatly not. most modern hardware dvd players may well play it but there will be others that stick strictly to the standard which may choke or stutter at that point. thats also why i said a more consistant GOP.

so actually there is 2 problems. VBV and max bitrate. both need to be considered. as i also said strict max bitrate is also very important for CBR.

actually i do remember an old open source encoder that used to do a GOP to memory and if it underflowed or overflowed then redid the whole GOP so it was consistant quality and able to catch major spikes in all frames. any spikes could then be dispearsed to the other frames in the GOP aswell.

dragongodz, I believe you misunderstand the actual concept of VBV buffers.

The purpose of VBV buffer is to allow for spikes. If indeed the profile which was used was DVD, then there is a buffer of 224KB. Now lets start with a full buffer:

Initial Buffer Occupancy = 224KB = 1835kbits
Lets say that the first frame is 560kbits (14,000kbit/s 'spike').
Frame 1 = 560kbits Buffer Occupancy = 1275kbits
But at the same time that the frame is being read, the buffer is filling with new data at max rate, 9800kbit/s (correct me if wrong)
Buffer Occupancy = 1667kbits
The buffer is now a little smaller, but not even close to an underflow!

Maybe vmesquita meant SVCD? If thats the case:
Initial Buffer Occupancy = 112KB = 917kbits
Lets say that the first frame is 560kbits (14,000kbit/s 'spike').
Frame 1 = 560kbits Buffer Occupancy = 357kbits
But at the same time that the frame is being read, the buffer is filling with new data at max rate, 2600kbit/s
Buffer Occupancy = 357 + 104kbits = 461kbits
We've lost half of the buffer, but we are still not creating an underflow. Hardware players that can play SVCDs to spec will not have any problems!!!!

This is provided that the next frames do not consume the rest of the buffer before it will refill. This is a simplified model of how the VBV buffer works. However, to my knowledge, it is correct. I had so many problems playing SVCDs made with FFMPEG, Mencoder, and even TMPGEnc, because of underflows. Since modifying the way libav controlled the frame data, I have had no problems (with 3 full length encodes). To me, this shows the model works. I haven't tried interlaced material, but I assume that the two fields are taken together (as a frame) and dealt with by the decoder. I maybe wrong here, as I said I haven't tried interlaced encoding.

In summary, a huge spike is completely meaningless unless it is larger than the actual size of the buffer, or larger than the remaining buffer.

A HUGE SPIKE IS MEANINGLESS AND DOESN'T AFFECT PLAYBACK UNLESS GREATER THAN BUFFER SIZE/REMAINING BUFFER

Maybe I should write a tool to analyse mpeg streams for underflows?
I don't have time for that sort of thing. Multiplexing tools usually tell you if there will be a problem with playback (TMPGEnc and BBMPG do).

FFMPEG used to create huge numbers of underflows (sometimes over 100 in a 1500 frame test encode). This was hopeless. I perform the same test with the modified ratecontrol and no underflows occur. Of course there are spikes, and for the best quality, one would expect them to be there. I frames take significantly more space than B frames for example. I expect to see spiking for I frames, a spiking B frame would trigger my attention, but it just may mean that the scene being encoded is of high motion.

mean's explanation (4 posts back) is almost spot on in explanation, except as far as I see, the VBV model in libav is *NOT* short term

[dragongodz]

I believe you misunderstand the actual concept of VBV buffers.
well you can believe what you like but you missed the point. :)

notice i said its 2 problems and not 1 ? VBV undeflows are just 1 part. max bitrate is also a factor which some hardware players MAY have problems with. to asume a large spike is ok so long as VBV is ok is a bad assumption. if the hardware player is strict with its max bitrate per frame limiting or reading it will have problems with such a spike no matter the VBV. actually full rate control takes in to account min bitrate and average/target bitrate aswell of course.
actually i can give a good example of how underflows are not the whole story. in some of the DVD-Rebuilder threads and on other forums its been shown that on some dvd players if min rate falls below a certain size (some its under 500 some its under 150) playback can stop or show serious problems. now thats only on certain dvd players. the target is to work on all and not just most.

I have had no problems (with 3 full length encodes). To me, this shows the model works.
no it shows it works for you with your player etc. a universal "works" is a big call from that. :D

A HUGE SPIKE IS MEANINGLESS AND DOESN'T AFFECT PLAYBACK UNLESS GREATER THAN BUFFER SIZE/REMAINING BUFFER
well thats another big call. so you have tested that on how many players exactly ? :sly:

[Peter Cheat]

This is my interpretation of the SVCD/DVD profiles.
I have not read anywhere that there is a 'hard limit' for max bitrate except for the max size of the VBV (this is the 'hard limit' that the encoder abides by). Do some test encodes for your player, and see if they play correctly. If you know about any 'hard limits' tell what they are, and they can be implemented immediately, and posted on the website. It's as easy as that.