QuEnc 0.59 Beta 4

[Nic]

http://nic.dnsalias.com

Minor update:
Slight change to rate control settings to help with undersize with 2-Pass (I'm not sure if 2-pass is really worth the effort with current libavcodec).
Should have fixed MPEG-1 file creation. (i.e. now writes the AR correctly)

It's now been compiled with MinGW/MSYS for the first time. Cygwin doesn't like my machine anymore. I'm not sure if they'll be a speed or stability difference. But it seems ok to me.

Cheers,
-Nic

[dragongodz]

Should have fixed MPEG-1 file creation. (i.e. now writes the AR correctly)
did a quick 352x288 mpeg1 cbr and bitrate viewer and gspot both say 4:3(as i had selected in QuEnc) and it plays fine on pc. 1 bizarre thing is if you go in to properties in MediaPlayerClassic it says AR 176:135 for video size. as i said though plays fine on pc but dont know if that would make any difference on hardware players. :)

I'm not sure if 2-pass is really worth the effort with current libavcodec
or to be more exact the rate control that if you use 1 setting cant make high bitrates and if you use another wont always keep below max bitrate. ;)

[lithoc]

352x288 mpeg-1 will play just fine with hardware player.
If encoding to VCD, i think concern should be the bitrate.

I don't think there's Aspect Ratio standardise in mpeg-1.
That;s why i'm always get weird Aspect Ratio for mpeg-1 in media player classic but i can play with hardware player once mastered into VCD.

If you play the VCD mastered file in MPC. It will display correct aaspect ration..

Weird....

:)

[video_magic]

^ Does that mean the DAT file which I think you can get off a VCD has extra data for an mpeg1 which means it can properly on the hardware players. I guess this is information like aspect ratio and FPS and other things to do with an MPEG stream? :confused: I have renamed DAT to MPG sometimes when I want to play a file on the PC and it plays fine. I wonder if QUENC could write some extra non-standard-mpeg information to the files like this which would be useful but harmless to whatever application or player has to deal with them (harmless to applications & players that wouldn't need it)?

[dragongodz]

well i did a little tests setting it to 16:9 and yes it stretched as it should on playback with MPC. so it is working. :)

as for VCD well i have doubts AR would be needed anyway in the sense that VCD doesnt support 16:9 playback etc so hardware players should just play it 4:3 anyway. people would have to test that for themselves however.

asi said i just found it funny to see. :D

[freelock7]

Yes, Nic, it's better about prediction. But be careful about bitrate spikes caused by a low quantizer.
The underflow risk is very high with this version. If I should choose a version among the 059's, the beta3 is the best for its stability and the good work of the quantizer. Now, the undersize prediction appears in VBR high bitrate (>4500kbs). Encoding in CBR high bitrate >5000kbs can resolve the problem and you have a great quality too.

[Nic]

Thanks for the feedback. At present, we're trying to find a nice middle ground. It's tricky.

Apart from bitrate, I think everything's pretty stable now. Probably will go to 0.60 next time and may do something directly to the rate control.

-Nic

[Affar]

Hi

I've got an XviD video that, when converting using avisynth with QuEnc, it is reduced 3 frames in the final video. It doesn't matter if I charge it with directshowsource, avisource or load video direct, the final movie is always the same.
Here there is a sample of the file.

SAMPLE (http://www.divxhouse.com/pedazito.avi)

Thanks. See ya.

[dragongodz]

The underflow risk is very high with this version.
it shouldnt be any riskier than 0.54 was however since thats what it is based on, plus some more tests i ran. ;)
i will do more tests with settings later to keep trying to find a better middle ground until something is done about the RC.

It's tricky.
understatement of the year here folks. :D

may do something directly to the rate control.
just so people realise this is not as easy as it sounds. the options are
1. add some small in code changes to make RC atleast a bit better
2. major changes in code to make RC work properly or close to it
3. add secondary new rate control, based on another working RC

now even with option 1 Nic would have to either get the changes accepted in FFMpeg or make the changes every single time libavcodec is updated for QuEnc. neither of which is as easy as people may think. ok making the changes every time may not be that hard if they are few but its still a bugger to have to do.

[Koepi]

I encoded a simpsons episode (that one where Homer builds a website, need to show some collegues that one as they always want blinking, flashing things on the company's homepage) which ran yesterday evening to SVCD - once with HC and once with QuEnc 0.59b4.

Average bitrate should be 2500kbps, max set to 2554kbps.

The encode came out just fine, with being visibly better with QuEnc than with HC.

Thanks for the program Nic, it really rocks! (and also thanks to the libavcodec/mpeg2encoder coders of course :) )

Cheers
Koepi

[Nic]

@Koepi: Glad it went well :)

@Affar: Dragonball z samples are always welcome :) Is the 3 frames missing still occuring even when using that short sample? I'll look into it.

@Dgz: I'm hoping I can make it so I can control a frames QP on a frame by frame basis from QuEnc, then i'm going to try and do the ratecontrol from inside QuEnc. I'm hoping to use x264's ratecontrol, because I really like the code (it's easy to adapt). Of course, this could all be an impossible pipe dream that I haven't thought through properly yet, but it's worth a try :)
(x264 only does encoding on a frame by frame level at present (unless using CBR or non existing adaptive quant) so I should be able to apply it pretty easily)

-Nic

[dragongodz]

I'm hoping I can make it so I can control a frames QP on a frame by frame basis from QuEnc, then i'm going to try and do the ratecontrol from inside QuEnc.
hmm interesting idea. you may want to have a look at Avidemux by Mean sometime and see what that does since it has an option to use Xvid RC from the Xvid.dll with libavcodec if i remember correctly.

Of course, this could all be an impossible pipe dream that I haven't thought through properly yet, but it's worth a try :)
if anyone can do it i know you can. ;)

[Nic]

"it has an option to use Xvid RC from the Xvid.dll with libavcodec if i remember correctly."

Really?! Jeeze, that would solve all our problems. I'll look into that now. (XviD's ratecontrol code is very good (but a lot more difficult to get to grips with than x264's)

-Nic

ps
Looks interesting, for those interested:
avidemux-2.0.38rc2b\avidemux\ADM_xvidratectl\ADM_xvidratectlVBV.cpp
Is the file to look thru in the source. Even mentions it uses some of Peter Cheat's ideas. Could be promising.

pps
Ripped the XviD rate control out of AviDemux and got it compiling under VC6. Now just got to whack it into QuEnc. This will be interesting. Maybe I could contact Mean, do you speak to him dgz?

[dragongodz]

Really?! Jeeze, that would solve all our problems.
yes i thought it may be easier than reinventing the wheel. :D
then again i also thought it was an interesting idea here
http://forum.doom9.org/showthread.php?threadid=84479&perpage=20&pagenumber=3 ;)

Maybe I could contact Mean, do you speak to him dgz?
no only talked to him on here. unfortunatly he doesnt seem to drop in often. you can probably catch him on the avidemux forum however
http://avidemux.berlios.de/forum/
or you could try the contact email on the main avidemux page
http://fixounet.free.fr/avidemux/

[Koepi]

I'm already starting the la ola waves here - looking forward to the beta5 incorporating xvid's ratecontrol. (If it has "strict scaling" I'm even dancing around naked for you! ;) ).

Cheers
Koepi

[dragongodz]

If it has "strict scaling" I'm even dancing around naked for you! ;)
koepi - that is an image i really didnt need before going to bed you know. :eek:

i agree strict scaling would be good aswell but even without it should be an improvement on the current RC.

[gigaset2004]

Hi,

Nic wrote :

(I'm not sure if 2-pass is really worth the effort with current libavcodec).

So i wa a little suprised.

Is it recommended to use Quenc in single pass ?

[Nic]

@gigaset2004: I really don't know. Never done enough testing to see if 2 pass helps quality enough to be worth it. It definitely doesn't work as it should.

@all: If you do some tests i'd be happy to hear the results :)

-Nic

[Koepi]

...do you have a "new" version with the xvid scaling code flying around already? Did I miss something?

(I don't know if 2pass delivers better quality than 1pass currently - I naturally assumed this is the case as the bitrate is distributed more even, and from watching some svcds i made with quenc i _think_ 2pass looks like a close-to-proper quant distribution).

I hope this helps (and is what you wanted to know ;) )

Cheers
Koepi

[Nic]

@Koepi: Close to a new version. Writing it now. Able to control the Quant being used to Encode on a frame by frame basis. Also retrieve the Quant and FrameType after encoding the frame. Now just to hook it up :)

EDIT: Well, not sure if i'm putting in the right numbers. But just asked it to do a 5mb encode, and a 5mb encode came out. :D :D

EDIT2: Motherf**king computer crashed and now I can't load AVS Files. Grr.

[PatchWorKs]

Any chance to have a FairUse (http://www.fairusewizard.com/Release/fu-src.zip)-like features version (i mean direct DVD encoding, 4:3 auto-crop feature, etc) ?

[Nic]

@PatchWorks: No, that would be for another app.

@all: Just a progress report on ratecontrol:
The ratecontrol only works for 2 pass encodes. Which is cool, because the 1 pass encoding is ok with QuEnc. I've got the code working. Not definitely sure i'm pumping in the right numbers. But most of the time it's looking very promising.

I don't think it will be perfect, there's a tradeoff between always getting the exact size and getting no buffer overflows (Especially when encoding where the avg bitrate will be close to the maximum).

Exciting hey?

-Nic

[video_magic]

very! :D Really looking forward to playing with a new version, thanks a lot Nic :)

[dragongodz]

Never done enough testing to see if 2 pass helps quality enough to be worth it. It definitely doesn't work as it should.
the problem with libavcodecs 2 pass is that it doesnt make enough effort to do what its meant to. that is bitrate constraints and distribution. both are done so in theory should be better than 1 pass BUT for example max bitrate can be exceeded(spiking) because the RC basically makes 1 go at altering it and then if its still not right doesnt recheck and try again. so as Nic said it doesnt work as fully as it should. 2 pass CBR appears better than 1 pass CBR in the couple of small tests i did though but this shows how 1 pass mode isnt working 100% either since 1 pass CBR should be easier, encode frame -check size - if not right encode again until it is, thats not being done. oh and setting a min bitrate, which should really be done for some hardware dvd players, throws the RC off aswell. are people starting to see how flawed libavcodecs RC is now ? :D

Motherf**king computer crashed and now I can't load AVS Files. Grr.
NIC!!!!! i dont think i have heard such language from you before. ;)

The ratecontrol only works for 2 pass encodes. Which is cool, because the 1 pass encoding is ok with QuEnc.
cool. maybe replace 1 pass in the future aswell if this goes well , even if just CBR ?

I don't think it will be perfect, there's a tradeoff between always getting the exact size and getting no buffer overflows (Especially when encoding where the avg bitrate will be close to the maximum).
well once you have this working that sort of thing can be looked at more if need be. still should be better than current RC.

Exciting hey?
indeed, indeed. :D

[Fishman0919]

Nice, Nice....very nice!

[kitsaros2000]

yep i also did an vcd test and works well !
Well done nic !!!
Thx !:cool:

[mean]

There is also a couple of patchs to lavcodec itself that may be of interest :
- 3:2 pulldown

- better enforce vbv with "native" 2 pass libacodec rc code. It is not as good as the external ratecontrol as lavcodec does not try to see there is a intra coming and let the buffer underflow but at least it tries to keep it whereas with unpatched code it does not try at all.

- separate actual bitrate/vbv buffer size from the one in the mpeg header vs the one used

The big advantage of using the external ratecontrol is that you can have several ratecontrol the api (if you kept the interface) is very simple.

[dragongodz]

mean - nice to see you dude.

The big advantage of using the external ratecontrol is that you can have several ratecontrol the api (if you kept the interface) is very simple.
which would mean in theory Nic could use x264 rate control ,or whatever, instead then.

also looking at the cvs date avidemux\ADM_xvidratectl\ was last updated 3 months ago and just says things like "began adding constrained ratecontrol to xvid rc" and "smarter rc" so not totally up to date or complete ?

[Nic]

@Mean: Thanks for posting. Do you have any links to those patches? Or are they part of avidemux? (and if so are they marked in the source?). I'm on the ffmpeg-dev mailing list and haven't noticed those patches.

I kept the interface. Very nice and easy to use. Although it's hardcoded to use a min quant of 2. Which causes problems with high bitrate (DVD style) Encoding. You may want to lower that for avidemux.
One question; When using setVBVInfo, i'm currently using: setVBVInfo(9800000/8, 0, 224) Does that look right to you for DVD encoding (max bitrate 9.8MBits, 224 buffer size?)

What's your current opinion of avidemux's XviD VBV ratecontrol?
Also, what do you think is better for compliant DVD Encoding, ADM_newXvidRc or ADM_newXvidRcVBV? They both seem to have VBV code.

Cheers,
-Nic

[mean]

They have been posted & rejected.

The 3:2 pulldown appears from time to time, Mans did an updated patch toward cvs not long ago.

The min quant of 2 is to avoid qz overflow when using quant=1 with libavcodec. it might not be needed anymore.

224 is ok for DVD.

The ratecontrol/VBV is performing ok most of the time.

The anticipation of intra is not working in all cases, a big
intra coming in might cause a underflow from time to time, but it is rare (like 5 time on a whole movie).

All patches in adm_libavcodec has a //MEANX in it to easily spot them/merge them.

[mean]

You might also be interested in looking into the ADM_mplex directory
which offers a simple interface to use libmplex as output muxer which works better (in my experience) than lavformat mpeg muxer.

[Nic]

@Mean:

"They have been posted & rejected."
Michael can be picky.... ;)

"The 3:2 pulldown appears from time to time, Mans did an updated patch toward cvs not long ago."
I found the pulldown patch (dated: 29/11/04). I'll look into adding that. Thanks.

"The min quant of 2 is to avoid qz overflow"
It definitely causes problems, although quant 1 has always been bad for ratecontrol (still is with XviD), it is often necessary for high bitrates.

Thanks again, I'll look through avidemux's code. Nice app.

-Nic

[dragongodz]

although quant 1 has always been bad for ratecontrol
and it really should not be. does mpeg2enc's 2 pass use quant 1 ? i ask since that is meant to never exceed max bitrate etc.


DVD encoding (max bitrate 9.8MBits
9.8Mb/s is max for video only of course. if you intend to have any audio and/or subs etc then they need to be subtracted. so say you have 1 224Kb/s audio stream aswell you would need to reduce the 9.8Mb/s by 224Kb/s + a small extra small amount for muxxing overhead.

[Nocturno]

i did some testing with this version and RC doesn't seem to work correctly for 1-pass either

running with:

Quenc.exe -i "E:\avi2dvdtest\Movie0.avs" -o "E:\avi2dvdtest\Movie0.m2v" -maxbitrate 8000 -b 5650 -1 -aspectratio 16:9 -gopsize 12 -mpeg2 -vbr -hq -scene -priority 3 -nointerlaced -cmatrix "C:\Program Files\avi2dvd\matrix\Standard.txt" -mpeg2mux noaudio -auto -close

gave peaks above 10000 (10542 highest)
and multiple above 8000

the file won't multiplex with dvdauthor's mplex (to many framedrops)


Num. of picture read: 157728
Stream type: MPEG-2 MP@ML VBR
Resolution: 720*576
Aspect ratio: 16:9 Generic
Framerate: 25.00
Nom. bitrate: 8000000 Bit/Sec
VBV buffer size: 112
Constrained param. flag: No
Chroma format: 4:2:0
DCT precision: 8
Pic. structure: Frame
Field topfirst: No
DCT type: Frame
Quantscale: Linear
Scan type: ZigZag
Frame type: Progressive
Notes:


can't wait to see the new RC :p

[dragongodz]

i will be continueing discussions with Nic by PM so dont expect to see too much more indepth discussion here, though Nic does tend to want to tell people what he is doing. really most people either wont fully understand what we are talking about or want to start asking for more. just the bits we have discussed already has started "cant wait" responses and got people excited when there is no saying when this will be ready etc. knowing Nic it could be any time. :D

[Nic]

@dgz: :) I like kicking up a bit of excitement and interest. It motivates/reminds me to continue. 1 pass may have been effected by the lmin = 1 change.

@Nocturno: Thanks for the feedback. Next version will probably be 0.60 which will have Beta 3's 1st Pass RC and XviD's 2 pass RC.

-Nic

[freelock7]

Great job guys!
Hope to use the v.060 soon!

[dragongodz]

1 pass may have been effected by the lmin = 1 change.
had a quick look and you disabled the lambda for 1 pass mode aswell and then comment "Seems to work well for 1 pass, so keep it seeing it is "correct" ". so you should have moved that down to the if ( nPass == 0) section and added the lmin = 1 to the else. you do use lambda in the low bitrate part however which would be fine for either type.

see thats an example of what i thought doesnt need to be discussed here, internal workings. most people will have absolutly no idea what i am talking about. :D

[Koepi]

Well, I for one like to know that. It helps me to see if you talk about peanuts :p

;)

Cheers
Koepi

[Nic]

@dgz:
"thats an example of what i thought doesnt need to be discussed here"
But if I don't talk about it, how will others learn? Also, the main reason I do it is because it's a nice record for me to look back on when I forget the info. Which is inevitable ;)

[dragongodz]

uh-oh, i think i am being ganged up on here...or atleast out voted. ;)

ok then let me make a comment here that i already said by PM.

go back to the thread where i originally said using Xvids rate control would be an interesting idea(link posted already). peter cheat makes an interesting remark about how Xvids RC could still underflow and how another check is needed to help insure it shouldnt. very relevant for using quant 1 because the simple fact is quant 1 should not be a problem for rate control at all. there are plenty of encoders that use quant 1 fine. :D

[Nic]

@dgz:
I'm not sure how relevant Peter's post was in that instance (he wasn't looking at avidemux's current RC), but I can see what he's saying. It can still underflow. Although I think it will be very rare. I think my best current solution is something that Peter tried, and that is if underflow is detected, re-encode the frame again with a different Quant until there is no underflow...
...However, I need to have a clip that causes underflow to test.

EDIT: Hmmm, not sure if this will work.

-Nic

[dragongodz]

he wasn't looking at avidemux's current RC
no he looked at Xvid 1.1 branch RC where VBV was introduced so is relevant somewhat.

if underflow is detected, re-encode the frame again with a different Quant until there is no underflow...
well actually it would be re-encode GOP until no underflow or max quant limit reached(if not using skipped MB for P and B frames). so you would have to buffer the frames , do the VBV check, redo if not correct, write once all ok. from what i understand HC does it similar to this for example. :)
if writing every frame as read then the last frame is the only place you could try and adjust so then you couldnt insure VBV 100% of the time, though you could also reduce the risk if you actually slightly underdid early frames in the GOP i guess.

Hmmm, not sure if this will work.
which part ?

[Nic]

The re-encoding the frame. Re-Encoding the GOP would work. Real Pain though. I'll put out a version without this. It reports on what it detects as overflows. People can then report how good/bad it is. While it's out i'll work on the re-encode GOP. It may also just happen with big I-Frames as mean suggested. In which case, it might be easy to fix with just one frame.

i'll PM you now with an early release. So you can look.
(I'll PM Koepi as well)

-Nic

[dragongodz]

actually since you should be able to get the achieved bitrate from the first pass and then able to scale each GOP and frame to the target bitrate you would just have to also account for VBV and max limits. if the GOP exceeds then reduce more so it doesnt. store undersize amount and give it back to the next GOP that can have it (or even a part) without exceeding, that way also trying to maintain target bitrate.

well i should be back in maybe 6 hours give or take. so plenty of time for others to think about it and make of my ramblings what they will. :D

[hank315]

so you would have to buffer the frames , do the VBV check, redo if not correct, write once all ok. from what i understand HC does it similar to this for example.Yes, something like it.
Just check which needed frames should fit into the VBV buffer at each moment.
If they overshoot just adjust quants for these frames and check the VBV again.

[dragongodz]

ok some tests ran with 0.6 early bird special version. ;)

max bitrate. 1 example target 2000, max 2400, 0.59B4 peaked 2700 while 0.60 peaked at 2408. so it may not be perfect but is pretty good and much better than 0.59B4.

i did notice the occasional large P frame poking up aswell. anyway i will do a longer encode tonight and try to author it with a few different programs and see if any complain. :)

hmm why is 0.60 starting with IIIPPBB (or decoded order IIIPBBP) ? much better IPPBB(decoded IPBBP) for first GOP i would think.

hmmm now on to some funny business.
2 pass with 0.59B4 - encodes fine
1 pass with 0.60 - encodes fine
2 pass with 0.60 - 2 frames short and quits or sometimes freezes ????

now before blaming QuEnc i have to say HC has now started being funny too. that is 0.12 and early 0.13 betas all encode correct number of frames but tend to freeze up at the end. the latest 0.13 beta however ends fine but encodes 1 frame short (and that means 1 frame less than it says in its gui aswell).

WHAT THE ?????

so i tried uninstalling Avisynth 2.55 and reinstalling it etc and the same result. tested with avisource(), directshowsource() and mpeg2source() (dgindex etc) even using different clips etc. nothing made any difference.
so i will download Avisynth 2.56 pre-release later and see what that does, not holding my breath though.

[Koepi]

I didn't look deeper into the spec-compliancy of the bitrate control (I encode at 950kbps with max bitrate at 8000 so that would be quite useless ;) ).

After the second pass I get a crash now ("Access violation at 0x0000048"). The resulting mp2 file works fine though. I didn't check the file for DVD compliancy. The image quality is fine.

(I do a two pass of a video with audio, resized to "half" ntsc resolution [352x480]. The sound is encoded as ac3 with 192 kbps. MUXing is done into "DVD". GOP size is 18. 2bframes max.)

One suggestion: as it is possible to produce muxed a+v output, on the save file-dialog the extension mpg would be useful.

Thanks a million again,

cheers
Koepi

[Nic]

You know the moral of this story? Nic shouldn't test his QuEnc with B-Frames turned off. D'oh! I get the crash too now, easily fixed.

I'll add .mpg as an extenstion.

Back to testing...

-Nic

ps
It's also b-frames causing the weird 3 I frame start. I'll try to compensate.

EDIT: @Dgz & Koepi: Download from the same link again for the updated version. Make sure it states it's RC2, so you don't download the same file again due to cacheing.

[dragongodz]

Nic shouldn't test his QuEnc with B-Frames turned off. D'oh! I get the crash too now, easily fixed.
cool. atleast QuEnc will be back working.

still doesnt explain what the hell has happened with HC though, thats really strange. as i said old versions encode correct amount of frames but dont close(hitting exit button does work) while the latest beta closes but is 1 frame short and hank315 cant reproduce it.

i installed Avisynth 2.56, i uninstalled any programs from the last week, i cleaned the registry, i even restored back a week and nothing made any difference. so as you may guess i had no time to test authoring the produced files.