QuEnc 0.54 released

[Peter Cheat]

Sorry for the double post, but here is an important quote from http://www.spawns.dk/svcd/main.htm
When capturing to SVCD with default SVCD-template of 2400kb/s videobitrate and 224kb/s audiobitrate - you might sometimes experience VERY short peaks up to about 3000kb/s. And you might ask yourself if this is not a problem since the SVCD-specs states a maxbitrate of only 2600kb/s ? Will it not cause my SVCD to stutter in my dvd-player ?

Well - no it wont - actually its a very common mistake to focus exclusively on max bitrate when creating compliant svcd. What is really much more important is muxingrate and packetsize. No encoder - software or hardwarebased is capable of keeping precisly within the max bitrate you specify, not even the DVCII. But this is not the biggest problem in itself - most modern standaloneplayers are capable of handling these very short peakbitrates within reason - the biggest problem is that most softencoders do not encode using the right muxingrates and/or packetsizes for SVCD - and this is mostly what is causing playbacktroubles like stuttering and breakups in video with ugly big green blocks sometimes even with peakbitrates lower than those of the DVCII. The DVCII however DO encode using the correct muxingrate and packetsize which is why a SVCD created with the DVCII will play correctly on 99% of the players even if the peakbitrates is close to 3000kb/s.

The correct values for SVCD are pack size = 2324 and mux rate = 2,788,800

Now the muxingrate is closely tied together with the datarate - this means that IF the peakbitrate much exceeds about 3000kb/s - the datarate WILL get to high for the standard SVCD-muxingrate to handle - and this will cause playbacktroubles - even if you do have a player capable of playing files with higher bitrates. The cure for this is simply to demux the file and mux it back together using BBmpeg setting the muxingrate to 0 - this will force BBmpeg to calculate the needed muxingrate to fit the datarate. The muxingrate will get higher and will nolonger be 100% SVCD-compliant - but it will make the file play correctly. This cure can also be used if you have older SVCD-files created with softencoders that are giving you playbacktroubles.

Please note however - that there are still limits to what bitrates can be used. Different players have different cappabilities in handling bitrates - some players may be able to handle almost any bitrate, while others may play fine with peaks at 2600 but immediately start breaking up with peaks at 2700 - muxingrates cannot ovecome these barriers - so you will have to try and figure out what peakbitrates your specific player can handle.


If 3000kbit/s for SVCD is the hard limit you want, as soon as I get home from Uni, a strict limit will be introduced, not allowing frames to be larger than 15% of the max bitrate, or user set. It will (hopefully) be uploaded later today (around 8:00pm GMT +10:00).
Maybe I should implement that the average bitrate for a GOP must not exceed the maximum bitrate...but I have the feeling this will not prevent huge spikes...

[dragongodz]

max rate, 9800kbit/s
you already quoted max bitrate spec for dvd (video component) playback yourself. to spike way over that is not a good idea to start with.

so again(and finally) i say its not just VBV but max bitrate aswell that needs to be considered for consistancy. i will do some fuller testing on sunday anyway and report what i find.

[Peter Cheat]

9800kbit/s is the maximum video data rate for DVD. This doesn't mean that a single frame cannot be 20,000kbits, it means that this is the speed the DVD is read!

[dragongodz]

thanks for that i will pick a couple of bits from that quote which actually back up what i have said.

will play correctly on 99% of the players even if the peakbitrates is close to 3000kb/s.
not 100% of players though is it ? which is what i said the target was. i even said most modern players do handle such streams but thats not the point.

The muxingrate will get higher and will nolonger be 100% SVCD-compliant - but it will make the file play correctly.
ah so long as it plays ok in 99% of players it doesnt matter that it breaks specs. hmmm.

Different players have different cappabilities in handling bitrates - some players may be able to handle almost any bitrate, while others may play fine with peaks at 2600 but immediately start breaking up with peaks at 2700 - muxingrates cannot ovecome these barriers - so you will have to try and figure out what peakbitrates your specific player can handle.
which is exactly what i have said several times now. :D


If 3000kbit/s for SVCD is the hard limit you want
no, bitrate limiting to max bitrate given is whats desired. whats the point of having a max bitrate setting thats not respected ?

[dragongodz]

9800kbit/s is the maximum video data rate for DVD. This doesn't mean that a single frame cannot be 20,000kbits, it means that this is the speed the DVD is read!
yes i know that. how well do you really think every dvd player will handle a 20000kbit spike though ? there are players that may start to have problems once it spikes over the 9800kbit. that quote you posted already says that type of problem for svcd, it holds true for dvd aswell.

[Peter Cheat]

What is the point of the VBV buffer then?
Do manufacturers stick to standards, or do they see the 9,800kbit/s limit and think to themselves, hmmm, I am lazy, so at 9,800kbit/s for a PAL DVD source, the max frame size will be 49KB, so my hardware decoder will decode no more than 49KB at a time and thats it!

If IC manufacturers design hardware like this, then there will definately be problems with playing back anything, including original DVDs from the shop.

If a DVD has a 224KB buffer, it is "safe" to assume that the decoder is able to handle at least 224KB per frame. Or is the buffer just there in case a frame is larger than 49KB, but really frames shouldn't be.

There seems to be a lot of contradictions in literature I have been reading. Sticking to the max bitrate for each frame is a little silly. It's like CBR, and everyone knows CBR sucks big time. I've just coded in maximum bitrate enforcement, I'll test in when I get home as I don't have MingW32 here at Uni :).
This is really going to suck though, but if this is necessary for strict compliance with players, then :D

[dragongodz]

as i already said small spikes should very rarely cause any sort of problem. other factors are being ignored though when you are considering VBV. audio, muxxing overhead, subs, padding/private stream - they all have to be loaded/played at the same time as the video aswell. once you start adding in those sort of factors you should see that its not only about video.

oh and yes CBR may suck but if someone wants to make a standard VCD for example it is needed. :)

[Nic]

Thanks Peter :) I'll have a look at it all on Tuesday (away till then :( )

You may want to make you max bitrate enforcement optional...if you can, or at least clearly marked in source code so it can be #ifdef'd out.

Great work BTW :)

-Nic

[mean]

I'd be interested to check your modifications.
Fixounet at free dot fr.
Thanks.

You are right, "short term" is not a good explanation.
I should have said blind.

As you said earlier, it works only if there is no big surprise in the current and very next images.

I did some testing some months ago, in the same type of code that you did. Project the next half gop frames using the current quantizer and it it is too low increase it, and this increase is slowly removed
(-1 quant per frame).


For interlaced encoding, the issue is that there is twice as much field with the I&P&B relationship. It means the code must be changed to
keep fields in the VBV buffer longer as they are needed longer

The present code handles field as special frames, so it will discard the
frames from VBV buffer at each new input, the relation ship is complicated in that case.

[Peter Cheat]

Modified Libavcodec Site (http://petercheat.host.sk/libav/) has been updated with a new DLL (FFMPEG), Mencoder (generic cpu) and also the modified source. The bitrate spikes should now be significantly reduced. If its not good enough yet, give me details about which frames are causing problems (mostly I frames) and by how much. I have one spike at 6000kbit/s when the max was set at 2600kbit/s, which probably isn't good for old dodgy hardware players. Anyway, this should be better, not perfect yet, but being worked on. Also added Mplex 2.0, a better muxing tool than some others.

[dragongodz]

probably isn't good for old dodgy hardware players
actually the player does not have to be old or dodgy at all just because it doesnt accept or displays a problem with out of spec streams. standards are set for a reason. :)

on the site you ask about CCE. well in the DVD-Rebuilder forum its been shown that CCE can spike past max bitrate occasionally aswell. what was the result ? problems on playback on certain players, certain models of sony and pioneer from memory. not exactly dodgy hardware. :)
jdobbs reduced the max bitrate setting DVD-RB set to help with these problems.

also for those testing for dvd output they can use this
http://users.adelphia.net/~liquid64/dvdauthorgui.html
to author. it uses mplex aswell. :)

oh a side note about 1 pass. Michael(of FFMpeg) was talking about doing per MB rate control which would be suitable for both 1 pass and 2 pass. just another idea in the pot. :D

and thanks for all the work you are doing.

[Peter Cheat]

How would macro block rate control work? Does this mean assigning quatisers to each macro block of a frame rather than a frame?
I can sort of see some benefits, but it probably won't improve predictability of a frame size...

The reason I said "dodgy hardware players" was because when engineers are designing hardware (engineering being the profession I am studying at uni), it is important to expect the unexpected. Sony and Pioneer are expensive brand name DVD players, yes true. But the actual mpeg decoder they have inside may be the exactly the same as the el cheapo players that cost 75% less... A strictly valid stream is hard to create, so a player expecting a stream to be 100% compliant is a little to idealistic, if not outright stupid :) I've tried my encodes on 5 different players, newer models all played fine, only a friends 4-5 year old DVD player than only supported VCD didn't want to play it (not surprisingly). Anyway, is the updated code better?

[dragongodz]

Does this mean assigning quatisers to each macro block of a frame rather than a frame?
more of a bitrate allowance/sum and adjustment. so like how you do a few frames and then work out what the other frames in the GOP come out as(size wise etc) based on those and then adjust quant if needed. same type of thing but do a MB and work out the rest at that quant, if not desired result then adjust quant accordingly etc.
Michael didnt go in to it in detail but that is what i am assuming. i did actually read some material on per MB and per frame rate control some time ago, will have to see if i can dig it up again for you. :)

it is important to expect the unexpected
true but the whole point of setting a standard is that there should be no surprises. or if there is then minor ones. as such most modern players do handle out of spec streams reasonably but not all and its never quarenteed. so you really cant critisise a manufacturer that works to the standard only as its not their fault if someone creates out of spec material.

the actual mpeg decoder they have inside may be the exactly the same as the el cheapo players that cost 75% less
yes there are only so many chip makers, i doubt many of the cheap dvd players use pioneer chips though. pioneer i believe do generally make their own chips so again not exactly dodgy. :)

a player expecting a stream to be 100% compliant is a little to idealistic, if not outright stupid
no its what the whole point of standards is about. take the CCE spiking problem i mentioned. now that only occasionally happens and from memory it was with 2 pass and max set to over 9000kbit. set slightly lower or maybe even doing 3+ passes and it wasnt a problem. i have never heard of this problem with Tmpgenc or Procoder by the way but who knows.
thats why i said 3 pass MAY be advantages in the poll thread you posted. professional producers do not seem to have any problems producing standard streams either. :D

Anyway, is the updated code better?
well i have already said i wont get time to do any testing until sunday, even then it will probably be limited. so anyone else ? come on guys help make a free encoder even better.

[dragongodz]

bump so people notice the update. :D

[lamster]

Thanks for the update!

Is there a discussion anywhere of why we might want to or not want to use the custom matrix?

[jdobbs]

Thanks, dude. Nice work!

I've already updated DVD-RB with the version check and the cmatrix commands, but won't post until tomorrow. Bad timing on my part.

[dragongodz]

Is there a discussion anywhere of why we might want to or not want to use the custom matrix?
it has been discussed in both the xvid and cce forums in the past. using a custom matrix such as the built in QLB one is good for low bitrate encoding. example - you encode a file at a constant quant and it ends up size A, do the same but use a custom matrix such as QLB and the same file turns out 10% smaller(will vary depending on footage etc) but look the same unless you zoom in for the tiniest details.

Bad timing on my part.
nah bad timing by us both i guess. :)

[dragongodz]

just curious on the aspect ratio. so is it all ok now for everyone or still a problem i can try to replicate ?????

[jdobbs]

Ok, I'm confused. I tried 0.54 and trying to use "-cmatrix" or "-nocmatrix" fails -- and "-qlb" or "-noqlb" works... am I missing something here?

On the bright side, the new version works with DVD-RB 0.57 for that reason.

[dragongodz]

hahaha you are right. i missed 2 changes. bugger. see now thats what happens whena person rushes to get a release out before another program is updated(meaning DVD-RB). bloody failed on both counts. :D

no rush for me to fix it is there ? it can wait a week say so i can actually make some other changes ?

ok just updated the first post history. :)

[jdobbs]

Wow, you're good. Even your missed changes are helpful. That saves me from having to put an interim DVD-RB version out also!!!

[lamster]

Just tried using the latest (0.54), and got a Priviliged instruction exception (0xC0000096) at address 0x10176343. This was in avcodec.dll, which was loaded at 0x10000000; the instructions at the place where the exception occurred are:

10176343 66 0F 7F 4A 00 movdqa xmmword ptr [edx],xmm1
10176348 F3 0F 7E 53 40 movq xmm2,mmword ptr [ebx+40h]
1017634D F3 0F 7E 43 48 movq xmm0,mmword ptr [ebx+48h]
10176352 66 0F 6F 59 20 movdqa xmm3,xmmword ptr [ecx+20h]
10176357 66 0F 6F 79 30 movdqa xmm7,xmmword ptr [ecx+30h]


I was running it from DVD Rebuilder 0.57, on a Windows 2000 Professional system. Let me know if you need any more information.

[dragongodz]

you wouldnt happen to running on a pentium 4 by any chance ?

[lamster]

Originally posted by dragongodz
you wouldnt happen to running on a pentium 4 by any chance ?

Got it on the first try! :)

Interestingly, the error happened on the second file that Rebuilder invoked QuEnc on, and after I killed the debugger, it processed a few more uneventfully before I killed everything and restarted.

I'll know in a few minutes if it's repeatable...

[lamster]

Originally posted by lamster
I'll know in a few minutes if it's repeatable...

This time it got past that point without problem?!?

[dragongodz]

i also just uploaded tp the QuEnc download page the avcodec compiled with the memalign hack. so if you have any more problems can you test that ?

if that doesnt completely clear it up then next release i will go back to a newer cvs release. i assume 0.53 didnt have any problems ?

[EpheMeroN]

I shall do a sample encode to 1/2 D1 NTSC MPEG-2 in the morning to verify if the AR issue has been resolved :)

Keep up the good work on this awesome encoder.

[Peter1234]

dragongodz,
I tested QuEnc 0.54 with memalign hack avcodec.dll on P4 and there were no SSE2 problems. Also encoded 352x480 NTSC MPEG-2 and it plays in WinDVD with proper aspect ratio.

[EpheMeroN]

Okay, I just ran a test on QuEnc v0.54 and the 1/2 D1 NTSC Aspect Ratio Error is fixed! Good job =)

[dragongodz]

thanks guys, good to know. :)

[insanescape]

Question about QLB, what it does, and what exactly comprises "Low Bitrate" when dealing with DVDs?

I've got a movie that's going to stretch it, about 190 minutes or so, probably around no more than mid 2-ish for the bitrate. Should I turn on QLB or just leave the regular?

[dragongodz]

in simplest terms it will drop some very fine detail allowing for better compression.

my best advice is to do a small clip or 2 both using it and not. then compare them playing and see how much difference you can see. if you cant see any real difference then using QLB is definatly a good idea for lower bitrates.

whats a low bitrate foe dvd ? well around 3500 and up would be the majority of standard dvds i would think. so once you start to go under 3000 you head towards a low bitrate where picture quality will start to degrade noticably.

[EpheMeroN]

I'm getting an error when trying to encode a video:

AVS File is not outputting
(Use ConvertToYV12() at end of script)

I don't understand becase the last line of my script is ConvertToYV12(Interlaced=True), yet I'm still getting that error. I'll test on another machine w/ same input video to narrow down the issue.

[dragongodz]

are you using avisource() in the script ? there is a known bug that it wont do 2 passes with that. directshowsource() works fine however.

also how many frames is it ? another known problem is 1 or maybe 2 frame sources also not doing 2 pass.

[EpheMeroN]

I was using AviSource in the script. I switched to DirectShowSource and it's still giving me the ConvertToYV12() error.

The video is 1229 frames so the other issue is not present here either...

Okay I just tested something and fixed it. I tried dragging the AVS Script into Vdub. It told me that PeachSmoother needed YUY2 color format and gave me an error. Usually my captured AVI files are in YUY2 colorspace natively but the video I was dealing with I edited and processed out of Vegas Video for a DVD I'm working on. Vegas Video made it RGB32 colorspace. QuEnc for some reason gave me a YV12 error but I believe it was trying to tell me PeachSmoother needs YUY2 like Vdub said.

I tossed a quick ConvertToYUY2(Interlaced=True) in my script and it fixed everything.

[dragongodz]

ok. probably need to change that warning then to say there is a possible error in the avs and that you may have to add ConvertToYV12() at the end if it isnt.

[Peter Cheat]

Doesn't AviSynth pass the details of the error to QuEnc???
If it does, then just mirror the message to the user. Couldn't be too difficult I imagine.

[dragongodz]

i just had a quick look at the avsreader and the only functions it has at the moment is for getting the frame size, frame rate and total frames etc. you would have to ask Nic about adding getting the error message as i havent really looked at the source since 0.54 and wont for awhile. i am currently busy trying to squeeze in something else. :)

[Nic]

It's tricky to get the error from avisynth...also QuEnc reports a possible YV12 trouble, with all avisynth errors (because the error is being output in something other than YV12). So it can be confusing. Ill try and remedy that...

-Nic

[Mug Funky]

one question:

quenc works very well in interlaced mode (target svcd bitrate), but is there a way of selecting alternate scan? is it even necessary?

i'm a little clueless about mpeg-2 interlacing as there is a lot of conflicting information about how it is actually done (like field-based motion, field pictures, alternate scan, or all the above).

[Nic]

Well i'm a bit clueless about it all too ;) Hence, why alt-scan isn't an option, but I could add it if you want/need it?

-Nic

[dragongodz]

"MPEG-2 defines two different zigzag scanning orders: zigzag and alternate scans as shown in Figure 15. The zigzag
scan used in MPEG-1 is suitable for progressive images where the frequency components have equal importance in
each horizontal and vertical direction. In MPEG-2, an alternate scan is introduced based on the fact that interlaced
images tend to have higher frequency components in the vertical direction. Thus, the scanning order weighs more on
the higher vertical frequencies than the same horizontal frequencies. In MPEG-2, the selection between these two
zigzag scan orders can be made on a picture basis."

taken from
http://www.ee.eng.chula.ac.th/~supava/doc/mpeg12.pdf]

i also read elsewhere that alternate scan can even give better quality with progressive mpeg2 encoding but that seemed more of an opinion since no hard data was given. :)

[Mug Funky]

thanks, dragongodz :)

so this would help then?

but if it can be set per-frame, it might be useful to allow QuEnc to handle it (maybe that would require avcodec to be modified).

AFAIK DV video does this per-block, but that's probably a bit much (and wouldn't work in mpeg-2).

[edit]

about it having better quality on progressive... that sounds a bit naff. i could only think of it looking better on NTSC where vertical res is more important, but i would think the chroma would be messed up and offset the very slight possible benefit.

rather than adding an alternate scan checkbox, it should probably just be tied to the "interlaced encoding" option. i think nic said he didn't want the interface to get cluttered. i suppose it makes sense - if people want to tweak, they can do it with ffmpeg at the command-line.

[dragongodz]

so this would help then?
with interlaced encoding it would seem so.

but if it can be set per-frame, it might be useful to allow QuEnc to handle it (maybe that would require avcodec to be modified).
hmm well for static/near zero motion scenes zig-zig may do better. without looking at the avcodec source i doubt it does per frame. so i suspect its an either/or case.

about it having better quality on progressive... that sounds a bit naff.
ye i found it strange which is why i said there was no data to prove that. i will look in to it more when i get the time though. :)

rather than adding an alternate scan checkbox, it should probably just be tied to the "interlaced encoding" option.
i agree.

i think nic said he didn't want the interface to get cluttered.
yes plus also make it straight forward enough for anyone to use. give people 100 options they can fiddle with and many will f*ck it up. many options can be set by just following certain basic rules. using alternate scan for interlaced mpeg2(no checkbox) is a perfect example. i have read comments by many "experts" who really know bugger all. you would not believe some of the ideas people have, they want to be able to "tweak" everything they can, no matter if its more likly to mess things up. :D

[Nic]

So what's the concensus then? Should I turn it on when interlaced encoding is turned on? (U know me ;) I like to keep it to as few configurables as possible...so don't really want to add an extra option)

-Nic

[Kika]

My suggestion:

If interlaced is turned on, use alternate scan, if it is turned off, use ZigZag. Many Encoders are doing it this way.

And if you like, include an Expert-Setting-Mode where people who like to play can choose their own settings. ;)

[Mug Funky]

@ dragongodz: yeah, tweekers can often do more harm then good.

just look at the LAME encoder... no matter how many times it says "use the presets", people will just download the program, and type "lame -?" and then complain to the devs that their encoder sucks :(

[edit]

and i'm for tying alternate scan to the interlaced button. there's no good reasons not to from my (very limited) research.

[dragongodz]

Should I turn it on when interlaced encoding is turned on?
well i vote yes.

people will just download the program, and type "lame -?" and then complain to the devs that their encoder sucks
yes thats exactly what we would rather avoid. :D

[EpheMeroN]

I also agree that "Alternate Scan" should be enabled when checking the "Interlaced Encoding" Mode.

When I used CCE months ago, switching from "Zigzag Scanning Order" to "Alternate Scanning Order" always yielding me more quality whe dealing with Interlaced Material.

On a side note, I have a ??? about DC Precision. I've always set it to 10 for everything I've ever encoded. What are all the pros/cons to using a DC Precision of 8 or 9 besides slightly faster encoding time?

[dragongodz]

What are all the pros/cons to using a DC Precision of 8 or 9 besides slightly faster encoding time?
what it comes down to is quality(at least in theory). higher dc precision should give you slightly better quality.

there are limits to what can be use for what of course. example - vcd uses 8, svcd can use 8 or 9, dvd can use 8,9 or 10. dc precision 11 is used for HighProfile@ (such as 20Mb/s or more)