QuEnc 0.59 Beta 4

[onesoul]

@dragongodz

Maybe the missing frames could be related not to the encoder as jdobbs says in the following quote (which can be found here (http://forum.doom9.org/showthread.php?s=&postid=639885#post639885)) :
Actually I gave it some thought last night and realized there are instances -- usually at the end of a segment -- in which bad cuts might make it impossible to decode and present a frame (or even two) to the encoder. The frame would show in the overall frame count -- because that is created by counting picture start codes, but would be dropped. The only alternative in those circumstances would be to present and encode garbage. An example might be a B-FRAME when a cut point was made at an open GOP. I've also seen other examples of forward references against which there are no forward frames.

In those instances the drop of a single frame isn't significant -- and the PTS is reset at the start of the next segment. So what I'm going to do to cover these possibilities is to, instead of aborting, make a note in the status box and continue. If the difference gets above two frames I will continue to stop the encode.

The reason I added that sanity check in v0.83 was because CCE v2.70 appears to be bombing fairly consistently due to changes in the way it handles high-demand frames and balances them against the maximum bitrate. This "catch" code is there to make sure I'm not passing over segments that might be a thousand frames short because of the crash.

I'll modify the code for v0.85.

[dragongodz]

ok just tested new RC version. all good. no crash and all frames encoded. :D
oh and no more III either. :cool:

Maybe the missing frames could be related not to the encoder
nope this was Nics fault. :devil: ;)

i just tested the release version of HC 0.13 aswell. for results of that and download link go to the "who wants to try a new mpeg encoder" thread. hmm may suggest he start a new thread with that version with a better title. :)

[onesoul]

I had understood that dragongodz, just was trying to point some situations (like the early hc) when the encoder couldn't decode a frame, not being his fault, which apparently can be solved, (maybe the way jdobbs described).

But then again I am just passing information, I am not really in the field :).

[dragongodz]

ah but thats the thing. the last frame with HC 0.12 is decoded and encoded, its just the quitting that no longer works since i got that beta. doesnt matter now since 0.13 release version is working fine. so enough talk of HC here methinks. :)

[Nic]

Well, if I don't hear anything bad about QuEnc from you two, RC2 will probably become the released 0.60.

-Nic

[Koepi]

Gimme an hour to test please :-)

Cheers
Koepi

[dragongodz]

i would have liked a chance to test with authoring aswell(after wasting all that time) but since the peaks are lower etc i am easy. definatly is an improvement to RC by my books.

things like tighter control (maybe foxers strict scaling would help with this too) etc can be looked at later anyway.

so i am easy. :D

[Nic]

@dgz & Koepi: Take as long as you want. No rush. Just adding Mean's pulldown patch now, so got to test that first anyway. Keep testing, if you can, there's no rush to get this out of the door.

[dragongodz]

cool. i have some spare time tommorow morning (currently very late night here) so i will do some more tests then and let you know.

[Koepi]

Ok, works fine as it should! :)

No crashes, nice constant quality from a short visible test. IMO you can put it out :)

Cheers
Koepi

[freelock7]

-Nic & Dragongodz
Any chance to test a beta5(?) with the new RC2-last- version before the official 0.6?

[Nic]

@freelock: Looks like I can't PM you. So a bit tricky to send you the link.

[freelock7]

Sorry! It is corrected now!;)
Before all, THX again to let me try the RC2!
The main change comes from the bitrate distribution. It moves much more with a linear quant.
The fast scenes encoding gives better and better results but it could affect some quite scenes with a lower bitrate to compensate the high bitrate distribution.
Some scenes are encoded with a min_bitrate of 700! It is too low (macroblocs).
Maybe you could correct it with a min_bitrate setting?
Settings: VBR=3000/High quality/GOP=15/Dct=10

[guada 2]

Hello NIC,

I have make a particular test with quenc4.
Would be without fear, these parameters are special GUADA 2.
Look at:

Test1:
mpeg2(svcd) /bitrate:180 kbps /vbr hq /treillis quant /2 pass /gop:12, max bframe:2 /dc:9


Test2
mpeg2(svcd) /cq:14 /vbr hq /treillis quant /gop:12, max bframe:2 /dc:9

Note:
Test 2, is 4 times faster than the test 1.un favours certain for a pass.
its size is identical to test 1 but -1. (Test2:4.75 mo, Test1:4.76 mo).
The quality of image is better than test 1. the macroblocs are almost not perceptible.


Interrogation:
The CQ would not be it preferable with mode 2pass, more rapid and more precis with this bitrate.
It is not a honourable test, but I took this example because the time of encoding is of primary importance for all one each one, quality and the final size is the business of all.

Which is your opinion on this kind of test?

[dragongodz]

The main change comes from the bitrate distribution. It moves much more with a linear quant.
yes. strict scaling would reduce this somewhat aswell and generally provide a more consistant distribution i would say. 1 thing at a time though, remember this is a pretty major change Nic has done.

Some scenes are encoded with a min_bitrate of 700! It is too low (macroblocs).
no it depends on the footage etc. you will see most encoders will use or reccomend a min bitrate of around 500 anyway. as i said above though, the distribution needs some tweaking/changing but that can come later. in my opinion this RC is working better since it is not spiking anything like the old RC can do.

ok Nic i did some tests including authoring. no complaints from other programs, results play fine, all looks good. :D

[guada 2]

Hello,

If I have make this test, it was to test the mode One pass VBR of NIC.
(ah! I forgot scene detection and max bitrate 180 kbps).
What I can say it, it is "as" effective as CCE.


In a word it is good work.
Remain, that this bitrate is not a working tool.
But my tecnique rests on low bitrate, in order to find precision adjustments for the high bitrate.

Bye

[Nic]

@All: My computer's proper dead now. :( So that will delay the next release somewhat. Also, got to get the source off the harddrive....

[dragongodz]

My computer's proper dead now
now that sucks. so what part died ?

[Nic]

Just had it up on the bench in engineering at work. Appears to be the motherboard. Or at least that bit is definitely not working, if another faulty bit caused it, I can't detect it (CPU & Memory in another machine worked fine).

Don't know how long it will be before I'm up and running again....
....Isn't this what roughly happened to Peter Cheat? Maybe it's the "RateControl Curse"

-Nic

[dragongodz]

Isn't this what roughly happened to Peter Cheat?
his cpu blew up.

Maybe it's the "RateControl Curse"
dont say that. look at what most of the work i have done with both Rejig and QuEnc have been to do with. :scared:

Don't know how long it will be before I'm up and running again...
ok, please try and keep us informed. :)

[video_magic]

Sorry to hear of your PC troubles Nic - not just because of Quenc development being temporarily halted, but generally ;)

couple of things I wanted to ask. I had almost forgotten about 'Closed GOP', and wandered what the status was and if it is yet implemented in a 'proper' full spec-compliant way suitable for authoring DVDs - I don't know if that is a libavcodec thing - I'm not very technical knowledgeable about all this sorry :(, I just read about it on here a few times and wondered about the current status.
Do we need Closed GOP for DVD-spec compliancy anyway? I can't remember that I ever made out a definitive answer from reading around

Second question - how do you work so damned fast!? :D This rate Control seems to have been a big problem for a lot of people for ages, will your solution be taken up by the team that uses libavcodec (ffmpeg?), I'm not clear on these projects but I do often try out Mencoder and it seems to be a related project also. Last I read in the Mencoder dosc it said Closed Gop not yet implemented.

[dragongodz]

'Closed GOP', and wandered what the status was and if it is yet implemented in a 'proper' full spec-compliant way suitable for authoring DVDs
not something i have looked in to so i wont comment.

Do we need Closed GOP for DVD-spec compliancy anyway?
it is required for things like multi-angle encoding but no not for something like a plain movie.

will your solution be taken up by the team that uses libavcodec (ffmpeg?)
chances are practically nil. they will want a LGPL rate while Xvids is GPL. if they wanted to add a second GPL only rate control which was only avalaible when compuiled as GPL they could have added/used Mpeg2Encs(for example) along time ago as that is meant to be ok. infact Michael said a long time ago he had half written a new/better rate control but has not got around to finishing it. when i say he said it long ago i mean LONG AGO, so dont hold your breath waiting for that either. ;)

[Nic]

@video_magic: I hate hardware troubles..grr. Although my computer has never been 100% stable, and it isn't due to overheating or overclocking (I never managed to track the problem down). So maybe it's been the motherboard all along. Got to go through Abit's RMA process. I'll see how successful that is....

Closed GOP is needed in the DVD spec, AFAIK. However, I believe it is only really needed on Chapter points, so the DVD player can seek to the exact point (LBA) and not worry about the GOPs before it. However, with Closed GOP on, I believe all GOPs QuEnc creates will be closed. Not really too bad a thing. (Also, scene detection gets turned off, not that bad a thing either, stops having multiple I-Frames one after another, etc)

Ratecontrol has been a big problem, but I never considered tackling it like this, the credit really goes to Mean and his avidemux (he should really changed the name of that App, it's a bit misleading). The 2pass ratecontrol is good now in the 0.60 Release Candidates, not perfect, but IMO libavcodec's 2pass ratecontrol is pretty worthless.

@freelock7: Considering the bitrate was 3000, it's not too surprising that it could go as low as 700. But maybe a minimum bitrate setting wouldn't be too bad...I wouldn't recommend it, but it might help. I'll test it when my computer's back up. If you have any more test results, let me know.

One of the longest posts I've ever written here...

-Nic

[video_magic]

Thanks for your answers :)

This is an exciting time for me that has been getting into DVD authoring and other things related to DVD back up and video editing, it seems like everything is happening here!

It is a hard area to become involved in because of so much information to intake and try and remember and make sense of, so many facts! I realise that many of you have a lot longer experience (so use of your time) and I'm very grateful for the help- and the brilliant programs too, maybe Nics computer (and Peter Cheats') died because of all the work being done on them :p

I think I dipped my toes into an area of politics with asking about different projects that might or might not want to integrate - sorry. I can see that there are individual visions of how peoples work wants to develop; I have a few times tried to get into coding and I realise it's a lot of hard work and dedication so I will step back from queries like that in future :devil:
Anyway, I think Quenc is aimed at people like me because of having a GUI, I would like still to be in some control so I think pop up help descriptions that briefly explain what the options mean would be nice for the user - it would also keep the user informed and help them gain knowledge and more appreciation for what they are trying to be involved in :)

[Nic]

"I think pop up help descriptions that briefly explain what the options mean"
They've already been added and will be in the next released version :)

[dragongodz]

maybe a minimum bitrate setting wouldn't be too bad...I wouldn't recommend it, but it might help.
it would be good for some hardware players that are meant to have problems playing if the bitrate drops below 500 or so(i think on a few it was below 300).

as for closed gops well i have never seen closed gop being required for chapter points, just that they need to start on an I frame(the chapter points that is).
my undertsanding is the same as what Arky says here
http://forum.doom9.org/showthread.php?threadid=85836
not that i would use closed gops all the time but when they are required and when not. :)

[Nic]

May well be correct, but I remember at least one DVD where closed_gop was only set on chapter points. I remember it from doing ReJig and trying to set chapter points correctly with authoring.

Got my old computer up and running. AMD 1800 XP, ATI Radeon 9500 pro. It's like 1999 all over again. I'll get my source code off my drive tomorrow (this old beast doesn't have SATA) and probably release QuEnc soon.

-Nic

[dragongodz]

I remember at least one DVD where closed_gop was only set on chapter points.
unless you see it consistantly it could be just the way it was authored or produced. for example ,from memory, most other encoders close the first GOP even when you encode open Gop. now imagine if for some strange reason they encoded each chapter seperatly, that would give you a closed GOP at each chapter, but it would not mean it was needed.

Got my old computer up and running. AMD 1800 XP, ATI Radeon 9500 pro. It's like 1999 all over again. I'll get my source code off my drive tomorrow (this old beast doesn't have SATA) and probably release QuEnc soon.
good news. :D

[freelock7]

Frame extracted from a slow scene

Bitrate/quant
__________________________________
B3 frame | 1301-2.48
RC2 frame | 707-4.40
NU0.01(min_bitrate=1004)| 1255-3.40
NU0.01(min_bitrate=4) | 889-3.20
HC013 1350-4.58
Inconclusive indeed but even though, it shows the compression level too high with RC2.
It wouldn't be a problem if the quantizer could go down but it stays far above.
Naturally, the best encoding comes from B3 and NU with a min_bitrate=1004!
Surprised to see that B3 and NU0.01 (min=1004) have the same settings!
It means the 059 versions have a min_bitrate fixed?

[dragongodz]

freelock7 - but how do they look playing ? numbers alone dont say what effect it has on your viewing pleasure. if it is hardly noticable and you have to really look for it to see a difference then it is not really a problem.

if you read back you will see i have said though that RC will still need a bit of refinement/tuning. in general though it seems to me to be better because it is not spiking like old RC can do and i didnt notice any real quality difference in playback of the clips i tested with.

It means the 059 versions have a min_bitrate fixed?
no. min bitrate is 0 as it always has been. using anything else with libavcodec can throw the RC for loops and totally stuff your bitrate target.

[video_magic]

I apologise in advance for the lack of my technical knowledge in this area but I want to understand how I,P & B-Frames and Closed Gop works. So, it's all to do with I, P and B Frames right?

I think (at the moment) that I-Frames are encoded directly from the source frame.
P-Frames use information both from source-frames and from I-Frames both preceding and after it.
B-Frames only use information from P-Frames - before and after? so can be bad quality.

So when a Gop is closed does this generally mean that scene detection is forced off and an I frame is inserted at the beginning of the next GOP that will not reference any previous frames - from the previous GOP I mean - I am not clear on this because I read somewhere that MPEG does not see frames as linear in order (is that just within each GOP?).

An easy to understand answer for a guy that is trying to begin to understand these matters would be appreciated :scared: :)

[gigaset2004]

@ video

Hi,

I am not am expert for mpeg , but your understanding of I,p and b frames is not correct.

So an I frame is the basic for the p and B frames and the I Frames includes the hole informations for the pictures. - like jpeg -

The p and B frames are only including the differences between the actually I and the following I frame.

That´s it in short words.

The big problem in the degenaration is that no encoder uses the information from the source.

The transcoder use all the info from the source.

An encoder is like a camera and do not care about the original I, P or B frames from the source.
This effect makes the degenarion from copy to copy.

Each encoder have to analyze the bitrates from the source to make a decision for the copy at your rage for Gop lenght ,bitrate decisions
and movements for the resolutions.

tempgenc has a small option to force I,p and B frames from the source but it is not possible to force that option for a hole movie.

The goal would be to take these info´s from the source to give it to the encoder.
If this would be possible we had no more Problems at all.

so i am no crack but i think the goal wold be to combine a program like rempeg2 with an encoder.

Sorry for my english - hope it helps

[Nic]

@video:
So when a Gop is closed does this generally mean that scene detection is forced off and an I frame is inserted at the beginning of the next GOP that will not reference any previous frames
Generally correct yes. Scene detection doesn't have to be turned off, but currently libavcodec does not support Scene Detection AND Closed GOP.
from the previous GOP I mean - I am not clear on this because I read somewhere that MPEG does not see frames as linear in order (is that just within each GOP?).
Yes that's just within the GOP (or at least a closed GOP)

AFAIK

-Nic

[freelock7]

freelock7 - but how do they look playing ?

LGX-Zeppelin crash-clip of 120sec
The Johnson Family vacation-full movie encoding

I chose the lowest scene (frame) and compared between each encoder.

[dragongodz]

video_magic - for a little understanding of the different frame types and what they do etc have a read here
http://www.faqs.org/faqs/compression-faq/part2/section-2.html

I chose the lowest scene (frame) and compared between each encoder.
yes i understand that but a single frame does not say what it looks like when playing. i have analysed a stream produced with the new RC and yes it does some large variance in frames but on playback i could hardly tell any difference comparing it to the old RC. i will try some different scenes again later and see if i can.
my question though was could you see any difference PLAYING and not just a still frame ?

[Trahald]

frames are stored out of temporal sequence.. they are stored in the sequence needed to decode them. usually i(1)p(4)b(2)b(3)p(6)b(5) etc.. (the numbers represent temporal playback) the i is the jpg..its needed first. the first b is next but cant be decoded with out the p so the p is stored next. then the b's. gops are stored with the i frame first (open or closed) the difference is on a open gop the temporal first frame is the b.. it needs the info from the last p from the gop before and the first I in the new gop i(2)b(1)p(5)b(3)b(4)p(8) etc..

[freelock7]

could you see any difference PLAYING and not just a still frame ?
Yes, of course (macroblocs). Compare withHC013. it's a good test.[

[dragongodz]

Yes, of course (macroblocs)
so the new RC showed more macroblocks than the old RC. ok will look at that as i said with some more very slow scenes later.

Compare withHC013. it's a good test.
i already know the types of GOPs HC is outputting and indeed the new RC is not as even a distribution yet, hence the need for tweaking etc. i dont need to look at another encoder to tell me that. :)
look in the thanks.txt that comes with HC 0.13 , dgz is me. ;)

[Nic]

Sounds like the RC is over-compensating a little on the low motion scenes. I'll look into that. Soon as I get the data off the HD i'll post you source code too dgz. It's pretty easy to get to grips with. (compared to the mess that's libavcodec RC)

Thanks for all the feedback guys. My motherboard is being RMA'd, but my old computer should be able to cope for now.

-Nic

[dragongodz]

ok did some tests with some low motion/near static scenes. yep new RC showed blocks worse than beta4, note beta4 had the blocks aswell just lighter. when there was some motion aswell, such as a person moving their hand however the reverse was true. that is beta4 showed more blocks and the new RC had less. which all confirms what freelock7 said basically. note though that i did have to look for these as they lasted a fraction of a second so would be hardly noticable if just normally watching. thanks for the concise reporting freelock7, made it easier to find. :)

koepi - if you are reading, where can Nic and i get the source for foxers strict scaling since searching for "foxer strict scaling" and "xvid strict scaling" doent really help. ;)

i'll post you source code too dgz. It's pretty easy to get to grips with. (compared to the mess that's libavcodec RC)
cool. :D

[Koepi]

Tomorrow i have a day off and then can send/upload the strict scaling code.

Cheers
Koepi

[dragongodz]

cool. :D

[freelock7]

-Dragongodz
Glad to know you validate my findings.:)

[EpheMeroN]

I'm gonna be encoding an XviD AVI using the latest QuEnc beta and wanted to know if I should do the levels correction (going from 0-255 to 16-235 for dvd output) in the AviSynth script, or if it's handled automatically in QuEnc?

[mikeytown2]

Awesome work on QuEnc, i have a couple of requests...

I have a dual AMD 2000+ computer (its a 4U server box i got for free :)) and when i use 0.59 Beta 4 it crashes when ever it starts the second pass, doing 2 vids at once. I'm encoding one video on each CPU (ctrl-alt-del, set process to CPU). I have not tried 1 pass and it looks like with this current ver it doesn't really matter, lol, seeing how 2 pass doesn't do too much according to the posts.

So I'll wait for 0.60 when the RC is working right and then I'll nag u guys some more, unless u would know why it would crash on the second pass. I have the .exe in 2 different dirs and the .avs files are in the same dir.

In case you guys are wondering what I'm encoding, I'm making a x-men cartoon dvd set. the 92-97 one. I'm using DVDStyler to author it.

My other questing I think is answered, I need to set the max bitrate lower. the vids where crashing on my dvd player, and would act odd on VLC.

thanks
-Mike

[dragongodz]

I have a dual AMD 2000+ computer
then i would wait until i have looked at adding some sort of threading and just do 1 encode at a time for now if you are constantly getting a crash running 2 QuEncs at once.

seeing how 2 pass doesn't do too much according to the posts.
well thats not quite true. all of libavcodecs rate control is not good. we have had to put certain contraints on it to behave a bit better but it still wont be perfect.

unless u would know why it would crash on the second pass.
bit hard to replicate and track without having a dual cpu system aswell. ;)

[communist]

I got the same thing yesterday when I tried to encode 4 svcds over night when running 4 QuEncs:
- running 2 (or more) QuEncs with both (or all) doing 2pass encoding -> every instance CRASHES when starting 2nd pass
- running 2 QuEncs with one doing 1pass and the other 2pass -> no CRASH

I'm running on:
XP SP2 / 1x AMD Athlon 2600+ / 768 RAM / no overclocking / properly cooled ;)

Other than that - keep up the great work on QuEnc :)

[Nic]

@communist & mikeytown2: This is because QuEnc has a fixed filename that it writes the 2pass log file to (inside your TEMP directory). I didn't ever consider people running two QuEncs at once. I'll make the next version randomly generate the 2pass file name, that'll fix it. Thanks for the feedback :)

On another note, should get my motherboard back today, I'm away all weekend, but come tuesday should be developing QuEnc again.

-Nic

[lark]

perhaps you could use the process id as a postfix for the log file...

regards
t :)

[communist]

Ah glad that its easy to fix. One suggestion: wouldn it be better to place the log file in the same directory as 'my_source.avs' and have it named 'my_source_pass.log'? That way one could easily recognize the log files and rerun 2nd pass when needed (for ex. bitrate change) easier :)
That would of course need a "run 1st pass" or "2nd pass"-only mode.

The other thing I noticed similar to the pass log file problem is that you can run 2 QuEncs (or more) and have them encode the same source.avs to the same source.mpg at the same time. Shouldn there be some protection saying "Hey that file is already in use!".