QuEnc 0.70b: Very bad artifacts...WHY??

[drcl]

Encoding a capture of Back to the Future from analogue TV - fairly noisy. 544x576 huffyuv 25 fps 48000hz mono

avisynth script:

LOADPLUGIN("D:\Documents and Settings\DRCL\Desktop\cnr2_v261\CNR2.DLL")
LOADPLUGIN("D:\Documents and Settings\DRCL\Desktop\COLORMATRIX.DLL")

SEGMENTEDAVISOURCE("D:\CAPTURE.AVI")
DELAYAUDIO(-0.20)

CONVERTTOYV12()

A=FADEIO0(TRIM(208,24504),5)
B=FADEIO0(TRIM(31004,85354),5)
C=FADEIO0(TRIM(91854 ,130604),5)
D=FADEIO0(TRIM(136854 ,176685),5)

ALIGNEDSPLICE(A,B,C,D)

CNR2()

LETTERBOX(64,64,16,16)

#TEMPORALSOFTEN(4,4,8,15,2)
#SELECTRANGEEVERY(15*25,15)
ColorMatrix( mode="Rec.601->Rec.709")

Quenc settings:

mpeg-2, 1850kbit, VBR, HQ, 2 pass, GOP15, 2 b-frame, scene detection, 4:3 AR, DVD mux profile, 8 bit DC, 2000 max bitrate, Sharktooth's EQM v3ULR CQM, mp2-128kbit audio

Here is a 50 frame xvid Q1 sample. Bttf.Avi.Zip by Bigupload.Com (http://www.bigupload.com/d=2ECC3C69) (1.1MB)

this happens all the way through the encoding.

EDIT: It doesnt happen with QuEnc 0.54 which uses an earlier CSV.

[digitalman]

I am having the same issue with major artifacting.

[manolito]

You say: Fairly noisy clip, average bitrate 1850 kbps, max bitrate 2000 kbps. I think this is a very demanding task for any encoder. (And the target is DVD, not SVCD?)

The reason why QuEnc 0.54 does a better job here seems paradox, but it is because version 0.54 has a very bad rate control. QuEnc 0.54 will not care too much about the specified max bitrate, which in your case is a good thing because complicated scenes will get more bits. You will probably get the same result in QuEnc 0.70b if you set max bitrate much higher (8000 kbps or more).

Why is your average bitrate so low? Is "Back to the future" more than 5 hours long? A 2 hour movie on one DVD5 at a size of 720 x 576 will result in an average bitrate of almost 5000 kbps.


Also some general tips for ultra low bitrate encodes:

You have to denoise the clip before encoding. Noise is hard to encode, life gets much easier for the encoder with a properly denoised clip. Just using CNR2 is not enough.

Use an agressive low bitrate matrix. I am not familiar with Sharktooth's EQM v3ULR matrix, but Amnon's AutoQ2 or even BVCD are good choices for very low bitrates.


Cheers
manolito

[drcl]

Yes some are the settings are odd. I wanted to make X-SVCDs that could easily be transferred to 1/3 dvds later - that's why there is such a low bitrate.

As to the noise issue, it is very well represented except in those frames. I did use some noise reduction as i captured with Vdub. I'm not looking for perfect quality so I cant be bothered to set up denoisers, just something that is watchable.

The matrix is an ultra low bitrate matrix which i find preserves alot of detail.

If it wasn't clear, the artifacts are more prominent in the higher motion scenes but still obvious in low motion ones too. It's very much like a pumping effect but worse than anything i have ever seen.

Maybe i should have used the SVCD mux profile. That was an oversight by me, but i dont think it should cause this problem.

The 2000kbit bitrate is conservtive because of the rate control never keeping to the specified one. I'll try 2500kbit then to see if the rate control is sincere.

If this doesnt work, i dont see how Quenc can be used to make SVCDs.

On a final note, when i view the file using FFDshow, with quantizer visualisations, I get quantizers of 62 :confused: in those very bad frames... God knows what that means!

edit: just checked the bitrate on the original enocoding with bitrate viewer and it peaks at 2448. so much for rate control.

[foxyshadis]

Your b-frame max quants are going way too high. (Any "pumping" bad quality is usually caused by b-frames at high quants.) You need to limit them down to 40 or so, depending on what you can get away with while maintaining the bitrate.

[drcl]

OK. And how am i supposed to do that?

I though quants only went up to 31 anyway?

Its not only b-frames.

here's a typical sequence of quants suggested by FFDshow visualisations starting with an I-frame.

62
62
62
62
62
62
62
62
62
62
62
62
62
62
62
20
4
4
4
4
4
4
4
4
4
4
4
4
4
4
6
12
12
12
62

something is seriously wrong here. seems like a whole GOP gets a Q of 62?

I'm pretty sure a CBR encode wouldnt look this bad.

[foxyshadis]

I think dragongodz once explained it as something to do with how lavc decodes mpge2, internally doubling the quants, so it's actually using 31 and 2 and so on. But that is pretty insane. Ouch, hopefully he'll have an idea why.

[Mug Funky]

MPEG-2 is just no good at this bitrate unless you severely reduce the frame size.

i thought it was a matter of macroblock quants actually going up to 62, and frame quants going to 31. so both nonlinear and linear quantiser scales would yield the same range of block quants, but would map to different frame quant values.

so block quant 62 = frame quant 31... i suspect if quenc was writing quants outside the range of mpeg-2 that it simply wouldn't decode.

the bitrate is much, much, much, much too low. PAL analog TV is probably going to be interlaced (due to NTSC to PAL conversion), and noisy. 2000kbps is simply nowhere near enough for that, CBR or not.

try scale down to CIF size (352x288) and encode in mpeg-1 for VCD if you insist on such a low bitrate. or split it onto more discs.

[dragongodz]

just checked the bitrate on the original enocoding with bitrate viewer
i personally do not trust bitrateviewers results. try confirming it with something like mpeg streameye or even DGmpegdec. and while your at it check what 0.54 did aswell.

Sharktooth's EQM v3ULR CQM
also try using a different matrix. this was of course is an ultra low matrix made for Xvid. i can not guarentee results using custom matrices.

seems like a whole GOP gets a Q of 62?
which suggests that section/GOP has a lot of motion(real or noise etc) so the ratecontrol is doing its job to trying to limit its size to your set max bitrate.

so much for rate control.
tell you what, when you feel like rewriting it for us just let me know. if you cant then keep your smart comments to yourself.

nonlinear and linear quantiser scales
yes this was talked about in another thread. people really shouldnt look at these values and try and guess what they mean without actually knowing. ;)

finally, i DO listen to constructive criticism. i DO NOT listen to smart remarks or people being insultive when they dont know what they are talking about.

[drcl]

@ dragongodz:

my comment about rate control wasnt intended like that. Manolito had referred to 0.54 having very bad rate control. But it cant be the cause.

I have encoded movies with 0.54 for SVCD with even lower bitrates (If i remember right I actually did matrix reloaded at 1400!) and they splutter slightly in parts on my DVD player when the bitrate peaks, but there was no artifacting anything like this. If you look at the clip you will see that its not particularly high motion. And the same blocking happens in some of the most low motion scenes also.

How can a whole gop be encoded at Q31? and the next be encoded at 2 in the same scene?

I have installed FFDshow and FFvfw and have used mencoder previously. I have played around with the settings in those. Do they interfere in anyway with quenc?

mpeg stream eye:

file type : MPEG2(PS)
video stream type : MPEG2
resolution : 544x576
profile:level : Main:Main
aspect ratio : 4:3
frames count : 157 231
max frame size : 47 104
min frame size : 0
file size : 1 601 374 208
--------------------------------------------

framerate declared : 25.00
--------------------------------------------
real : 25.00

bitrate declared : 2 000 000
--------------------------------------------
real max : 2 868 000
real avg : 2 036 800
real min : 800 000
--------------------------------------------


I'll try resetting to defaults and using standard matrix

edit: still the same

[drcl]

used HC in the end and it did it fine. I dont really know what is wrong. anybody?

[drcl]

I notice that this only problem of very high quants happens when i do the full movie which is 1h45m.

If i do a selectrangeevery(25*45,15) then i dont get anything like the same quants.

Because of this i dont think anyone would notice when using DVD rebuilder.

I'm going to try encoding a full DVD of a bryan adams concert to rule out or rule in the possibility that it is the noise.

[drcl]

still the same problem..... can anyone else reproduce this? Anyone have any suggestions why it's happening?

[M4ST3R]

Version used: 0.70b, haven't tried 0.71 yet

I have the same problem.
Target bitrate 2280, max bitrate 2600, for svcd output. I tried high quality, trellis, 2 pass, extreme & slow settings, scene detection, qlb matrix,... but it didn't help. The source is a xvid, 1050 kbps.

Then I tried ffmpeg (that I used before I found out QuEnc) and I was surprised to see that there's no such major artefact.
I tried to narrow down the bug and maybe (I'm a newbie) I found the reason:
According to virtualdubmod, the video encoded by ffmpeg contains NO b-frames, contrary to the one encoded by QuEnc. Being a newbie, I have no idea if it's a good thing or not (to have b-frames) but maybe it's better to avoid them for such low (SVCD) bitrates.

Then I set Max B-frames=0 in QuEnc to see if it fixes the artefact.
[mode off-topic on]
Maybe (I'm not 100% sure because I was away from keyboard when it happened) I found a bug: I check QuEnc's window during the 1st pass and everything was fine. When I went back later the encoding was finished. The problem is that the size (and the lenght) of file is only about the half of what it should be. It seems the encoding ended during the 2nd pass (without crash).
By the way, I couldn't run a 2nd pass with ffmpeg: ""2pass curve failed to converge". Maybe it's why QuEnc stopped during the 2nd pass, I don't know... I found this thread at the ffmeg-devel mailing list (http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2006-July/013004.html). It says it could explain why the output is often undersized.
[mode off-topic off]
I checked this (partially) encoded file (which according to virtualdubmod contains no b-frames) and the artefact is gone, like with ffmpeg. So I think it's better to have no b-frames (at least in my case)

@ dragongodz:
If needed I can provide the avs script, the xvid source, the file encoded by ffmpeg (no artefact) and the file encoded by QuEnc. I can upload clips or the complete videos.
I hope my message could help you to improve even better QuEnc.

@ drcl:
Try to Max B-frames=0, it fixed my problem.

[drcl]

i was under the impression that b frames were very useful at low bitrates. one thing i did notice is that 0.54 uses only 1 b-frame.

With no b-frames the artefacts may be absent but i would expect a huge drop is picture quality overall.

[dragongodz]

yes B frames are generally very useful. they are however usually the hardest hit trying to keep bitrate limits.

please try QuEnc 0.71 now aswell as it is the latest and i did change a couple of minor things with 2 pass ratecontrol.

[M4ST3R]

Tried 0.71

The quality is much better, the blocks are gone! Good work! It's better than ffmpeg too. Indeed ffmpeg's output contains no b-frames but there's another annoying artefact.
By the way 0.71 seems faster. I say faster because I didn't time it and I had other tasks running during the encodings. It's just a feeling.

But the bitrate control is worse. Same source, same settings (only replaced the exe), bitrate=2280, max bitrate=2600

0.70:

virtualdubmod stats

480x576, 25.000 fps
# of frames 60639
number of I, P and B frames 5054/15161/40424
I-frame min/avg/max total frame size 4349/24248/66001 119682k
P-frame min/avg/max total frame size 324/14011/70277 207446k
B-frame min/avg/max total frame size 1044/8196/48159 323559k
average bitrate 2198

mpeg streameye stats

file type : MPEG2(PS)
video stream type : MPEG2
resolution : 480x576
profile:level : Main:Main
aspect ratio : 4:3
interlaced : no
frames count : 60 640
frame size max : 79 016
avg : 12 245
avg/max (I) : 26 812 / 74 368
avg/max (P) : 15 565 / 79 016
avg/max (B) : 9 179 / 53 452
min : 0
file size : 742 583 072
--------------------------------------------

framerate declared : 25.00
--------------------------------------------
real : 25.00

bitrate declared : 2 600 000
--------------------------------------------
real max : 3 315 400
real avg : 2 449 000
real min : 597 000

0.71:

virtualdubmod stats

480x576, 25.000 fps
# of frames 60639
number of I, P and B frames 5054/15161/40424
I-frame min/avg/max total frame size 4349/23677/57361 116860k
P-frame min/avg/max total frame size 324/13091/70274 193831k
B-frame min/avg/max total frame size 1044/7339/36297 289724k
average bitrate 2028

mpeg streameye stats

file type : MPEG2(PS)
video stream type : MPEG2
resolution : 480x576
profile:level : Main:Main
aspect ratio : 4:3
interlaced : no
frames count : 60 640
frame size max : 79 016
avg : 11 387
avg/max (I) : 26 213 / 62 748
avg/max (P) : 14 626 / 79 016
avg/max (B) : 8 319 / 41 832
min : 0
file size : 690 558 008
--------------------------------------------

framerate declared : 25.00
--------------------------------------------
real : 25.00

bitrate declared : 2 600 000
--------------------------------------------
real max : 3 487 000
real avg : 2 277 400
real min : 558 400

off topic:
What's the best way/software to get the average bitrate of a video? I tried virtualdubmod, mpeg streameye, bitrate viewer, mpeg validator, they all give different stats. virtualdubmod seems to give the most accurate results (close to what I compute manually). I don't understand how mpeg streameye gives an avg bitrate of 2449000 (0.70)...

[dragongodz]

The quality is much better, the blocks are gone!
But the bitrate control is worse.
hang on, so it looks better but undersizes a bit so its worse ? you just cant please some people. :)

i did expect the max bitrate could be a little higher but its not by too much so thats not bad so far. the under bitrate i will look at later but consider that since it has fixed the problem you were having its atleast a step in the right direction.

I don't understand how mpeg streameye gives an avg bitrate
mpeg streameye is giving you the total average. that is not only the video but also audio and muxxing. so to see just the videos bitrate stats you must demux it(with something like Rejig) and then feed that .m2v in to streameye. you will see the average bitrate given by virtualdubmod and streameye are about the same. you of course will also see the real maximum bitrate for the video.

[M4ST3R]

hang on, so it looks better but undersizes a bit so its worse ? you just cant please some people. :)

i did expect the max bitrate could be a little higher but its not by too much so thats not bad so far. the under bitrate i will look at later but consider that since it has fixed the problem you were having its atleast a step in the right direction.
Sorry if my previous post looks a bit harsh to your eyes, it's not what I wanted, I said "good work". I meant the bitrate control itselft is worse (since it's more undersized than 0.70), not the overall quality of 0.71. So of course I consider it's a (big) step in the right direction. By the way I didn't post the virtualdmod and mpeg streameye stats to "complain" it's more undersized but to give informations about what has changed between 0.70 & 0.71's outputs. I thought it could help you to understand why the blocks are gone and why the avg bitrate is lower (actually since you're the coder, I guess you already know the reasons of the changes). Actually being a newbie, I don't know if these stats are useful or not.

I'm going to encode other episodes of the same TV show, I'll try to play with some settings. Let me know if feedback would be useful

[drcl]

i still get the same problem original problem.


@M4ST3R

What settings are you using in quenc?

[M4ST3R]

@M4ST3R

What settings are you using in quenc?
MPEG-2
bitrate 2280
max bitrate 2600
use VBR
High quality
2 pass encoding
GOP size 12
max b-frame 2
DC precision 8
other settings default/unticked

0.70 / 0.71 comparison:

QuEnc 0.70 frame 58 (http://www.fileden.com/files/25538/58%20-70.png)
QuEnc 0.71 frame 58 (http://www.fileden.com/files/25538/58%20-71.png)
QuEnc 0.70 frame 563 (http://www.fileden.com/files/25538/563%20-70.png)
QuEnc 0.71 frame 563 (http://www.fileden.com/files/25538/563%20-71.png)
QuEnc 0.70 frame 40954 (http://www.fileden.com/files/25538/40954%20-70.png)
QuEnc 0.71 frame 40954 (http://www.fileden.com/files/25538/40954%20-71.png)
QuEnc 0.70 frame 43003 (http://www.fileden.com/files/25538/43003%20-70.png)
QuEnc 0.71 frame 43003 (http://www.fileden.com/files/25538/43003%20-71.png)

[dragongodz]

Sorry if my previous post looks a bit harsh to your eyes, it's not what I wanted, I said "good work".
and i put a smilie after "cant please some people" which should have told you i was not serious.

I guess you already know the reasons of the changes
yes. ;)

I don't know if these stats are useful or not.
Let me know if feedback would be useful
yes, useful to know in which direction things need to be tweaked towards. dont be frightened to say your findings so long as you give information such as settings etc. though people should remember that if i cant replicate it then its awfully hard to fix. as such i would probably pester them for more information etc.

[drcl]

@M4ST3R and dragoongodz

I did a another capture from the same channel. I encoded it with the settings M4ST3R used. The only one that is different is scene detection off.

Suprisingly this removed the artefacts.:)

[M4ST3R]

I did a another capture from the same channel. I encoded it with the settings M4ST3R used. The only one that is different is scene detection off.

Suprisingly this removed the artefacts. good news!

yes, useful to know in which direction things need to be tweaked towards. dont be frightened to say your findings so long as you give information such as settings etc
More feedback on the bitrate control. Stats from virtualdubmod.

Another episode from the same TV show, 40 mn 35s, source Xvid 1062 kbps. Same settings than in my previous posts.

Target: SVCD, 80 minutes CDR, bitrate 2474kbps (expected filesize 800MB)

QuEnc 0.71: 2046 Kbps, filesize 666MB. bitrate about 17% lower than expected
QuEnc 0.71 QLB matrix: 1717 Kbps, filesize 579 MB. bitrate about 30% lower than expected

If needed I can give more stats (# of I/P/B frames,...)

though people should remember that if i cant replicate it then its awfully hard to fixThe source file is about 350MB, I can upload it somewhere, in case you need it.

[manolito]

@M4ST3R
This sounds very much like the quality has already maxxed out, and QuEnc does not do any (useless) padding just to reach the desired file size. Have a look at the quantizer values. If you really want it bigger, try to use a high bitrate matrix (I believe "Fox Home Entertainment" is one of those).

Cheers
manolito

[M4ST3R]

This sounds very much like the quality has already maxxed out, and QuEnc does not do any (useless) padding just to reach the desired file size.
It makes sense, the source file being already encoded at low bitrate (about 1000 kbps).

Have a look at the quantizer values.
I don't know very well how to get these values and how to interprete them. I opened the file with Bitrate Viewer that says Q level peak: 6.72, Q level average: 1.07
Attached: the detailed results generated by BV
So even if I'm not sure I interpret corrrectly, it seems you were right.

If you really want it bigger, try to use a high bitrate matrix (I believe "Fox Home Entertainment" is one of those).
Actually I don't really need it bigger, it's just I (apparently) wrongly thought a higher bitrate (= bigger filesize) ALWAYS gives a better quality, without thinking it also depends of the bitrate of the source.
I'll try such a matrix, for the feedback, even if I understand the quality won't increase much.

Thank you for the advices!

[dragongodz]

Q level average: 1.07
yep thats looking pretty saturated. the only thing that could be done is those few frames that are higher lowering their quant. then though you could be going over your set max rate even more etc.

The source file is about 350MB, I can upload it somewhere, in case you need it.
thanks but 350mb is WAY too big for me to be downloading.

I encoded it with the settings M4ST3R used. The only one that is different is scene detection off.
so what exact differences in settings from your previous encode ? if you could try activating any changes 1 at a time to try and pick exactly which setting was causing a problem for you then i would have something to specifically look at.

[drcl]

I asked M4ST3R for his settings.... the only difference between mine and his was that he had left scene detection OFF.

So I tried an encode with scene detection off and the artefacts i had complained about magically disappeared.

[M4ST3R]

thanks but 350mb is WAY too big for me to be downloading.

I could upload a clip but since we now know the reason of my "problem", I guess you don't need it anymore. It's up to you.

I tried some high bitrate matrixes, but it didn't work better. The default matrix is the one that gives me the highest bitrate.

I did some tests in order to find the bitrate that would generate a file at the expected bitrate.
For 2474 kbps computed by a bitrate calculator:
target bitrate in QuEnc: 3650 (with max bitrate = 2600, dunno how QuEnc handles things when target>max)
ouput bitrate (by virtualdubmod): 2488
In other words, to reach the expected bitrate of 2474, I have to set the target bitrate at 3650.

mpeg streameye:
frames count : 60 894
frame size max : 92 960
avg : 13 713
avg/max (I) : 25 978 / 92 960
avg/max (P) : 17 937 / 76 692
avg/max (B) : 10 596 / 55 776
min : 0
file size : 835 096 864
--------------------------------------------

framerate declared : 25.00
--------------------------------------------
real : 25.00

bitrate declared : 3 650 000
--------------------------------------------
real max : 4 606 000
real avg : 2 742 600
real min : 612 800
bitrate viewer:
bitrate:
peak: 3956 average: 1971
Q level:
peak: 5.12 average: 0.77
detailed result (http://www.fileden.com/files/25538/3650.xls)Even if the max bitrate is over the SVCD limit, Nero doesn't say it's not compliant.
I'll try to do metrics measurements, to see if the quality is better or not (than target bitrate=2474)

[dragongodz]

So I tried an encode with scene detection off and the artefacts i had complained about magically disappeared.
ok thanks. i am guessing here but if there is lots of scene changes being detected then that means lots of I frames and they are probably being size limited for your max rate rather hard. aswell of course B frames having to be hit hard aswell for the same reason. will look at that more later.

In other words, to reach the expected bitrate of 2474, I have to set the target bitrate at 3650.
as i already said, you are pretty much saturated. the only thing you are doing is forcing some frames that were quantised higher to try and respect max bitrate lower. you can see this with the higher max rate achieved.

dunno how QuEnc handles things when target>max
when max rate < target rate then max rate = target rate.

please remember with streameyes results its reporting not only video but audio and muxxing aswell. if you want to see just video demux it first.