VC1 - encoding dark scenes [Archive]

jubilex

9th March 2007, 06:52

zambelli

9th March 2007, 09:22

I've been messing with encoding to VC1 using wmcmd.vbs. The results I've gotten so far are excellent, encoding 1080 hdtv to 1080p wmv. Using constant quality 90, motion search at adaptive/true chroma, bframes 1, lookahead 16, buffer 60000 and bframe delta 1, most frames look practically identical to the original.
Those are good settings, though keep in mind that lookahead and buffer are irrelevant in 1p VBR mode. Lookahead only works for CBR encoding and the buffer is determined by the encoder based on the final maximum bitrate value.
I would also recommend enabling the loopfilter and using the auto modes for motion search range and match method. If the encoding slows down too much, remove the motion match option.

Dark scenes, though, are dramatically worse. Are there any settings that will force the encoder to use more bits while still using constant quality mode? It seems like it wildly underestimates the bits required for those scenes.
Not in that mode. Quality mode means fixed QP - the same quantizer is applied to every frame/field. You can only improve the motion vector prediction, but you can't really change the QP per frame.
Dark scenes are notoriously difficult to encode without artifacts. A lot of the HD-DVD/BluRay encoding challenges revolve around getting the dark scenes to look good.

Also, is the main difference in v_performance speed vs filesize? I saw no noticeable quality difference using 0 vs 60, but the speed was almost 4 times greater (average bitrate was 8.86k vs 8.47k).
Yes, in 1p VBR mode greater performance level will typically equal smaller filesize. The actual difference will vary based on content complexity.

jubilex

9th March 2007, 16:17

Thank you for the quick response, and for your work on and explanations of wmcmd.vbs. I've avoided non-gui apps forever, but this has gotten me into Avisynth, and the combination is so fast, flexible and easy I'm amazed.

Thanks for pointing out that the lookahead and buffer aren't relevant to CQ mode. I was aware that most people advise enablin the loop filter, but I was worried about the caveat that it can create a softer picture. I've been doing so much A/Bing in the past few days that my eyes are melting, but I guess now I'll try it and do some more. Time to polish my jeweler's loupe. I'll also try a CBR of around 8000, which seems to give good results for a lot of people.

As for motion match and search level, do you recommend those settings as good defaults for all encoding, or just ones where I'm having a problem with blocking in dark areas?

The performance setting was very surprising to me. A setting of 0 gave me an encode of a 2-hour 1080i movie to 720p in about 7 hours on a P4 3000, versus around 27 hours at the default setting. The filesize is bigger, but not deal-breakingly so, and even going frame by frame I see no perceptible quality difference.

The dark thing drives me nuts. I'm trying to encode my hdtv cap of Jaws. The beach scene, with crowds of people rushing around, water splashing, waves rolling, etc, looks absolutely flawless, virtually identical to the source, even in frame-by-frame comparisons. But when Hooper goes underwater to check out the wreck of the Orca, suddenly I'm looking at a bad VCD. Bitrate on the beach scene is in the 8mb range; for the underwater scene, it's 2 or less. Oh well, more tweaking awaits...

One more thing -- on one machine, I frequently get an error "error in transcoding: error 0xC00D002E". This usually occurs as soon as I run the script, after it gets the source duration. Once it occurred about 2 hours into encoding. In all cases, the .avs plays back fine in Media Player Classic, and what's even odder is that the exact same .avs and wmcmd script work at other times. Any idea what that error code indicates?

zambelli

10th March 2007, 00:03

I was aware that most people advise enablin the loop filter, but I was worried about the caveat that it can create a softer picture.
It might create a very slightly softer picture, but overall it ought to improve quality. I think you'll be happy with it.

As for motion match and search level, do you recommend those settings as good defaults for all encoding, or just ones where I'm having a problem with blocking in dark areas?
Sure - those are settings that make the encoding operation more complex (hence the longer encoding time), but ought to always yield better image quality.

One more thing -- on one machine, I frequently get an error "error in transcoding: error 0xC00D002E". This usually occurs as soon as I run the script, after it gets the source duration.
Apparently it stands for NS_E_LATE_OPERATION: "An operation requested for a particular time could not be carried out on schedule."
Is your Avisynth sourcing from a network share by any chance?

jubilex

10th March 2007, 00:23

No, not from a share. What's odd (and annoying) is that it fails at different times, sometimes 75% into a 9 hour encode. This last time, it failed several hours into an encode of a source that I had already previously encoded in its entirety using different quality settings (i.e., same avisynth script referencing the same .d2v and .ac3 files).

Anyway, thanks for all the suggestions; I'm going out of town for a few weeks and will resume trial-and-error and feverish A/Bing when I get back.

zambelli

10th March 2007, 01:42

No, not from a share. What's odd (and annoying) is that it fails at different times, sometimes 75% into a 9 hour encode. This last time, it failed several hours into an encode of a source that I had already previously encoded in its entirety using different quality settings (i.e., same avisynth script referencing the same .d2v and .ac3 files).
Oh, AC3... Are you using AC3Filter by any chance?

Either way... Try decoding the AC3 to WAV and then loading that in Avisynth. I bet you the problem will go away. It seems that many DShow AC3 decoders have trouble delivering the same exact samples in that 2nd pass.

diogen

11th March 2007, 19:51

Dark scenes, though, are dramatically worse. Are there any settings that will force the encoder to use more bits...If you mean that black in the original source is not black in the encoded version, I don't think it has anything to do with a bitrate "diet".
I think it is the dreaded [16,235] -> [0,255] conversion: the video black in the original [16] is faithfully represented as [16] when encoded that is not black [0].

You can read more about this here (http://www.avsforum.com/avs-vb/showthread.php?s=&postid=4969789) from a while back.

Stacey Spears on AVS calls it "head and toe clipping". It is also called blacker-than-black (BTB) ans whiter-than-white (WTW).
I don't think there is a simple way to make this issue go away when encoding.

Diogen.

zambelli

13th March 2007, 03:15

I don't think there is a simple way to make this issue go away when encoding.
Keep in mind luma remapping and clipping will only happen if you're converting from YUV to RGB and back. If your input is YV12 or I420, WME9 will not do any color space conversion as WMV uses 4:2:0 anyway.

jubilex

2nd April 2007, 00:45

I'm getting stumped (and somewhat frustrated).

With CQ 95-98, I find that the most complex, detailed scenes imaginable come out pretty much flawlessly, identical to the source for all intents and purposes. But the ball is dropped bigtime on the simpler scenes, which end up with macroblocking all over the place or big loss of detail. Why is the required quality so drastically underestimated here? A q96 encode of an hdtv cap of Blade Runner (2hrs) came out to 2.99gb, and while 95% of all frames look fantastic, the other 5% are pretty weak.

Here are some examples of good and bad scenes, with the source and the wmv encode for comparison:

http://thenerdparty.com/wmv

Even using mslevel 4 doesn't make much of a difference. I think I've tried the various options at this point, and while they massively increase encoding time, the improvements in visual quality are almost always insignificant. Or do none of those even apply to CQ mode? I've tried VBR without much more luck (but with much greater encoding time).

Is this something x264 just does better than vc1? I've compared my encodes from hdtv .ts with x264 ones, and the wmv ones are significantly better on the majority of frames -- more detail, better preservation of film grain, more texture-- but the x264 definitely seems to handle stuff like dithered backgrounds and smooth areas of uniform color much better, without the blocking I'm getting (while retaining texture).

I thought I might be able to force higher quality by using a CBR of 8k, but when wmcmd has finished, it says "expected bitrate 800000, average bitrate 364000". Huh? Why is it using less than I tell it to?

diogen

2nd April 2007, 15:41

I thought I might be able to force higher quality by using a CBR of 8k, but when wmcmd has finished, it says "expected bitrate 800000, average bitrate 364000".That just means that in order to achieve the requested quality result no more that 3.64Mbps are needed.

Where do your sources come from? OTA HD captures? What are the file sizes?

Diogen.

Sagittaire

2nd April 2007, 17:02

Seem to have no actived options (open wmvenc.dll with simple text editor):

F u l l F r a m e R a t e
N u m B F r a m e s
F o r c e O v e r l a p
F o r c e B F r a m e
D e l t a Q P
F o r c e B F r a m e D e a d Z o n e
F o r c e V i d e o S c a l i n g
F o r c e M e d i a n P e r c e p t u a l O p t i o n
B D q u a n t
P a n i c M o d e
D a r k I n t e r B l o c k D e a d z o n e L e v e l
D q u a n t S t r e n g t h
D q u a n t O p t i o n
D q u a n t T e x t u r e T h r e s h o l d
L o w D q u a n t T e x t u r e T h r e s h o l d
F o r c e N u m T h r e a d s
F o r c e L B X D e t e c t i o n O f f
F o r c e L o o p F i l t e r
F o r c e N o i s e E d g e R e m o v a l
P r e p r o c e s s i n g L e v e l
D a r k n o i s e F i l t e r L e v e l
I n i t i a l Q P
F a v o r I n t e r M o d e
I n t r a D C L e v e l C o m p e n s a t i o n
D C Q u a n t i z a t i o n T h r e s h o l d
I n t r a A C Q u a n t i z a t i o n D e a d z o n e
M o t i o n S e a r c h R a n g e
M o t i o n S e a r c h L e v e l
M o t i o n M a t c h M e t h o d M a c r o b l o c k
M o d e C o s t M e t h o d B F r a m e
D e l t a Q P
C o m p r e s s i o n O p t i m i z a t i o n T y p e
M o t i o n V e c t o r C o s t
M e t h o d M o t i o n V e c t o r C o d i n g
M e t h o d F o r c e R a n g e R e d u c t i o n
D e n o i s e O p t i o n
S p a t i a l D e n o i s e W i n d o w S i z e
V i d e o T y p e

set "deadzone" and "dark" option can perhaps solve that.

jubilex

3rd April 2007, 01:35

Hmm, I don't have a wmvenc.dll and wouldn't know what to set those options to in any case.

Here is a frame from the sequence I am using as my reference for the problem I'm having:

http://thenerdparty.com/blade2/source.png

Here's the wmv version, vbr 4953k, mslevel 4, mmatch 0, msrange 0, perf 20:

http://thenerdparty.com/blade2/wmv.png

And here is an x264 version of the same frame:

http://thenerdparty.com/blade2/mkv.png

Now, it's not directly comparable, because the x264 has sharpening obviously applied, but from my tests, sharpening makes my wmvs look even worse in this scene, while the x264 (same bitrate as the wmv) comes through like a champ. Are there any settings that can improve my results? Why is the dithered, mostly uniform background in this scene such a mess while what would seem to be a fairly similar thing (dithered, mostly uniform sky) in other scenes fine, retaining the grain rather than blocking up? Like this one:

Original:
http://thenerdparty.com/blade2/7.png

wmv:
http://thenerdparty.com/blade2/7-wmv.png

I would particularly like to use CQ mode for its speed, but the same quality setting that gets awesome results in most scenes drops to 2k or so for scenes like this, with results being what you'd expect for such a low bitrate. When I up the CQ from 96 to 97, this scene is a little better (still not as good as the x264), but the filesize ramps up enormously, despite the fact that 99% of the scenes look essentially identical to the q96 versions.

zambelli

3rd April 2007, 02:05

Seem to have no actived options (open wmvenc.dll with simple text editor):
wmvenc.dll? Which version of WMV encoder is that?

jubilex

3rd April 2007, 18:19

Which of the -v parameters have any effect on CQ mode? I just did a CQ test using -v_preset best, and despite it taking 18x realtime (versus 3x realtime for my usual settings) and making a bigger file, the "best" encode had at least as many frames looking worse than the default as it had frames that looked better.

I have been unable to find any settings that get the dithering of the background in this scene anywhere near as fine (non-blocky) as the 4900k bitrate x264 version; and aside from the grain, the x264 does a much better job of preserving the light/shadow of the scene -- flipping back and forth between the source and x264, at worst there is a slight loss of grain, but the overall image is identical; flipping between the source and wmv, and the shapes of light and shadow in the background changes pretty dramatically.

Even a cbr 8k at preset "best" doesn't come close; and if 8k isn't enough to render the scene well, then why does CQ mode DROP the bitrate in this scene to well below the rest of the movie (to around 2-3k while the movie on average is close to 4k)? A vbr using the same bitrate as the x264 and preset "best", while taking 22x realtime, likewise came up significantly short in quality.

zambelli

3rd April 2007, 21:31

Which of the -v parameters have any effect on CQ mode? I just did a CQ test using -v_preset best, and despite it taking 18x realtime (versus 3x realtime for my usual settings) and making a bigger file, the "best" encode had at least as many frames looking worse than the default as it had frames that looked better.
CQ mode uses a constant quantizer. Every frame is compressed at the same compression level. Better motion estimation makes the frames compress to smaller sizes, so overall you ought to be seeing an improvement. You're probably noticing the B-frames looking worse. Try running a "-v_profile best" encode but add "-v_bframedist 0" after it to disable B frames.

Even a cbr 8k at preset "best" doesn't come close; and if 8k isn't enough to render the scene well, then why does CQ mode DROP the bitrate in this scene to well below the rest of the movie (to around 2-3k while the movie on average is close to 4k)?
1-pass VBR mode doesn't really work in terms of bitrate. It uses the same QP for every frame, so the result bitrate is completely arbitrary. The only thing that improves quality in that mode is better motion search.

jubilex

5th April 2007, 01:37

Okay, I think I have to accept that there are at least some scenes that vc1 simply cannot do as well as x264. This 50 second clip looks very close to the source in x264, while every vc1 I do, no matter what settings, has many frames where the guy's face is smoothed over, and the background is usually blockier. The x264 seems to be much more consistent from frame to frame as far as keeping the fine dithering and grain; the face never seems to go "out of focus" for a frame here and there as the vc1 does. Granted, this was using a profile that took 50x realtime, but maxing out every option in vc1 (vbr of the same bitrate as the x264, preset "insane", performance 100; I've also tried with a couple other options tacked on) still doesn't give me a clip as consistently close to the source as the x264 -- and sometimes it is pretty significantly worse -- though it does take half the time.

I'd be happy to upload the 50 second clip and the x264 encode of it somewhere if anyone would like to take a whack at it.

jubilex

5th April 2007, 03:01

Just did an x264 encode using HQ-Slow profile and M4G custom matrix, total encoding time was only 7x realtime, and it still looks noticeably better than the 22x realtime maxed out vc1 at the same bitrate.

Guess I'll just have to wait until the 360 is hacked so someone can make XBMC for it :(

zambelli

5th April 2007, 03:20

I'd be happy to upload the 50 second clip and the x264 encode of it somewhere if anyone would like to take a whack at it.
Sure, let me give it a shot. I'll send you an upload FTP location by PM. Please upload the source HuffYUV compressed (YV12 ffdshow Huffyuv is fine), as well as your best x264 and WMV9 encodes, and provide the command-lines used with wmcmd.vbs and x264.exe to produce those encodes. Thanks!

zambelli

12th April 2007, 12:07

Hey Jubilex,

I took a look at your source. You're right, it's a very difficult source to encode. It's dark, grainy and has existing banding from the previous codec - which makes it difficult to compress at anything other than the lowest QP values.

I think encoding at a fixed quality (96) is not going to help bring out the best in those difficult dark scenes because fixed quality doesn't allow the encoder to use lowest QPs for those tricky scenes. What you might want to try instead is take the average bitrate of your existing Q96 encodes and then do a CBR or 2-pass VBR encode at that bitrate. That might seem like it wouldn't change much in terms of bitrate, but it actually will allow the encoder to use as low as QP=1 on the difficult scenes.

Another options you should try are:
-v_dquantoption 2 -v_dquantstrength 3
-v_dquantoption 2 -v_dquantstrength 4

First see which one of those produces better results for you, and then also try adding this on top of that:
-v_percopt 2

I think you will see a noticeable improvement in dark areas and film grain preservation if you try those options.

jubilex

12th April 2007, 17:48

Another options you should try are:
-v_dquantoption 2 -v_dquantstrength 3
-v_dquantoption 2 -v_dquantstrength 4

First see which one of those produces better results for you, and then also try adding this on top of that:
-v_percopt 2

dquantstrength 4 gave me "invalid parameter" error. Percopt 2 looked worse, if anything.

Dquantoption 2 with strength 3 looked better. With preset "better", it was more comparable to the x264 encode, albeit at just under 3fps versus the x264's just over 4fps. In a good chunk of the sequence, the tricky background looked as good or better than the x264, and there were some parts that looked essentially identical to the x264, BUT the x264 still kept the faces consistently sharper throughout the clip.

But, unlike the x264, the vc1s had a dramatic dip in quality around the 21 second mark that continued for the duration of that camera shot, around 10 seconds, and remained excessively soft well into the next shot. Here are some shots of what I'm talking about:

vc1: http://thenerdparty.com/bl/b1-vc1.png

x264: http://thenerdparty.com/bl/b1-x264.png

vc1: http://thenerdparty.com/bl/b2-vc1.png

x264: http://thenerdparty.com/bl/b2-x264.png

zambelli

12th April 2007, 18:39

dquantstrength 4 gave me "invalid parameter" error. Percopt 2 looked worse, if anything.
Sorry, I had forgotten to mention: try the new updated wmcmd.vbs that I just released last night. It has extended dquantstrength support.

But, unlike the x264, the vc1s had a dramatic dip in quality around the 21 second mark that continued for the duration of that camera shot, around 10 seconds, and remained excessively soft well into the next shot. Here are some shots of what I'm talking about:
Are you still encoding in mode 2 at quality 96? Or are these CBR/2pVBR encodes?

jubilex

12th April 2007, 20:26

Those are from 2p vbrs, as were the vc1 and x264 clips I sent you -- I wanted to make sure I was comparing the two codecs at the same bitrate.

I did another comparison of another encoded hdtv cap, comparing my regular default vbr settings with the dquantop 2 dquantstrength 3 settings used above, and while much of the clip preserved more detail (and grain) using the new settings, sometimes the default dquantoption and dquantstrength did a much better job, as in these shots:

default: http://thenerdparty.com/noquant.png
with dqo2 and dqs 3 : http://thenerdparty.com/quant.png

default: http://thenerdparty.com/noquant2.png
with dqo2 and dqs 3 : http://thenerdparty.com/quant2.png

default: http://thenerdparty.com/noquant3.png
with dqo2 and dqs 3 : http://thenerdparty.com/quant3.png

zambelli

12th April 2007, 20:53

I did another comparison of another encoded hdtv cap, comparing my regular default vbr settings with the dquantop 2 dquantstrength 3 settings used above, and while much of the clip preserved more detail (and grain) using the new settings, sometimes the default dquantoption and dquantstrength did a much better job, as in these shots:
Thanks for the screen shots.

The new wmcmd.vbs documentation is a little bit clearer about the applications of dquant. Basically, v_dquantstrength 0 is a good generic dquant which looks for smooth areas in general and tries to encode them better. Values 3 and 4 are more targeted at capturing dark regions - so you might seem them work better there but not as good in general.

Adaptive deadzone (v_percopt 2 and up) ought to help capture some of the film grain and finer details too.

Both dquant and adaptive deadzone are good techniques for capturing the finer detail in video, but they're still not "magic bullets" - you kinda have to play with them and find that one that works the best or sacrifices the least.

Also try just the regular dquant:
-v_dquantoption 2 (implies -v_dquantstrength 0)

And try adding -v_percopt 2 on top of that - and see if it helps with the film grain.

jubilex

12th April 2007, 21:57

I'll try these as well; thanks.

I was wondering if it would be possible to have wmcmd tack on the encoding time (better yet, fps) and avg bitrate to the filename on completion? That'd be really convenient when setting up a batch cranking out a half dozen encodes with various options, making it so we wouldn't have to scroll back through the cmd window to track all that info down and match it up to the right file.

zambelli

13th April 2007, 00:12

jubilex

13th April 2007, 00:21

No consistent improvements with any of these options -- i.e., one may do a certain segment of the clip noticeably better than another, but then the latter will do a different segment noticeably better than the former. No magic bullet is right, unfortunately (for vc1; x264 with the same settings I've been trying once again did a great job for the entire clip).

Also, the new wmcmd seems to hang upon finishing an encode, just after giving the encoding stats. Have to use Task Manager to close the cmd window, and of course it only does the first job out of however many are in the .bat.

zambelli

13th April 2007, 03:35

Also, the new wmcmd seems to hang upon finishing an encode, just after giving the encoding stats. Have to use Task Manager to close the cmd window, and of course it only does the first job out of however many are in the .bat.
Looks like a bugfix I implemented isn't working as well as I thought. Can you tell me exactly what your command-line is, exactly as you're calling it (including any paths and such)?