Advices on "hard scenes" encoding for DVD

[Alex-Kid]

I have a video stream (filmed with my miniDV camcorder :)) that I want to encode for DVD-video. But when I try to encode it (I use HCEnc 0.26), some scenes look very ugly, especially the forest ones, which have many "textures" (Here's a sample pic (http://www.mediafire.com/view/2qy1ufjbt5u88tu/test.png)). This happens even at 9200 kbps, so I made a quick encoding with these scenes to see if I was doing something wrong.

So I did a 1.00 CQ encoding with a max bitrate of 40 mbps, and realized that the "perfect" bitrate for the scenes were about 17-18 mbps. So I concluded that an acceptable quality would be around 10-12 mbps anyway, which surpasses the upper limit of DVD compliancy.

So I'm asking for tricks and hints (on filters for AVISynth, VirtualDub, etc.) to adequate the picture for a better compression and encoding, while keeping a "good watching" to the eyes.

Here's a sample vid for testing (http://www.mediafire.com/download/xp0p46yvr86nwlx/test.avi) (Lagarith, 157 MB)

Thanks!

[Warperus]

Just few random thoughts: deshaker, smoothing(blurring)

[fvisagie]

I'm not sure what you mean by textures, but if you mean the fine leaf detail that appears messed up by interlacing, that's one thing you don't need to worry about, assuming you're encoding interlaced. The hardware DVD player will take care of deinterlacing, and just make sure your software player does also.

Chances are that your camera records luma to the [16,255] range, which seems confirmed looking at the bright areas in the screen grab. Measure this with ColorYUV(analyze=true).Blur(1.0). If you see occasional values below 16, those will be legal undershoots you need not concern yourself with. RGB conversion in your workflow will clip luma to [16,235] causing clipped whites (if you want to do any processing in VirtualDub, you'd have to convert to RGB32, since VirtualDub messes up the chroma of interlaced YV12). If you take care to use the Avisynth PC conversion matrices you may avoid some of the clipping, but best is to normalise luma with something like Levels(0, 1.0, 255, 2, 235, coring=false) or ColorYUV(gain_y=-18, off_y=-2, gamma_y=0, cont_y=0). YLevels() also works. Beware that Levels() also modifies the chroma range (see the documentation). For my DV camera I use ColorYUV() - it just looks better to me.

For scenes with large difference between background and foreground exposure as in your sample screen grab, you could also try HDRAGC() and/or Local Contrast Enhancement (https://forum.doom9.org/showthread.php?t=161986). Do any luma and/or chroma manipulation like these after normalising levels.

PS. If you are prepared to put some effort into ensuring best encoding quality, read up and play with all HCEnc settings, one at a time*. Note which ones improve quality at the same bitrate, and intelligently combine those into your final encoding configuration.

*Note that some settings may need to be enabled by another before taking effect

Cheers,
Francois

[Ghitulescu]

That's the interlace "artifact".
If you don't see it on TV if you connect your camcorder to it, chances are that you won't see it from the DVD player either.

[Alex-Kid]

No, I'm not talking about interlace artifact.

I leave you the resulting MPEG-2 file (http://www.mediafire.com/download/cl7nmebsvsun18e/test.m2v) for better understanding. It's about MPEG-2 artifacts.

And here's the HCEnc INI file:
*INFILE f:\iframe\marquez caceres\postprocessed\encode.avs
*OUTFILE F:\Iframe\MARQUEZ CACERES\DVD1\test.m2v
*BITRATE 9200
*MAXBITRATE 9600
*FRAMES 164400 164900
*PROFILE best
*AUTOGOP 18
*AQ 3
*DC_PREC 10
*INTERLACED
*BFF
*CLOSEDGOPS
*INTRAVLC 2
*MATRIX mpeg
*LUMGAIN 3

[fvisagie]

You mean the blocking, e.g. on the backpack? That's usually from bitrate starvation. Two-pass encoding should help a lot. Beyond that, I'm not familiar with configuring HCEnc so can't help with that.

You may also find that blocking behaviour changes once you've done dynamic range corrections for luma and/or colour.

[TheSkiller]

Try a different quant matrix in HCenc. The "MPEG" matrix you are using is rather bitrate-hungry. I recommend "KVCD notch" matrix for your purpose (in HCenc this matrix is simply called "NOTCH").
You can also try other matrices, generally the higher the numbers the higher the compression but that doesn't mean a matrix with very high numbers will look good for your video...

A had a look at your video and yeah, it's definitely a very bitrate demanding one. Also your highlights are most likely clipped like fvisagie mentioned.

Edit: If you can't get rid of the blocking on the dark part of the backpack, I suggest you set "dead zone settings" to a fixed value of 1 or 2 for both.

Also I noticed the GOP structure of the video is constantly like IBPBPBP... You may be able to reduce the overall blocking by unticking "Autogop" which will result in a GOP like IBBPBBPBB... Make sure the GOP length is set to 18 frames in Settings 1.
Why are you using closed GOPs? That will eat valuable bitrate for no reason!

[Alex-Kid]

First, I want to thank all your replies :thanks:.

I tested several settings, following your suggestions: NOTCH matrix, dead zone settings and Autogop off with 18 frames of GOP length. Though the video is improved, it still shows many artifacts. I think I'll definitely have to smooth it a little.

About closed GOPs:
I understand this is mandatory for DVD compliancy.

About clipped whites:
I do all editing and filtering through AVISynth (with Cedocida DV codec). I only use VirtualDub for intermediate encoding (with YV12 Lagarith) and configured output and input as YV12. I never convert to RGB in my workflow. On the other hand, I already set the values for HDRAGC and NonLinUSM (http://forum.doom9.org/showthread.php?p=1555234#post1555234), but by your words, I guess it's not enough. But after all, do I check any other related settings?

In the last tests, I realized this scene doesn't demand the bitrate it needs, that means if I set a max bitrate of 9500 kbps, it's never reached in any part of the stream. Am I doing something wrong or is there something I'm not doing?

[TheSkiller]

About closed GOPs:
I understand this is mandatory for DVD compliancy.
Unless you are creating a multi angle video, open GOPs are DVD compliant and there's no point in using closed GOPs. ;)
For chapter points you need one closed GOP before the I frame of a chapter point – in other words if you want to use chapters you need to define them in HCenc.


In the last tests, I realized this scene doesn't demand the bitrate it needs, that means if I set a max bitrate of 9500 kbps, it's never reached in any part of the stream. Am I doing something wrong or is there something I'm not doing?
No, you're not doing it wrong. This is expected behaviour by HCenc, it never really hits the max bitrate.
I haven't ever tried this but maybe you can hit the desired bitrate better when using the new "1 pass VBR" mode which can be ticked in the main settings.

Edit: Tomorrow I will try and encode your sample myself. :)

[fvisagie]

About clipped whites:
I do all editing and filtering through AVISynth (with Cedocida DV codec). I only use VirtualDub for intermediate encoding (with YV12 Lagarith) and configured output and input as YV12. I never convert to RGB in my workflow. On the other hand, I already set the values for HDRAGC and NonLinUSM (http://forum.doom9.org/showthread.php?p=1555234#post1555234), but by your words, I guess it's not enough. But after all, do I check any other related settings?


According to the guys at forum.videohelp.com (see this post (http://forum.videohelp.com/threads/354292-Preparing-this-Rec-601-YV12-clip-for-Rec-709-MPEG-2-encoding?p=2226253&viewfull=1#post2226253) and the rest of pages 1 and 2 if you're interested), all displays operate in RGB, therefore at least that final conversion from YV12 to RGB will take place in your workflow. YUV values falling outside of the gamut of legal RGB values at that point will lead to clipped whites and/or crushed blacks (mostly illegal luma issues) and/or colour shifts (clipping of individual RGB channels) even for otherwise legal luma values. Although they weren't designed to, tools like HDRAGC help "a little" with reducing out-of-gamut values. However, to do a good job of preventing problems with YV12->RGB32 conversion you need to use one of the dedicated Levels(), ColorYUV() or YLevels() tools.

As mentioned, you can use ColorYUV(analyze=true).Blur(1.0) to measure the need for luma and/or chroma normalisation. Histogram("classic") and Histogram("levels") display luma and chroma values graphically.

The link above also mentions tools for checking legal YUV combinations (while Y, U and V values may be within range individually, their combination may still result in out-of-gamut RGB values). However, this is getting a little esoteric, in the sense that once normalisation e.g. Levels() and dynamic range adjustment e.g. HDRAGC() have been done, output is either within range or close to it and any remaining clipping will be hard to spot, except for the fanatic. ;)

[Alex-Kid]

all displays operate in RGB, therefore at least that final conversion from YV12 to RGB will take place in your workflow.
That is something I never took into account. So I must thank you for that.

By chance, last night I did some tests regarding the luma range issue and must say that it looks better. I added Ylevels(0, 1.0, 255, 2, 235) and then applied HDRAGC settings which resulted in better overall luminosity. I hope to test compression with these new values today.

However, I would like to return on the first line: Does this mean that luma range must always be stretched to the [16-235] interval? And if it does, why is low output set to 2 on the Levels settings you suggested? Shouldn't it be set to 16?

[fvisagie]

Does this mean that luma range must always be stretched to the [16-235] interval?

BT.709 (http://en.wikipedia.org/wiki/Rec._709) and BT.601 require Y to be [16,235] and U & V to be [16,240]. Therefore many conversion mechanisms perform clipping to those values when converting between colourspaces. Clipping leads to loss of information which becomes visible as washed out whites (no gradation left) and crushed blacks (also no gradation). One way to avoid that is normalising to valid ranges as we're discussing here. Another way is to 1) use the Avisynth PC matrices when converting, 2) set the coring option to false wherever available and 3) hope that your display can handle the out-of-range values. See the Avisynth 2.6.0 ConvertTo...() and Levels() documentation for information on TV/PC matrices and coring.

And if it does, why is low output set to 2 on the Levels settings you suggested? Shouldn't it be set to 16?

This post (http://forum.videohelp.com/threads/354292-Preparing-this-Rec-601-YV12-clip-for-Rec-709-MPEG-2-encoding?p=2226658&viewfull=1#post2226658) explains why.

By the way, do yourself the favour of comparing Levels() and ColorYUV() before deciding which to use. Levels() also shrinks the chroma range, unnecessarily in the case of my DV footage that has valid chroma. With ColorYUV() and YLevels() you can operate on luma alone. The effect on my footage of ColorYUV() + HDRAGC() is far more attractive than Levels() + HDRAGC(), no matter how I tweak Levels or HDRAGC to try compensate.

Cheers,
Francois

[TheSkiller]

Simply using the PC matrix will result in washed out blacks (unless the camcorder stores black at 0 which I highly doubt any camcorder does).
The issue is simply the camcorder stores it's whitest white at 255 because it doesn't need to bother compressing it to 235 or clipping it.
You don't have to fix that, but it's advisable to do to make sure the video looks the same regarless of where it is played back.

Whether you use
Levels(0, 1.0, 255, 2, 235, false)
or
Levels(16, 1.0, 255, 16, 235, false)
to do that doesn't matter. The latter will clip anything below 16 but since there shouldn't be anything to begin with it doesn't matter. I always use the latter for my own camcorder videos but I recommend to use SmoothLevels() (http://forum.doom9.org/showthread.php?t=154971) instead of Levels() or YLevels() because they create rounding errors which result in posterization.


With ColorYUV() and YLevels() you can operate on luma alone.
Alternatively you can wrap Levels() inside MergeLuma() or MergeChroma() to affect only the luma.
Example
MergeChroma(Levels(16, 1.0, 255, 16, 235, false), last)

Edit: SmoothLevels() even has an option, chroma=0, to process only the luma.

[fvisagie]

Simply using the PC matrix will result in washed out blacks

Are you sure that's what you meant? Can you show an example? As I understand it, the whole point behind behind using PC matrices (and coring=false) is to directly map [0,255] to [0,255], therefore none of the grey balance shifting that you mention.

PS. Levels(16, 1.0, 255, 16, 235, false)

... The latter will clip anything below 16 but since there shouldn't be anything to begin with it doesn't matter.

While generally true preferences may vary. My camera, for instance, produces perfectly legal undershoot in dark scenes and clipping that visibly reduces accentuation. Alex-Kid may want to play around with some dark clips before deciding which handling is preferred for his camera.

[Alex-Kid]

Thanks fvisagie & TheSkiller, your posts helped me a lot.

But I have to say I give up with MPEG-2 DVD encoding. Though the video has been enhanced, the encoding artifacts are still there, even with "close-to-top" bitrate, and I ran out of ideas to increase compressibility without hurting quality too much. The best results I got was using x264 encoder.

Maybe the video still have some issues that can affect MPEG-2 encoding. Maybe noise, contrast, sharpness, etc., but at this point I feel I got stuck. I think this has became very challenging and it would be good that someone can use the sample posted some time ago and do the appropriate video processing to get a good MPEG-2 DVD encoding.

[fvisagie]

Unfortunately I have very little experience with HCenc. In case you would like to compare results, these are the two-pass ffmpeg command lines I use for DVD MPEG-2 encoding ('-top 0' = BFF):

ffmpeg -y -top 0 -async 0 -i input.avi -muxrate 10080000 -acodec ac3 -ab 128000 -flags ildct+ilme+mv0 -g 15 -mbd 2 -trellis 1 -bufsize 1835000 -strict 0 -bf 2 -b_strategy 1 -mbcmp 6 -preme 2 -precmp 2 -pre_dia_size 2 -cmp 2 -dia_size 2 -subcmp 6 -last_pred 2 -mv0_threshold 0 -maxrate 9000000 -vcodec mpeg2video -b:v 6090000 -pass 1 output.mpg
ffmpeg -y -top 0 -async 0 -i input.avi -muxrate 10080000 -acodec ac3 -ab 128000 -flags ildct+ilme+mv0 -g 15 -mbd 2 -trellis 1 -bufsize 1835000 -strict 0 -bf 2 -mbcmp 6 -preme 2 -precmp 2 -pre_dia_size 2 -cmp 2 -dia_size 2 -subcmp 6 -last_pred 2 -mv0_threshold 0 -maxrate 9000000 -vcodec mpeg2video -b:v 6090000 -pass 2 output.mpg


(In case anyone wonders, I'm not using '-target pal-dvd' because it has a shortcoming - it forcibly scales 704x480/576 to 720x480/576.)

[TheSkiller]

I think this has became very challenging and it would be good that someone can use the sample posted some time ago and do the appropriate video processing to get a good MPEG-2 DVD encoding.
Sorry for the long delay. :o
I did some encoding tests with your sample. In addition to HCenc I also used ProCoder – as to be expected with this particular kind of video ProCoder delivers the least artefacts and most pleasant to look at video (of course fully DVD compliant).


The script I used was very basic, I just corrected the levels so that the luma is in the 16-235 range. Interestingly the brightest white in your video is not at 255 but rather around 243. There also is stuff below 16 which seems to come from the camcorder's contouring (edge enhancement) overshooting, i.e. it's not important and can/should be clipped.
So I used this
SmoothLevels(16,1,243, 16,235, chroma=0, limiter=2)

I would love to upload to MediaFire but they require an account these days which bugs me so I used another hoster... :rolleyes:

ProCoder 1.5 result (mastering quality setting):
http://www28.zippyshare.com/v/84023600/file.html

HCenc_026_beta result:
http://www67.zippyshare.com/v/24331625/file.html

As you can see there is a small improvement over your own HCenc encoded version due to better settings but compared to ProCoder there's a huge difference. Every encoder has it's strong and weak sides. I'm afraid to say it but with this kind of video HCenc is being pushed over the edge. If it's of any consolation to you I can assure you that ProCoder looks rather bad with other kinds of video material.

HCenc settings
*INFILE c:\neu avisynth script (6).avs
*OUTFILE C:\test_hcenc.m2v
*BITRATE 9600
*MAXBITRATE 9600
*FRAMES 0 499
*PROFILE best
*GOP 18 2
*AQ 4
*DEADZONE 0 1
*INTERLACED
*BFF
*INTRAVLC 2
*MATRIX hclow
*LUMGAIN 1
*WAIT 0

[Alex-Kid]

@TheSkiller: thank you!

I watched the two samples and I must say that both look great, but Procoder's looks better.

I used Procoder some time ago, but I found that the output was too soft. And now that is an advantage!

On these days I tested HCEnc with a little softer input (used Unfilter, didn't test any others) but didn't like the output.

I think I'll follow your advice :)