HCenc 023 released

[Emulgator]

Hank, Happy New Year and my highest appreciations for HC 0.24 beta 27-11-2009!
This version finally kicks almost everything from the shelf.
Using no deadzones I can finally have dark and grainy parts encoded without the well-known block artifacts.
Grain stays grain as in CCE, no moving dark blocks as in Procoder (Express 2 I checked on)
or chequered-structure-changing-blocks as in Mainconcept.
I can force bitrate usefully into these dark parts as I like. No bitrate flipping as in CCE.
Picture is still sharp, even using MPEG Matrix.
Almost no visible artifacts encoding text onto gradients.
High motion is usefully encoded, no heavy block rows for 1 frame as sometimes briefly appearing in TE4XP.
I would say, I see HC on par now with CCE, better than Procoder, better than TE4XP, better than Mainconcept.
Very good work and the only MPEG-Encoder I have seen improving in terms of picture quality for the last 2 years.
And for now I have only tested SD resolutions...

[Gromozeka]

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...

[tin3tin]

The HCenc024.exe Beta 27 - 11 crashes for me in the end of first pass(says 100%) with a 16 sec long avs script using the Soundout plugin too. It renders without errors in Hcenc 0.23.

This is the ini:

*dc_prec 10
*WAIT 0
*BITRATE 7000
*MAXBITRATE 7500
*AUTOGOP 15
*AVSMEMORY 1024
*PROFILE BEST
*PROGRESSIVE
*PRIORITY HIGH

[manolito]

First time that 0.24 beta got the sizing wrong in 1-pass VBR mode.

This is the deal:
Source: 90 min MPEG2 from a DVB-T capture.
Reason for reencoding: Logo removal
Encoder: Latest HC 0.24 beta Novemver 2009
Matrix used: Manono1
No filters except LogoAway, no resizing
Source characteristics: Last 4 min consists of vertically scrolling end credits, requires high bitrate
Average Bitrate: 6400
Max Bitrate: 8500
Base Q determined by HC: 2.20

The encode came out almost 4% oversized. Next thing I tried was disabling Deadzone Quantization, but the oversize was the same (speed was significantly faster, though).

Out of curiosity I did the same encode with an older 0.24 beta from May 2009, but no change, still almost 4% oversized.

And the last thing I tried was throwing this source to my old and obsolete 1-pass VBR plugin for DVD2SVCD (Hybrid Mode), and this encode came out perfectly (1% undersized).


Ok, this is purely academic. I used DVDShrink to shrink the size of the oversized encode and compared the result to the encode using my old plugin, and I was completely unable to notice any visual difference.


Cheers
manolito

[manolito]

I just ran into some nasty problems using the latest HC 0.24 beta in 1-pass VBR mode.

Source:
DVB-T capture, interlaced, average bitrate ~3500 kbps

Encoding settings:
HC 0.24 beta 1-pass VBR mode
Average bitrate 4950 kbps
Max bitrate 8500 kbps
GOP: Auto
Interlaced: Auto
IntraVLC: Auto
DeadZone: Auto
DC Precision: 10
Scene Change: On
Matrix: Manono2 (aka Sony medium)


Using my old and obsolete 1-pass VBR plugin (which is basically a CQ_MAXBITRATE mode with a couple of tweaks to ensure correct sizing) with the same HC settings these artifacts were all gone. This was also true when using HC 0.23.

Download the test clips here: http://www.mediafire.com/?jztmtj1kqum


Cheers
manolito

[Emulgator]

Oops. opv.mpg had a bitrate valley allocated at the scenecut, where a bitrate peak should have been,
especially with scene change on.

Hm. looks like poor bitrate allocation in 1-pass VBR?
In my tests I was only checking 2-pass VBR.
and this is what I use regularily together with Matrix MPEG.
(Other matrices did not help too much at my side.)

[Emulgator]

tin3tin

*PRIORITY HIGH

what if you leave priority at "idle"?
"High" may force its way through too much, while not offering any gain in speed,
not letting other processes in, which you may need for task to complete.

[txporter]

Can someone step me through the correct way to encode using 1-pass Quality VBR with HCenc? Is this similar to doing CRF encoding with x264?

I can get HCenc to encode using 1-pass, but it requires me to given it an average bitrate target. I guess I am expecting something more like x264 where I tell it I want some level of quality and it uses whatever bitrate is necessary to do this. Is that constant quant in HCenc?

Sorry for the beginner level questions. I have used HCenc for a while now with 2-pass target bitrate, but I wanted to test various avisynth functions out for compressibility and I can't see how to do that when I need to give a target bitrate.

edit: I am using HCenc 0.24 11-27-09 release

[hank315]

@Emulgator & manolito
Seems 1 pass VBR is seriously broken in the latest 024 beta.
In my development version I even get random crashes if all compiler options are on, running in debug mode is OK making it hard to track down the error

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Originally Posted by Gromozeka

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

[txporter]

Originally Posted by hank315

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.


Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

Thanks, Hank! It does work in HCenc_023. I can use that to test different filters and then just apply that to 2-pass HCenc_024. I like targeted bitrate just fine, I just didn't know how to tell what effect my filters were having. Thanks a lot for your support and your program!

[manolito]

But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Huh? HC 023 had a 1 pass VBR mode already? How stupid of me, I thought this feature was introduced in the first 024 beta

Just FYI I reencoded the clip which gave me those problems using the very first 024 beta (Nov. 2008), and the artifacts are identical. So this bitrate distribution problem in 1 pass VBR mode must have been there from the beginning.

The clip is a TV broadcast from last New Year's Eve, it is "Fleetwood Mac live in Boston 2004". The artifact comes up at the end of the last song, and this last song is a lot less compressible (requires higher bitrate) than most other parts of the clip. It looks like HC is desperately trying to reach the desired target size, so it does not even allow bitrate peaks at scene changes.

Anyway, I will give it another shot with yet another 024 beta version and report back...


Cheers
manolito

[manolito]

In the meantime I ran the same encode using 3 different 024 beta versions, but the results were identical.

Conclusion: The 1-pass VBR mode strategy in HC 024 can fail under certain unfavorable conditions.

The source is really very demanding. But OTOH HC CQ_MAXBITRATE and HC 2-pass VBR do handle this source without problems, and even QuEnc 1-pass VBR can do it. I tried to cut out a 5min segment of the source with the problematic spot right in the middle, but this was no problem for HC 024 1-pass VBR mode.

In order to test different bitrate distribution strategies it is probably necessary to process the whole clip. The size is 1.2 GB, let me know if I should upload it....


Cheers
manolito


BTW I also tried to use *PROFILE BEST instead of FAST, but same result...

[Gromozeka]

Originally Posted by hank315

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

You will add her(its) in the future time?
Thank you, This very good function.

[Emulgator]

An interesting finding while further comparing HC vs CCE.
Source was 16mm Film (1.38:1), Duration 1:52:20, Transfer, PAL speedup to interlaced PAL-VHS, ugly and grainy.
I restored this PAL-VHS 25i to 25p using TGMC, selecteven, MC_spuds. No residual combing visible.

Now encoding using CCE, 2-Pass VBR @5,1Mbps, "Progressive frame"
and HC0.24 beta 27-11-2009, 2-PassVBR @5,1 MBps, "Progressive".

A: Comparing results on Notebook-TFT 1920x1200 using MPC+motion vector overlay, resized 200%, VitualDub resized200%, Konran's Bitrate viewer.
HC (tweaked: VBR bias 10, minBRF 1.20, no deadzones, matriy MPEG) shows more useful bitrate distribution than CCE,
Especially in the beginning, where CCE comes slowly up from poor bitrates, leaving some blocks in dark parts.
HC swings faster in to average bitrate with a light overshoot and starts from the very first frame with a smooth picture.
HC finds very useful motion vectors as well.

Later after some minutes both encodes look similar. No faults that would be visible without hard scrutiny.

B: Comparing results from DVD after authoring (DVDLabPro2.51) and burning to disc.
Panasonic BD-50 -> BD50-upscaler -> HDMI -> Loewe Connect 37 (TFT-TVFullHD37")
Here the TFT showing the DVD coming from HC sometimes shows some thin glitches,
height 1 single 1080HD line and width maybe 16 HD pixels
that were not part of the MPEG-2 stream before authoring.
These line glitches were only visible during 1xplay
and could not be seen when pausing and stepping frame by frame using RC.

(Later I saw another 23.976p pulldowned DVD exhibiting the same behaviour on this particular player/TFT combination.
The reason could be tracked down to the Panasonic BD-50's upscaler, when progressive frames should be displayed.
No problems on interlaced frames.
Panasonic DMR-EX95, while having an uglier upsizer with bad ringing, did not exhibit these glitches from the same DVD.
Phew, so not my restoration/encoding fault...)

Now comparing streams using DGIndex and ReStream.

HC: after specifying "Progressive", in GUI TFF/BFF Options are greyed out.
HC then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=no
Sequence Extension
"Progressive Sequence"=yes,
In GOPs: no tff flags.
Applies Scanning mode zig-zag.

Now CCE:
after specifying "Progressive Frame", in GUI TFF/BFF Options are not greyed out.
CCE then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=yes
Sequence Extension
"Progressive Sequence"=no,
In GOPs: tff flags=yes.
Applies Scanning mode zig-zag.

Now I reflagged the HC stream to the same settings as CCE would output
using ReStream.
Authoring, burning, disc testing as before:

Now HC output looks smooth and ok on this particular BD-player/TFT-TV combination!

Then I forced interlaced encoding on progressive footage in HC
but then alternate scan order was forced by HC as well
and so I lost quality, picture became a bit blocky then.

It seems that these flags exhibit a bug in player/resizer/TFT-TV combination
that can be avoided if one reflags HC output as interlaced, even if 25p has been encoded.

To overcome this possible trap with 25p footage,
and if Hank approves this approach
I would suggest to extend GUI behaviour.

After "Assume input frames as progressive"
(info: This does not mean that HC would deinterlace !)
I'd still like to have PictureCodingExtension TFF,BFF tickable,
because no tff=1 == tff=0 == BFF;
I'd suggest to have SequenceExtension "Progressive Sequence" tickable/untickable;
(info: Unticked helps for DVD-compatibility)
and I'd suggest to have Block scan order manually settable using 3 radio buttons
"Auto" (info: Applies "zig-zag" on assumed progressive frames, "alternate" on assumed interlaced frames) or
"Override to zig-zag" (info: Useful on progressive frames, interlaced may suffer) or
"Override to alternate" (info: Useful on interlaced frames, progressive may suffer)
Fenced meaning info line in GUI that declares the use of these flags.

[manolito]

@Emulgator,

interesting findings, and for the most part they are consistent with my experiences...

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC. Any specific reasons? Does it generally kill grain in dark areas?

For the different TFF flagging of progressive streams in CCE and all the rest of MPEG2 encoders I had my problems a while ago (and there was a thread here which I cannot find right now), but here is the summary:

According to the MPEG2 standard the TFF flag has no meaning whatsoever for progressive content. A software- or hardware player should simply ignore it. But in reality at least some (maybe most) standalone DVD players do not ignore the TFF flag for progressive content.

I have an old and cheap Cyberhome DVD player (Mediatek chipset) connected to a CRT TV via SCART (using SVHS). The player always outputs fields (interlaced), and it does so observing the TFF flag, no matter if the stream is progressive or interlaced. For progressive content this does not make a difference, the TV picture will look the same.

The problem I had a while ago was that my encodes from TV captures had jumpy end credits. The film itself was progressive, but the credits were interlaced TFF. When I started with DVD conversions I always used CCE SP 2.50, and these encodes looked good because CCE always flags a progressive encode as TFF. But when I started using QuEnc and HC I got these jumpy end credits. Simple reason: QuEnc as well as HC do not allow to set the TFF flag for progressive encodes, so my end credits simply had the wrong field order.

The reason why CCE flags progressive encodes as TFF is probably historical. Older CCE versions would always produce TFF output. If the source was BFF, they would apply an offset of 1 scan line, so the output would be TFF. Newer versions can do it both ways, but the flagging has not changed.

Ok, the "right" solution for this problem would be to encode the film and the credits separately, but that was too much hassle for me. I asked DGZ (co-author of QuEnc) if he could modify QuEnc to allow TFF flagging of progressive encodes. He checked it, but libavcodec would not allow it. Same is true for HC: if you specify progressive, then the TFF flag cannot be set, and scan order will always be zigzag.

Well, reading your post it seems that newer upscaling standalone players also do evaluate the TFF flag even if the content is progressive. As a solution to this problem (except using CCE) I normally use HC's "Automatic" interlaced mode. If you specify neither *PROGRESSIVE nor *INTERLACED nor *DVSOURCE then HC will try to detect interlaced or progressive per frame automatically. In this "auto interlaced" mode TFF is the default, but you can change this. Scan order is also determined automatically in this mode. Works very well for me, give it a try...


Cheers
manolito

[txporter]

This is an interesting discussion. I started using HCenc a few months ago to re-encode DVD rips (mainly NTSC material) into single file MPEGs with hardcoded subs. I do this to serve videos up to my Tivos. At any rate, I started by encoding using the automatic detection method you described above. The videos produced from this played fine in MPC-HC. When I watched them in WMP10 though, they exhibited what I was calling strobing. For a short period of time the video was fine, and then it seemed to jump up and down a few scan lines at a time, and then would return to normal and repeat. At any rate, my tivo playback looked like the WMP playback with strobing video/smooth video/strobing video..etc. It was extremely annoying.

Well, I was able to avoid the strobing video by specifically encoding as either progressive or interlaced. Frankly, I was really only encoding NTSC material at that point, so I don't know if interlaced material showed the problem...but progressive did. I wonder if this was somehow also related to the field flagging?

[Emulgator]

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC.
Any specific reasons? Does it generally kill grain in dark areas?

Hm, I simply fear to sacrifice something important where I could easily have given more bitrate.
For instance Procoder (2 Express) could be good, finds useful motion vectors even in the dark,
but blows it when giving bitrate there. A miser and therefore blocky.
Because only a few Y steps (16~32) are left from 16-235 to encode dark tones close to black,
I tend to value reproduction quality of dark areas higher than average.
Lots of Film lives from smooth darkness.
It may well happen that encoded output is watched on TFT with pushed levels, wrong settings etc.
Then anything blocky in dark parts becomes painfully visible and I hope to finetune my encodes to look better than average.
Even when pushed it should look smooth.
For testing I even go into a windowless and completely dark room,
doors closed, watching footage on a Laptop or portable TFT,
eyes offset from center, viewing from a top angle, just to find blocks in the dark.
If my encodes pass this, then I'm feeling safe.
As a sidenote: Sonic reps even suggested that compressionists using Cinevision
should tell editors to clip their shades below a value of 3 or 4 to pure black.
So a step from pure black to the next value of 3 or 4. ??
Introducing deadzones, therefore destroying shades. What a deal...
Do they fear to exhibit encoder flaws? I guess so.
I can be done better...

..but progressive did. I wonder if this was somehow also related to the field flagging?

I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

[txporter]

Originally Posted by Emulgator

I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

This is the first time that I have ever tried ReStream. For both cases, I am taking a 2min video clip that is 97.85% FILM according to DGIndex. I built a d2v file using Honor Pulldown flags and am running the video through TFM().TDecimate(). The input video to HCenc should be 23.976 fps progressive.

If I force HCenc to do progressive encoding, the video plays fine when I upload to my tivo. ReStream has tick marks on Frametype progressive and Progressive sequence. The field coding shows it as BFF with ----- as the tff-flag.

If I let HCenc use automatic interlace detection, the video experiences the strobing (not verified, but this is how I did it a few months ago when it did). ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.

In both cases, the scanning-mode is listed as zig-zig. Although I am not sure if that is just the type most often used or what.

I guess that I could play around with ReStream to see if I can get a strobing video to play back ok. Are the ReStream parameters what I should expect or should I be seeing something different?

[um3k]

Hank, HCEnc is amazing. Thank you for all your hard work.

The other day, I was wondering if it would be possible to add a tab, similar to the Preview/Zones tab, to select different sections of the video to be encoded as progressive or interlaced. This would be great for encoding files from mixed sources, and even better if 24p (soft pulldown) could somehow be incorporated, effectively allowing for variable frame rate MPEG2. I don't know if enough people would use it to make it worth adding, though.

Thanks again.

[manolito]

@Emulgator
All you are saying makes perfect sense, but it is all based on theory and assumptions. Would you be so kind to make some side by side tests using HC, once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way, but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Cheers
manolito