HCenc 023 released [Archive] - Page 4

Revgen

23rd September 2009, 13:42

Well, there's a problem.

I just encoded a tv capture and HCEnc 0.24 (latest beta) is adjusting the bitrate by -4.46%. While HCEnc 0.23 has done this at times in the past, it never went over 1%. 4.5% in any direction is way too much.

Here's the .ini file settings.

*BITRATE 6900
*MAXBITRATE 9500
*FRAMES 0 123672
*PROFILE best
*ASPECT 4:3
*AUTOGOP 12
*AQ 0
*DC_PREC 10
*PROGRESSIVE
*PULLDOWN
*INTRAVLC 2
*MATRIX mpeg
*COLOUR 4

I'll try encoding this with HC 0.23 tomorrow and see if the same behavior happens.

EDIT 1:

Other TV caps are experiencing 10% bitrate deductions.

EDIT 2:

Just finished encoding with HC 0.23. Encode size is what it should be. So something is definitely wrong with the way HC 0.24 calculates bitrate. Later today, I'll see if I can replicate the problem on a smaller sample.

hank315

23rd September 2009, 20:22

@Revgen
Probably this new beta will saturate easier.
The automatic intravlc is pretty effective, also quantization with a small deadzone is now used.
Please try adding the next line:

*DEADZONE 0 0

This will disable the deadzone quantization, just to see if it makes a difference.
Also a visible comparison between the two would be interesting.

Thank you , Hank, for the new beta.
HC still crashes in Lossless mode with AUTOGOP 12 ( not with AUTOGOP 13).
http://forum.doom9.org/showthread.php?t=140361 (post #15).
Thanks in advance.
I can't replicate this, does it crash at every segment of the movie?

mikenadia

23rd September 2009, 21:06

Hank, this is the ini file.
*INFILE c:\working\d2vavs\v05000100001002.avs
*OUTFILE C:\working\D2VAVS\V05000000001001.m2v
*DBPATH C:\working
*LLPATH C:\working
*BITRATE 4000
*MAXBITRATE 8000
*FRAMES 0 7571
*NOSMP
*PROFILE fast
*AUTOGOP 12
*NOSCD
*LOSSLESS
*INTRAVLC 2
*MATRIX mpeg

the avs is loaded (in info:input using ini file: allocated memory).
It happens in all avs file (generated by DVD-RB) of different movies.

#------------------
# AVS File Created by DVD Rebuilder
# VOBID:01, CELLID:01
#------------------
LoadPlugin("C:\Program Files\DVD Rip Prog\DVD-RB\DGDecode.dll")
mpeg2source("C:\WORKING\D2VAVS\V05.D2V")
trim(0,7571)
ConvertToYV12()

If I shift the AUTOGOP number from 12 to 13 back and forth, it is, on my setup, consistent behaviour.
Thanks for your help.

Revgen

24th September 2009, 06:43

The deadzones workaround worked.

Here are bitrate differences for the -10% video. Turns out it actually lost -15% bitrate.

http://img34.imageshack.us/img34/8497/hc24bitrate.th.jpg (http://img34.imageshack.us/i/hc24bitrate.jpg/)

The bitrate I entered was 6300, which HC23 and HC24 with deadzones seem to match closely. HC23 is about 13kbps short, but it's not too bad. HC24 with no deadzones is really bad, and it's doing this across the whole video, not just certain sections.

This movie is a b&w movie about miners and there are several dark mine scenes. The very low bitrate section on the far right is one of those darker scenes.

As far as how the image looks, the movie is b&w plus it's pretty dark at times, so I actually have to look for the blocks to see them, but when I do I can see the difference.

Emulgator

24th September 2009, 09:40

Many thanks to Hank again !
I was checking with poor, dark and grainy PAL-SD DV50 source @ 4950kbps.

Qualitywise you are so close to CCE @ comparable settings
that I have a hard time to spot any difference.

Because this grainy stuff tears the most regarding this half-starved bitrate
both encoders exhibit the same grain-modulated blockstuctures if stepped frame-wise.
OK on normal playback.

Very good work !

My 2 test encodes so far hit the bitrate.
First (LUMGAIN=4) @ 4900kbps hit the 4900.
Second encode LUMGAIN=0,AQ=3 @ 4950kbps hit 4951.

(Konran's BitrateViewer in GOP enhanced mode used)

Using AQ=3 both bitrate distributions (HC vs. CCE) have been very similar and useful.
The last part of the source containes smoothed and deinterlaced material
and both encoders managed to save on bitrate here.

weaker

24th September 2009, 11:08

Thank you Hank for further continuing with HCenc development! And thanks to those who beta-test it.

manolito

26th September 2009, 14:35

Thanks a lot Hank for the new version. Did a couple of tests in 1-pass VBR mode, works great!

I am curious about the new features. Is the *MINBRFAC command an equivalent to a Min Bitrate parameter? Is raising the value above the default 1.0 recommended to help avoid artifacts in dark scenes?

And more questions about the undocumented commands:
Even though you probably do not want the average user to play with these settings, could you elaborate a little about them?

I did google for Deadzone Quantization, but frankly this stuff is way above my head. Is it generally a good thing to use it, or are there situations where it should be turned off?

Other undocumented commands I found are
*MASK_VALUE
*MPEGLEVEL
*NOSMPDECODE
*QROUND
*PSYOFF

Could you explain just a little what these commands do?

Thanks again,
Cheers
manolito

hank315

4th October 2009, 23:00

@Revgen
Thanks for testing.
Your source is of course not the usual kind but it shows the settings are not optimal yet for all conditions, I will make it adaptive for different sources, bitrates etc.

I am curious about the new features. Is the *MINBRFAC command an equivalent to a Min Bitrate parameter? Is raising the value above the default 1.0 recommended to help avoid artifacts in dark scenes?
It manipulates the lower part of the compression curve, so yes higher values will prevent artifacts in dark scenes.

I did google for Deadzone Quantization, but frankly this stuff is way above my head. Is it generally a good thing to use it, or are there situations where it should be turned off?
Quantization is only a mapping of DCT values to a (smaller) set of discrete values, using a deadzone means small values are easier quantized towards zero.
So some noise will be quantized to zero, useful for low bitrate encodes but for high bitrate encodes it will saturate easier.

There are some hidden commands meant for testing purposes:

*MASK_VALUE Ymask Umask Vmask
If *MASK_SHIFT is used, these values are used for masking the top/bottom bars.
Default: Ymask=16, Umask=-128,Vmask=-128

*MPEGLEVEL level
HCenc will automatically set the level according to the resolution of the movie, with this command you can force the MPG level.
Possible values for level: MP@ML, MP@H-14, MP@HL.

*NOSMPDECODE
Normally HCenc encodes frames for GOP n and reads/processes input frames for GOP n+1 (look ahead) at the same time in different threads, this command just disables the parallel encoding/reading, it will not disable parallel encoding.

*QROUND R1 R2
The MPEG standard exactly describes how to decode an MPEG stream but a programmer is free how to quantize the frequency domain data (after forward DCT).
Part of the intra quantizing: (3 * Q + 2) >> 2, HCenc uses (R1 * Q + R2) >> 6, default: R1 = 48, R2 = 32, best to leave it at the default values.

*PSYOFF
This command disables fade detection and some other luminance on dark scenes adjustments.

manolito

5th October 2009, 11:57

Thanks Hank for your detailed explanations.
Very useful for me :)

Cheers
manolito

Emulgator

7th October 2009, 21:43

Many thanks from me too and I as well will continue to test
and maybe utter some helpful thoughts to bring HC to the top...

MilesAhead

20th October 2009, 02:19

Are there sort of "nightly builds" or stuff newer than Sept. beta for those of us who want to mess around "updating" converter suites, to download? Sept. beta is the newest I can find.

hank315

22nd October 2009, 23:13

Are there sort of "nightly builds" or stuff newer than Sept. beta for those of us who want to mess around "updating" converter suites, to download? Sept. beta is the newest I can find.Here's a new one.
Deadzone quantization is now adaptive, see HCgui tab settings 1.

link: HCenc024 beta (20-10-2009) (http://hank315.nl/files/HC024_beta_20-10-2009.zip)

Chumbo

22nd October 2009, 23:26

@hank,
I just started using hc023 again for some encodes I needed, but this time under Windows 7. The program works just fine, however, when the encoder closes after it's done and when you attempt to exit the GUI, it causes a program "crash." This happens consistently. When the encoder GUI finishes, you get something like "the mpeg2 encoder has stopped working" and when you click Exit on the normal UI, you get something like "HCgui 0.23 stopped working" and so on.

You can then click on Cancel or Close Program to get out. I've tried running in Windows XP and Windows 2000 modes but I get the same thing. It may be that I'm running under the 64-bit version? I thought you may want to know. Is there any chance to get a 64-bit version of HC023?

[EDIT] I tried the latest 024 beta from the link above and the GUI exists cleanly which is nice.

[EDIT2] Well, I was wrong. It looks like the exit has to do with whether an hc.ini file exists or not. After I finished an encode with 024 beta, the same thing happened with the encoder and then with the GUI. After I removed hc.ini, the GUI exits without the "crash."

MilesAhead

23rd October 2009, 07:31

Here's a new one.
Deadzone quantization is now adaptive, see HCgui tab settings 1.

link: HCenc024 beta (20-10-2009) (http://hank315.nl/files/HC024_beta_20-10-2009.zip)

:thanks:

Dkruskie

24th October 2009, 03:39

Thanks Hank

Blue_MiSfit

26th October 2009, 18:36

Nice! Adaptive deadzones huh?

HCenc is quite marvelous ;)

~MiSfit

stenews

26th October 2009, 21:39

@Emulgator
I saw from a couple of images you attached in this tread that you use an "unseen" bit rate viewer to check the *.m2v.
How can I get it?

@manolito
hi mate,
how do I set up this new version to make it works with D2S?
I mean, do I need just to replace it with the new one or what else?
I don't remember the way you explained me before.

Bye and thanks to both of you.
Stefano.

manolito

26th October 2009, 23:21

how do I set up this new version to make it works with D2S?
I mean, do I need just to replace it with the new one or what else?
Yes, and you should rename HCenc_024.exe to HCenc.exe.

Cheers
manolito

Revgen

27th October 2009, 00:47

Thanks Hank. I'll check it sometime this week.

stenews

27th October 2009, 09:17

@manolito,
many thanks for your kindly reply :)
which is the most great improvement of this new beta release?
I mean, could you suggest any extra command to add to my HC.INI?

Bye,
Stefano.

Emulgator

27th October 2009, 16:36

Konran's Bitrate Viewer is here:

http://www.winhoros.de/docs/bitrate-viewer/index.html#top

stenews

27th October 2009, 20:28

@Emulgator,
thanks a lot mate!

Bye,
Stefano. :thanks:

manolito

28th October 2009, 03:26

which is the most great improvement of this new beta release?
I mean, could you suggest any extra command to add to my HC.INI?
Hi Stefano,
all the latest HC improvements work very well using the defaults, so I do not have any extra commands in my HC.INI. The only commands different from the default values I use are
*LUMGAIN 2
*AQ 2
because they help a lot in avoiding artifacts in dark areas. And so far I have not noticed any adverse effects with these commands, so I use them all the time.

Cheers
manolito

Emulgator

28th October 2009, 11:54

manolito, I did not know too well: So I can have LUMGAIN and AQ together ?
I thought before to have read that if one specifies LUMGAIN > 0 then AQ would be disabled in HC.
But I would be happy to hear that I can specify both and both would be respected...
...I don't dare to check now for myself, x264 and extensive script frontends running for 20 more hours of testing...

stenews

28th October 2009, 19:20

@manolito,
sorry for my delayed reply, I have to say many thanks again for your great tips! :thanks:

hank315

28th October 2009, 20:59

manolito, I did not know too well: So I can have LUMGAIN and AQ together ?
I thought before to have read that if one specifies LUMGAIN > 0 then AQ would be disabled in HC.
But I would be happy to hear that I can specify both and both would be respected...
Yes, LUMGAIN and AQ can be used together, in the latest releases AQ = 2 is the default value.

Emulgator

29th October 2009, 22:05

Many thanks, Hank !

Revgen

31st October 2009, 17:11

This latest beta has been working fine so far. Quality looks great too.

manolito

1st November 2009, 00:55

This latest beta has been working fine so far. Quality looks great too.
Same here :)
Time to go stable?

Cheers
manolito

ron spencer

1st November 2009, 02:19

works for me too!!! Nice job. Would be nice to have an expert user document the new features though...in any case great!!!!

sunfish

5th November 2009, 07:01

Is there any reason why HCenc is only running at 70-75% CPU load? I believe running it at 100% at low priority is more optimal.

hank315

7th November 2009, 01:22

Is there any reason why HCenc is only running at 70-75% CPU load? I believe running it at 100% at low priority is more optimal.
It's because the way multi-threading works in HCenc, it's frame based. Because of the frame dependency the best it can do is encode 3 frames at the same time (P and 2 B-frames).
In the same time DGDecode or Avisynth decodes the input frames, this all results in 70-90% CPU load on a quad core.
It can be worse, if you use heavy filtering in Avisynth, CPU load can drop to 25-30% on a quad core, most filters are single core only, so the encoder has to wait for the input data.

mikenadia

27th November 2009, 20:51

hank315

29th November 2009, 22:00

For a given video, I have a bitrate of 3100kbps and avg quantizer of 5.5 for matrix Rockas5 and 6.5 for matrix Rockas4.
I was wondering if it were not possible to modify the matrix, after the 1st pass, to get closer to a desired avg quantizer of 6.0 for example.
Hc 3100 6.0 as a command (2-pass with a targeted avg quantizer).

We could also create a set of family of matrices that depends on only one parameter ( forcing the matrix to be symmetrical with same coefficient for i+j= Cte) and knowing the frequency spectrum ,find the parameter that will achieve the compression desired.

Another suggestion: a function to find CQ for a given bitrate. It is my undersanding that HC will be in a better position to make a better educated guess after a first pass than using Q* Size=Cte.

Creating a family of matrices to predict a desired Q should be possible, maybe something for the next release.
What do you mean with "forcing the matrix to be symmetrical with same coefficient for i+j= Cte" ?

cosmetics.
May be put the current fps to 0 when in "suspend" mode. That's a simple one, done...

BTW, here's (http://hank315.nl/files/HC024_beta_27-11-2009.zip) a new build, it fixes some bugs in the dead zone quantization and has 3 new matrices (thanks manono).
Still beta but almost final, working on some speed optimizing issues.

mikenadia

30th November 2009, 03:47

By looking the different Rockas matrices, I realized that most of them were having coefficients that were almost equal along the bottom-left to top-right diagonals:
From Rockas3.
*CUSTOMMATRIX
08 08 09 11 13 13 14 17
08 08 11 12 13 14 17 18
09 11 13 13 14 17 17 19
11 11 13 13 14 17 18 20
11 13 13 14 16 17 20 24
13 13 14 16 17 20 24 29
13 13 14 17 19 23 28 34
13 14 17 19 23 28 34 41

I assume that this matrix will have similar results than
*CUSTOMMATRIX
08 08 09 11 13 13 14 17
08 09 11 13 13 14 17 18
09 11 13 13 14 17 18 19
11 13 13 14 17 18 19 20
13 13 14 17 18 19 20 24
13 14 17 18 19 20 24 28
14 17 18 19 20 24 28 34
17 18 19 20 24 28 34 41

This matrix is defined by 15 coefficients (a11 to a18 and a28 to a88). Those 15 coefficients are in increasing order , could be defined with one parameter as a function of (a11=8,a88-a11=41-8=33) and all the coefficients could be adjusted with a factor "t" to get close to a targeted quantizer, the adjusted matrix, in this example, being a function of (a11=8,t*(41-8)).

Thanks for the new build,hank.

Edit 1: you could also have (I think it is overkill) have a second degree of freedom on each diagonal if you want the coefficients on each diagonal to have small variance between themselves.

Edit 2:Cosmetics in the GUI: it will be nice if Manono could provide us with a pct of Fox1 (or any other one) for his matrices (someone posted for example mpegstd=45 % and Manono2= 63 %) and if that "info" was added in the Matrix Tab. Or a bitrate range.
Also, to go to custom matrix, you have to unclick built-in matrix. The custom matrix appearing by default could be the latest built-in matrix instead of MPEG to make small changes or allow us to save to mtx file.

ron spencer

30th November 2009, 12:55

thanks....how does the min bitrate thing work? awesome work!!!

hank315

2nd December 2009, 22:27

how does the min bitrate thing work?
In a 2 pass encoding HCenc always starts the first pass with a Quantizer = 4.
Using this encoding data a bitrate curve is created, quantizers for I, P and B frames are estimated for each GOP for the second pass.
The min bitrate factor will tweak the lower boundary of the bitrate curve, a value > 1 will raise the lower boundary of the curve meaning lower quantizers will be used for the low bitrate parts of the movie.

Cosmetics in the GUI: it will be nice if Manono could provide us with a pct of Fox1 (or any other one) for his matrices (someone posted for example mpegstd=45 % and Manono2= 63 %) and if that "info" was added in the Matrix Tab. Or a bitrate range.
A compression percentage for all built-in matrices with respect to the MPEG matrix would be nice, it requires a lot of of tests to get an average value for each matrix.

Also, to go to custom matrix, you have to unclick built-in matrix. The custom matrix appearing by default could be the latest built-in matrix instead of MPEG to make small changes or allow us to save to mtx file.
Nice idea, will implement such a thing.

mikenadia

3rd December 2009, 03:19

From Hank315:The min bitrate factor will tweak the lower boundary of the bitrate curve, a value > 1 will raise the lower boundary of the curve meaning lower quantizers will be used for the low bitrate parts of the movie.

Is this adjustment a personal taste one (independent of source, bitrate...)? If it is source dependent, could it be based on the stdev of the expected quantizer distribution after 1st pass (value>1 if stdev is high).

Also, when you say first pass starts at Quantizer=4, it is Quantization at 4 Mbps or CQ=4 or just " starts at Q=4"?

Thanks in advance.

Edit: a link on MPEG2 FIFO adjustments (probably of no interest to HC).
http://forum.handbrake.fr/viewtopic.php?f=4&t=13396

hank315

4th December 2009, 21:41

Is this adjustment a personal taste one (independent of source, bitrate...)? If it is source dependent, could it be based on the stdev of the expected quantizer distribution after 1st pass (value>1 if stdev is high).
The default minimum bitrate is source dependent, it's mainly based on the maximum and average bitrate from the first pass but also on luminance.

Also, when you say first pass starts at Quantizer=4, it is Quantization at 4 Mbps or CQ=4 or just " starts at Q=4"?
It's CQ = 4, during the whole first pass.

a link on MPEG2 FIFO adjustments (probably of no interest to HC).
No, not really, HCenc encoding speed is independent of the size of the allocated buffers, only if you hit the limit (allocated memory is close to memory left) you will run into trouble.
In HCenc the allocated memory is almost linear with nMB * GOPlength (nMB = nr. of MacroBlocks per frame).

~Revolution~

6th December 2009, 07:41

hank315

16th December 2009, 23:56

The next table shows the compression ratio for the built-in matrices.

Hcenc 024 beta testclip1 @ Quant = 4
matrix size (bytes) factor

mpeg 41250657 1.0000
qlb 41213068 0.9991
notch 38494522 0.9332
bach1 36971188 0.8963
hc 40922436 0.9920
hclow 44994171 1.0908
jawor1cd 35544248 0.8617
hvsgood 39084618 0.9475
hvsbetter 40610182 0.9845
hvsbest 42612575 1.0330
avamat6 32298950 0.7830
avamat7 31357828 0.7602
fox1 97076189 2.3533
fox2 82854942 2.0086
fox3 56116450 1.3604
manono1 53362584 1.2936
manono2 54737764 1.3270
manono3 46335685 1.1233
mpegstd 50305606 1.2195
It's based on one clip only so it's a rough estimation.

AlanHK

23rd December 2009, 11:18

I'm encoding an animation with some fairly strong filters.

The first pass went at 1.6 fps, but the second pass is much slower, 0.6 fps.
The PC isn't doing anything otherwise intense to reduce the priority.

I've never seen such a large difference between first and second passes.

The AVS file:
AVISource("Flat.avi",false)
UnDot()
Dehalo_alpha(darkstr=0.5,brightstr=0.5,rx=3,ry=3)
Edgecleaner(strength=20)
Toon(strength=0.8)
Deblock(quant=35, aOffset=0, bOffset=0, mmx=true, isse=true)
LSFMod(strength=50)
Dup(threshold=6.0, chroma=false, show=false, copy=true, maxcopies=20, blend=true)
LanczosResize(704,576).AddBorders(8,0,8,0)

And the HC command:

HCenc_024 -i %%~fA -o s:\mpg\~nA.m2v -b 2251 -aspectratio 16:9 -pulldown -profile best -matrix qlb -frames all -noini -2pass -maxbitrate 8000 -log %%~dpAhenc.log -avsreload

hank315

23rd December 2009, 22:44

Don't know why this is, both seem *very* slow...
In this case using -lossless will certainly speed up the second pass.

AlanHK

24th December 2009, 05:48

Don't know why this is, both seem *very* slow...
In this case using -lossless will certainly speed up the second pass.

Thanks, but I don't have the space for a 70 minute lossless file.

But though the slowness is irritating, it's the disparity between passes that seemed strange.

Another oddity: it's now halfway through the 2nd pass, after about (or maybe exactly?) 24 hours. But the "Time used" has become 13 minutes, and the average speed is insanely high, though current speed is still 0.6. Looks like the timer has rolled over.

I really hope it gets through without blowing up, it would be very annoying for it to abort after 3 days....

Blue_MiSfit

30th December 2009, 01:05

just a note, I did some testing with 023 and compared it to ProCoder's "mastering quality", and HC blew it out of the water at all bitrates for SD :)

~MiSfit

tom942

3rd January 2010, 03:30

What is the best situation in which to use the MANONO1, MANONO2, and MANONO3 matrices? And how do they compare to the FOX matrices and other matrices? I always get a bit confused as to how all of these matrices rank but I do have an idea that it has to do with the value of the bitrate output of the encode.

In one of the several PM's that Manono sent me, he told me, more or less, this:

FHE (very similar to FOX1) matrix is used as reference. The resulting testclip is marked as 100%. Then the other matrices are marked with bigger or smaller percentages, taken into account, their level of compressibility (final size of the encoded clip).
Matrixes with low compression will have a bigger percentage than FHE.
Matrixes with high compression, will have a smaller percentage than FHE.

Manono1 matrix could be used as an enhaced version of MPEG matrix (83/16). It is sharper than MPEG (to my eyes) and compress fine.
As a guide, in DVD-RB I use it normally between 3000 to 4000 kpbs.
In the tests done by Manono, he got a 68%.

Manono2 matrix is an "old" matrix, similar to the previous one, but compress more and it is bit less sharp (to my eyes) than Manono1.
In this one, he got a 63%.

Manono3 matrix could be used as an enhanced version of MPEG_STD matrix (83/33). It is sharper than MPEG_STD (to my eyes) and compress fine. As a guide, in DVD-RB I use it normally between 2000 to 3000 kpbs.
In the tests that he done, he got with MPEG_STD a 45%, so as this one compress a bit less, it could be a 48-50%.

I hope it helps :).