HCenc 023 released [Archive] - Page 5

Emulgator

3rd January 2010, 13:13

Hank, Happy New Year and my highest appreciations for HC 0.24 beta 27-11-2009!
This version finally kicks almost everything from the shelf.
Using no deadzones I can finally have dark and grainy parts encoded without the well-known block artifacts.
Grain stays grain as in CCE, no moving dark blocks as in Procoder (Express 2 I checked on)
or chequered-structure-changing-blocks as in Mainconcept.
I can force bitrate usefully into these dark parts as I like. No bitrate flipping as in CCE.
Picture is still sharp, even using MPEG Matrix.
Almost no visible artifacts encoding text onto gradients.
High motion is usefully encoded, no heavy block rows for 1 frame as sometimes briefly appearing in TE4XP.
I would say, I see HC on par now with CCE, better than Procoder, better than TE4XP, better than Mainconcept.
Very good work and the only MPEG-Encoder I have seen improving in terms of picture quality for the last 2 years.
And for now I have only tested SD resolutions...

Gromozeka

3rd January 2010, 18:54

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...

tin3tin

12th January 2010, 12:13

The HCenc024.exe Beta 27 - 11 crashes for me in the end of first pass(says 100%) with a 16 sec long avs script using the Soundout plugin too. It renders without errors in Hcenc 0.23.

This is the ini:
*dc_prec 10
*WAIT 0
*BITRATE 7000
*MAXBITRATE 7500
*AUTOGOP 15
*AVSMEMORY 1024
*PROFILE BEST
*PROGRESSIVE
*PRIORITY HIGH

manolito

17th January 2010, 01:53

First time that 0.24 beta got the sizing wrong in 1-pass VBR mode.

This is the deal:
Source: 90 min MPEG2 from a DVB-T capture.
Reason for reencoding: Logo removal
Encoder: Latest HC 0.24 beta Novemver 2009
Matrix used: Manono1
No filters except LogoAway, no resizing
Source characteristics: Last 4 min consists of vertically scrolling end credits, requires high bitrate
Average Bitrate: 6400
Max Bitrate: 8500
Base Q determined by HC: 2.20

The encode came out almost 4% oversized. Next thing I tried was disabling Deadzone Quantization, but the oversize was the same (speed was significantly faster, though).

Out of curiosity I did the same encode with an older 0.24 beta from May 2009, but no change, still almost 4% oversized.

And the last thing I tried was throwing this source to my old and obsolete 1-pass VBR plugin for DVD2SVCD (Hybrid Mode), and this encode came out perfectly (1% undersized).

Ok, this is purely academic. I used DVDShrink to shrink the size of the oversized encode and compared the result to the encode using my old plugin, and I was completely unable to notice any visual difference.

Cheers
manolito

manolito

20th January 2010, 01:12

I just ran into some nasty problems using the latest HC 0.24 beta in 1-pass VBR mode.

Source:
DVB-T capture, interlaced, average bitrate ~3500 kbps

Encoding settings:
HC 0.24 beta 1-pass VBR mode
Average bitrate 4950 kbps
Max bitrate 8500 kbps
GOP: Auto
Interlaced: Auto
IntraVLC: Auto
DeadZone: Auto
DC Precision: 10
Scene Change: On
Matrix: Manono2 (aka Sony medium)

Using my old and obsolete 1-pass VBR plugin (which is basically a CQ_MAXBITRATE mode with a couple of tweaks to ensure correct sizing) with the same HC settings these artifacts were all gone. This was also true when using HC 0.23.

Download the test clips here: http://www.mediafire.com/?jztmtj1kqum

Cheers
manolito

Emulgator

20th January 2010, 09:28

Oops. opv.mpg had a bitrate valley allocated at the scenecut, where a bitrate peak should have been,
especially with scene change on.

Hm. looks like poor bitrate allocation in 1-pass VBR?
In my tests I was only checking 2-pass VBR.
and this is what I use regularily together with Matrix MPEG.
(Other matrices did not help too much at my side.)

Emulgator

20th January 2010, 09:33

tin3tin
*PRIORITY HIGH
what if you leave priority at "idle"?
"High" may force its way through too much, while not offering any gain in speed,
not letting other processes in, which you may need for task to complete.

txporter

20th January 2010, 21:30

Can someone step me through the correct way to encode using 1-pass Quality VBR with HCenc? Is this similar to doing CRF encoding with x264?

I can get HCenc to encode using 1-pass, but it requires me to given it an average bitrate target. I guess I am expecting something more like x264 where I tell it I want some level of quality and it uses whatever bitrate is necessary to do this. Is that constant quant in HCenc?

Sorry for the beginner level questions. I have used HCenc for a while now with 2-pass target bitrate, but I wanted to test various avisynth functions out for compressibility and I can't see how to do that when I need to give a target bitrate.

edit: I am using HCenc 0.24 11-27-09 release

hank315

20th January 2010, 22:35

@Emulgator & manolito
Seems 1 pass VBR is seriously broken in the latest 024 beta.
In my development version I even get random crashes if all compiler options are on, running in debug mode is OK making it hard to track down the error :devil:

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...
Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

txporter

20th January 2010, 22:59

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

Thanks, Hank! It does work in HCenc_023. I can use that to test different filters and then just apply that to 2-pass HCenc_024. I like targeted bitrate just fine, I just didn't know how to tell what effect my filters were having. Thanks a lot for your support and your program!

manolito

21st January 2010, 14:51

But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.
Huh? HC 023 had a 1 pass VBR mode already? How stupid of me, I thought this feature was introduced in the first 024 beta :devil:

Just FYI I reencoded the clip which gave me those problems using the very first 024 beta (Nov. 2008), and the artifacts are identical. So this bitrate distribution problem in 1 pass VBR mode must have been there from the beginning.

The clip is a TV broadcast from last New Year's Eve, it is "Fleetwood Mac live in Boston 2004". The artifact comes up at the end of the last song, and this last song is a lot less compressible (requires higher bitrate) than most other parts of the clip. It looks like HC is desperately trying to reach the desired target size, so it does not even allow bitrate peaks at scene changes.

Anyway, I will give it another shot with yet another 024 beta version and report back...

Cheers
manolito

manolito

22nd January 2010, 23:08

In the meantime I ran the same encode using 3 different 024 beta versions, but the results were identical.

Conclusion: The 1-pass VBR mode strategy in HC 024 can fail under certain unfavorable conditions.

The source is really very demanding. But OTOH HC CQ_MAXBITRATE and HC 2-pass VBR do handle this source without problems, and even QuEnc 1-pass VBR can do it. I tried to cut out a 5min segment of the source with the problematic spot right in the middle, but this was no problem for HC 024 1-pass VBR mode.

In order to test different bitrate distribution strategies it is probably necessary to process the whole clip. The size is 1.2 GB, let me know if I should upload it....

Cheers
manolito

BTW I also tried to use *PROFILE BEST instead of FAST, but same result...

Gromozeka

23rd January 2010, 08:20

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

You will add her(its) in the future time?
Thank you, This very good function.
:thanks::thanks::thanks:

Emulgator

25th January 2010, 13:13

An interesting finding while further comparing HC vs CCE.
Source was 16mm Film (1.38:1), Duration 1:52:20, Transfer, PAL speedup to interlaced PAL-VHS, ugly and grainy.
I restored this PAL-VHS 25i to 25p using TGMC, selecteven, MC_spuds. No residual combing visible.

Now encoding using CCE, 2-Pass VBR @5,1Mbps, "Progressive frame"
and HC0.24 beta 27-11-2009, 2-PassVBR @5,1 MBps, "Progressive".

A: Comparing results on Notebook-TFT 1920x1200 using MPC+motion vector overlay, resized 200%, VitualDub resized200%, Konran's Bitrate viewer.
HC (tweaked: VBR bias 10, minBRF 1.20, no deadzones, matriy MPEG) shows more useful bitrate distribution than CCE,
Especially in the beginning, where CCE comes slowly up from poor bitrates, leaving some blocks in dark parts.
HC swings faster in to average bitrate with a light overshoot and starts from the very first frame with a smooth picture.
HC finds very useful motion vectors as well.

Later after some minutes both encodes look similar. No faults that would be visible without hard scrutiny.

B: Comparing results from DVD after authoring (DVDLabPro2.51) and burning to disc.
Panasonic BD-50 -> BD50-upscaler -> HDMI -> Loewe Connect 37 (TFT-TVFullHD37")
Here the TFT showing the DVD coming from HC sometimes shows some thin glitches,
height 1 single 1080HD line and width maybe 16 HD pixels
that were not part of the MPEG-2 stream before authoring.
These line glitches were only visible during 1xplay
and could not be seen when pausing and stepping frame by frame using RC.

(Later I saw another 23.976p pulldowned DVD exhibiting the same behaviour on this particular player/TFT combination.
The reason could be tracked down to the Panasonic BD-50's upscaler, when progressive frames should be displayed.
No problems on interlaced frames.
Panasonic DMR-EX95, while having an uglier upsizer with bad ringing, did not exhibit these glitches from the same DVD.
Phew, so not my restoration/encoding fault...)

Now comparing streams using DGIndex and ReStream.

HC: after specifying "Progressive", in GUI TFF/BFF Options are greyed out.
HC then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=no
Sequence Extension
"Progressive Sequence"=yes,
In GOPs: no tff flags.
Applies Scanning mode zig-zag.

Now CCE:
after specifying "Progressive Frame", in GUI TFF/BFF Options are not greyed out.
CCE then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=yes
Sequence Extension
"Progressive Sequence"=no,
In GOPs: tff flags=yes.
Applies Scanning mode zig-zag.

Now I reflagged the HC stream to the same settings as CCE would output
using ReStream.
Authoring, burning, disc testing as before:

Now HC output looks smooth and ok on this particular BD-player/TFT-TV combination!

Then I forced interlaced encoding on progressive footage in HC
but then alternate scan order was forced by HC as well
and so I lost quality, picture became a bit blocky then.

It seems that these flags exhibit a bug in player/resizer/TFT-TV combination
that can be avoided if one reflags HC output as interlaced, even if 25p has been encoded.

To overcome this possible trap with 25p footage,
and if Hank approves this approach
I would suggest to extend GUI behaviour.

After "Assume input frames as progressive"
(info: This does not mean that HC would deinterlace !)
I'd still like to have PictureCodingExtension TFF,BFF tickable,
because no tff=1 == tff=0 == BFF;
I'd suggest to have SequenceExtension "Progressive Sequence" tickable/untickable;
(info: Unticked helps for DVD-compatibility)
and I'd suggest to have Block scan order manually settable using 3 radio buttons
"Auto" (info: Applies "zig-zag" on assumed progressive frames, "alternate" on assumed interlaced frames) or
"Override to zig-zag" (info: Useful on progressive frames, interlaced may suffer) or
"Override to alternate" (info: Useful on interlaced frames, progressive may suffer)
Fenced meaning info line in GUI that declares the use of these flags.

manolito

25th January 2010, 22:01

@Emulgator,

interesting findings, and for the most part they are consistent with my experiences...

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC. Any specific reasons? Does it generally kill grain in dark areas?

For the different TFF flagging of progressive streams in CCE and all the rest of MPEG2 encoders I had my problems a while ago (and there was a thread here which I cannot find right now), but here is the summary:

According to the MPEG2 standard the TFF flag has no meaning whatsoever for progressive content. A software- or hardware player should simply ignore it. But in reality at least some (maybe most) standalone DVD players do not ignore the TFF flag for progressive content.

I have an old and cheap Cyberhome DVD player (Mediatek chipset) connected to a CRT TV via SCART (using SVHS). The player always outputs fields (interlaced), and it does so observing the TFF flag, no matter if the stream is progressive or interlaced. For progressive content this does not make a difference, the TV picture will look the same.

The problem I had a while ago was that my encodes from TV captures had jumpy end credits. The film itself was progressive, but the credits were interlaced TFF. When I started with DVD conversions I always used CCE SP 2.50, and these encodes looked good because CCE always flags a progressive encode as TFF. But when I started using QuEnc and HC I got these jumpy end credits. Simple reason: QuEnc as well as HC do not allow to set the TFF flag for progressive encodes, so my end credits simply had the wrong field order.

The reason why CCE flags progressive encodes as TFF is probably historical. Older CCE versions would always produce TFF output. If the source was BFF, they would apply an offset of 1 scan line, so the output would be TFF. Newer versions can do it both ways, but the flagging has not changed.

Ok, the "right" solution for this problem would be to encode the film and the credits separately, but that was too much hassle for me. I asked DGZ (co-author of QuEnc) if he could modify QuEnc to allow TFF flagging of progressive encodes. He checked it, but libavcodec would not allow it. Same is true for HC: if you specify progressive, then the TFF flag cannot be set, and scan order will always be zigzag.

Well, reading your post it seems that newer upscaling standalone players also do evaluate the TFF flag even if the content is progressive. As a solution to this problem (except using CCE) I normally use HC's "Automatic" interlaced mode. If you specify neither *PROGRESSIVE nor *INTERLACED nor *DVSOURCE then HC will try to detect interlaced or progressive per frame automatically. In this "auto interlaced" mode TFF is the default, but you can change this. Scan order is also determined automatically in this mode. Works very well for me, give it a try...

Cheers
manolito

txporter

25th January 2010, 22:23

This is an interesting discussion. I started using HCenc a few months ago to re-encode DVD rips (mainly NTSC material) into single file MPEGs with hardcoded subs. I do this to serve videos up to my Tivos. At any rate, I started by encoding using the automatic detection method you described above. The videos produced from this played fine in MPC-HC. When I watched them in WMP10 though, they exhibited what I was calling strobing. For a short period of time the video was fine, and then it seemed to jump up and down a few scan lines at a time, and then would return to normal and repeat. At any rate, my tivo playback looked like the WMP playback with strobing video/smooth video/strobing video..etc. It was extremely annoying.

Well, I was able to avoid the strobing video by specifically encoding as either progressive or interlaced. Frankly, I was really only encoding NTSC material at that point, so I don't know if interlaced material showed the problem...but progressive did. I wonder if this was somehow also related to the field flagging?

Emulgator

26th January 2010, 19:57

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC.
Any specific reasons? Does it generally kill grain in dark areas?
Hm, I simply fear to sacrifice something important where I could easily have given more bitrate.
For instance Procoder (2 Express) could be good, finds useful motion vectors even in the dark,
but blows it when giving bitrate there. A miser and therefore blocky.
Because only a few Y steps (16~32) are left from 16-235 to encode dark tones close to black,
I tend to value reproduction quality of dark areas higher than average.
Lots of Film lives from smooth darkness.
It may well happen that encoded output is watched on TFT with pushed levels, wrong settings etc.
Then anything blocky in dark parts becomes painfully visible and I hope to finetune my encodes to look better than average.
Even when pushed it should look smooth.
For testing I even go into a windowless and completely dark room,
doors closed, watching footage on a Laptop or portable TFT,
eyes offset from center, viewing from a top angle, just to find blocks in the dark.
If my encodes pass this, then I'm feeling safe.
As a sidenote: Sonic reps even suggested that compressionists using Cinevision
should tell editors to clip their shades below a value of 3 or 4 to pure black.
So a step from pure black to the next value of 3 or 4. ??
Introducing deadzones, therefore destroying shades. What a deal...
Do they fear to exhibit encoder flaws? I guess so.
I can be done better...
..but progressive did. I wonder if this was somehow also related to the field flagging?
I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

txporter

26th January 2010, 20:40

I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

This is the first time that I have ever tried ReStream. For both cases, I am taking a 2min video clip that is 97.85% FILM according to DGIndex. I built a d2v file using Honor Pulldown flags and am running the video through TFM().TDecimate(). The input video to HCenc should be 23.976 fps progressive.

If I force HCenc to do progressive encoding, the video plays fine when I upload to my tivo. ReStream has tick marks on Frametype progressive and Progressive sequence. The field coding shows it as BFF with ----- as the tff-flag.

If I let HCenc use automatic interlace detection, the video experiences the strobing (not verified, but this is how I did it a few months ago when it did). ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.

In both cases, the scanning-mode is listed as zig-zig. Although I am not sure if that is just the type most often used or what.

I guess that I could play around with ReStream to see if I can get a strobing video to play back ok. Are the ReStream parameters what I should expect or should I be seeing something different?

um3k

26th January 2010, 21:51

Hank, HCEnc is amazing. Thank you for all your hard work.

The other day, I was wondering if it would be possible to add a tab, similar to the Preview/Zones tab, to select different sections of the video to be encoded as progressive or interlaced. This would be great for encoding files from mixed sources, and even better if 24p (soft pulldown) could somehow be incorporated, effectively allowing for variable frame rate MPEG2. I don't know if enough people would use it to make it worth adding, though.

Thanks again.

manolito

26th January 2010, 23:35

@Emulgator
All you are saying makes perfect sense, but it is all based on theory and assumptions. Would you be so kind to make some side by side tests using HC, once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way, but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Cheers
manolito

KCE

27th January 2010, 08:40

Hi. There's a problem with one of the parameters. Passing -filesize input at the command line results in a very low calculated average bitrate used. For example at the command line, I pass an avs file, an ini file (has bitrate setting) and then a single -filesize argument which has a reasonable number such as 4395283 kilobytes for 1 hr 36 min 23.976 movie, which HC should calculate to around ~6000 kbs but instead HC calculates ~250 kbs when it starts encoding.

Also could I get some more information on how you calculate the file length/average bitrate in the gui window? I'm trying to automate this for a large number of files, which is why I'm passing filesize input. The formula I use is to just subtract the audio file size from the total disc capacity [no menu/subs/etc. and includes author overhead] to get the resulting space left entirely for video. I then input that filesize into the HC GUI and get the average bitrate. Actually if I save the ini file and reload it, then the filesize shrinks by a few megabytes which I'm guessing is HC approximating to the nearest possible filesize it can accomidate. I have a feeling I'm missing something really simple in my calculations...

I also use this formula (http://neuron2.net/LVG/ratesandsizes.html) or videohelp calc to get the average bitrate but the average bitrate I calculate tends to be lower by ~50 than the one produced by HC when I manually input the filesize in the gui window. However since the filesize argument doesn't work, it ends up using my calculated bitrate in the ini file. Then I'm left with about 25 MB of extra space on the disc which could go towards the video (every little "bit" counts :p)

The second problem is lossless file option does not work with autogop of 12. I forgot some of the other options I tried at the moment...but it would just crash a second after I start encoding.

Emulgator

27th January 2010, 12:17

manolito:
Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

Yes, was in my mind too.

Emulgator

27th January 2010, 12:26

(using HC autodetect)
txporter:
ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.
Are the ReStream parameters what I should expect or should I be seeing something different?
Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

midnightsun

27th January 2010, 16:28

txporter

27th January 2010, 17:21

(using HC autodetect)

Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

Just as a follow-up, I went ahead and created two clips of 3:2 pulldown material. Both were sent through TFM.TDecimate. One was encoded using autodetect and the other being forced as progressive.

1) Auto clip - experienced strobing on Tivo
2) Prog clip - smooth playback
3) Auto + ReStream set to BFF - jerky forward/back motion indicative of setting the wrong field order
4) Auto + ReStream set to Progressive sequence - smooth playback
5) Auto + ReStream set to BFF and Progressive sequence - smooth playback

I guess it is just how the tivo handles MPEG2 video. It's easy enough to fix by just forcing progressive encoding when I need it.

KCE

28th January 2010, 02:49

KCE, if you really want to maximize video size and you say you fall X MB short, add [(X*8*1000)/(running time in seconds)]kbps to the pre-calculated bitrate.
Be aware that for 25MB and 1:30:00 worth of footage, the bitrate goes up by just 37kb/s.

Thanks. Actually I thought it was the same for each file but it's not though I can live with a *small amount of space left over. Anyhow, the values in HC don't seem to follow the standard formulas. Maybe hank can give some insight?

midnightsun

28th January 2010, 09:29

It has to be the other way around actually, i.e. the formulae are not 100% correct in calculating DVD overhead (another explanation could be that they're tuned to leave you a little bit of "safe area", 20-25MB you're finding out to be unused, so you don't end up with a slightly oversized DVD).
If that's of any help and my memory serves me right, I empirically noticed that DVD overhead is about 1.8% of the combined size of video+audio streams for a typical 1:30:00-2:00:00 DVD (give or take 0.1%). If you use 2.2% you should *always* be alright:

[(4,700,000*8*0.978)/(running time in seconds)]-[total bitrate of audio/subtitle streams] kb/s

(4,700,000 is the size of a DVD-R in kbytes as you know, DVD+R is slightly larger at 4,706,000)

For 1:40:00 of video+(192kb/s)audio you get 5937kb/s, while videohelp calculator yields 5903kb/s.

If you feel like it, try it out on a project of yours to see if you come closer to full capacity while not going oversize.

KCE

29th January 2010, 23:44

Actually I've seen a suggestion of 1.1% of Length(sec) which seems pretty good to me...don't have the link on me but it was somewhere on videohelp. I understand it's tricky calculating, predicting, and actually hitting a target filesize.

I just did 11 encodes of popular movies and two of them (Pirates of The Caribbean II/III) were oversized by ~10 MB (the final authored DVD). The others were undersized from 0.13 MB (Pirates I in fact) to 22 MB. All audio is AC-3 192. Compared to the other movies encoded, Pirates is a lot longer and thus more frames, and of course the average bitrate used was relatively lower. Here's something I noticed

MB (remaining)|Length(sec)
22.1 5340
16.6 6240
10.8 6720
8.5 7020
7.2 7200
4.3 7860
0.125 8580

Based on DVD capacity of 4596992 kilobytes (~DVD-5). Bitrate and MB are also correlated - high mb = high bitrate. Other than the obvious, I'm not sure what other conclusion to draw from this? There seems to be a cut-off point (in seconds?) at which the file becomes too big. At that same point is where my calculations aren't useful anymore - my target average bitrate is too high. The m2vs produced by HC enc are right under the targeted filesize so the problem lies with authoring overhead.

I calculated average bitrate inputted into HC with the videohelp formula:

(Size - (Audio x Length )) / Length = Video Bitrate

or simpler

(Size / Length) - Audio Bitrate = Video Bitrate

I didn't include overhead into the "Size", mainly because videohelp didn't either so I figured okay. Had I did, the Pirates movies would probably fit and the others would be a bit more undersized, but like I said I can live with that. At the same time the 22.1MB movie could have used an increase in average bitrate because of the free space. So I guess that's what I'm looking for - that bitrate offset - or a better formula. The thing is I need the file size that HC generates to know how much more to add/subtract.

Actually I'm writing a program to automate multiple files->DVD, which I would describe as a robust scripting-made-easy program, so I'm looking for more of a general formula to go by. If it works well, I'll release it here. I'm sure we could use another conversion program...:p

Okay I'm getting off-topic now. Back to HC...

mikenadia

1st February 2010, 17:07

Looks solved in Edit: For a specific DVD (Taxi2), HC024 27-11 beta (HC023 too) keeps computing but hangs (nb of frames encoded do not move, time left is blocked..) but on Windows Task manager, it seems the application is still working (workload moving from 40% to 99%). It happens to me only twice.
in a 100 000 frames files; Trim(1,60000) works fine (using HC with an avs file produced by DVD-RB) . With identical Trim, it seems (I checked for n<12) that it hangs for every n>=7 if I add to the DVD-RB avs file SelectRangeEvery(n,1) and does not hang for every n<=6. I thought it was related to GOP length but it does not look like (I tried a lot of settings in HC.ini to change the behaviour for n=6 or 7 but to no avail). Also, the encoding speed is ,from frame 1, much quicker when it will not hang. Even more :confused:
Also (7,1) hangs ; (7,2) does not and (8;2) does. So the issue seems to be (n1-n2)>=6 when using SelectRangeEvery(n1,n2). If I encode the DVD-RB indexed file with Trim only, reindex the output with DGindex 1.5.7 , everything is fine. I can use any SelectRangeEvery(n1,n2).

For other info, the file was preprocessed with VOBblanker ( I may have split some cells but not at this location) but I still do not understand why HC only hangs with some SelectRangeEvery (probably, SelectRangeEvery cannot return a valid frame and HC may loop) .
I was wondering if the version of DGdecode.dll could be the problem.
http://forum.doom9.org/showthread.php?p=1370451#post1370451

Thanks in advance.

Edit: It is probably the DGdecode.dll used by DVD-RB. Using DVD-Decrypter in IFO mode (no File Splitting) with the preprocessed file, HC is able to process the file with any SelectRangeEvery with the file indexed with DGindex 1.5.7. End of story.

Edit 2: The issue with SelectRangeEvery may for small values of n1-n2 be fixed with the 10 frame window of ChangeFPS as explained by IanB (way above my head).
http://forum.doom9.org/showthread.php?p=1370823#post1370823
It is possible that the MPEG2Source used by DVD-RB is not completely frame exact (and that my source file was not linear in one way) but HC indirectly was able to find that.
http://forum.doom9.org/showthread.php?p=1370788#post1370788
It may be an issue with 1-pass VBR because I suspect that HC expects a frame exact source.
Edit3: Surprise,surprise: Was able to get the "hang" on the unprocessed DVD (straight from the rip).The movie is 29.97 fps (for DGindex 1.5.7) .

Emulgator

14th February 2010, 18:51

I still owed an answer to manolito:
Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way,
but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Now that I got back to my old test source, I could make a encode comparison
between a straight version (2-Pass VBR 5000kbps, 800 frames), deadzones=off vs. deadzones=auto
and a GradfunDB regrained version (2-Pass VBR 5000kbps, the same 800 frames) deadzones=off vs. deadzones=auto.

I found the same as manolito did, in both cases, straight and GradfunDB:

No visible differences between deadzones off and auto under my TFT testing conditions,

So it looks as if qualitywise deadzones=auto is safe to use instead of deadzones=off.

Filesizes came out identical to the single byte, bitrate distribution identical as well,
so HC 0.24 beta 27-11-2009 must have picked the same quantization for auto as for deadzones (0,0).

File size identical... this smelt a bit suspicious to me.
Because some of the settings in 0.24 GUI are not taken over on encoding time,
(you have to save the .ini first, close HC and reopen to be safe),
I repeated the encodes, but results stayed the same.

hank315

17th February 2010, 00:47

New HCenc024beta (http://hank315.nl/files/HC024_beta_16-02-2010.zip)

- 1pass VBR bugs fixed
- asm optimizations, a bit more speed
- some cosmetics

Also there's a special version included which doesn't use CPU dispatching which runs 2-3 % faster on my Intel Q9450.
Any CPU which can do SSSE3 will run it, it might also speed up AMD systems, Intel compiler options: /O3 /Og /QaxT /QxT
(There are rumours CPU dispatching by Intel compilers doesn't do that well on AMD :) ).
I don't have an AMD system at home or at work so I can't test it.

hydra3333

17th February 2010, 01:01

Thanks ! What is CPU dispatching and will I miss out on anything by using it (I also have a q9450).

edit: oh. http://www.agner.org/optimize/blog/read.php?i=25

hydra3333

17th February 2010, 11:40

Hmm, thanks for trying.
*INFILE D:\DVD\test.avs
*OUTFILE D:\DVD\test.mpv
*LOGFILE D:\DVD\test-HC.log
*PROGRESSIVE
*AVSMEMORY 256
*SMP
*1PASS
*LOSSLESS
*DBPATH G:\TEMP
*LLPATH G:\TEMP
*PROFILE BEST
*ASPECT 16:9
*LASTIFRAME
*AUTOGOP 15
*CLOSEDGOPS
*PULLDOWN
*DC_PREC 10
*MATRIX FOX1
*AQ 0
*BIAS 75
*LUMGAIN 1
*BITRATE 9200
*MAXBITRATE 9400
*CQ_MAXBITRATE 2
"C:\software\HC\HC024beta\HCenc_024_QaxT.exe" -ini "D:\DVD\test-HC.ini"
Yielded a windows error window "MPEG2 encoder has encountered a problem and needs to close ..."

(edit: i'd put in the *1PASS as a new option because i'd just spotted it, so it's my fault) It seemed to spend a minute or so "Sampling", before crashing, which I've never seen before and could potentially be very long for a glacial script ( even on a q9450 :) ). Commenting out the *LOSSLESS made no difference to crashing. Now I have to undo updating about 10 scripts ...

edit: same script and .ini (with and without lossless) worked just fine in v023 with and without any "Sampling" message thing whatever that was.

edit2: took out the *1pass and the "Sampling" message thing went away and it appears to be working. Looks like I don't understand what *1PASS does as compared to *CQ_MAXBITRATE which it appeared to over-ride. Any hints on the difference ?

PS nice work, with HC in videoredo :)

Emulgator

17th February 2010, 14:30

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.

This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4

What do you think ?

Gromozeka

17th February 2010, 18:19

hank315
Thanks, may be you add yuv input in next release :)

hank315

17th February 2010, 18:50

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.

This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)
This is easy to implement but DGPulldown can do all this AFAIK.

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4
This can be done with progressive pulldown, repeating frames instead of mixing fields.
Progressive sequence = 1, TFF = 0
RFF = 0 play frame 1 times
RFF = 1 play frame 2 times

16 --> 25: 1:2:1:2:1:2:1:2:2:2:1:2:1:2:1:2
16 --> 30: 1:2:2:2:2:2:2:2:1:2:2:2:2:2:2:2
18 --> 30: 1:2:2:2:1:2:1:2:2:2:1:2:1:2:2:2:1:2

Don't know if HW/SW players actually support this.

Emulgator

17th February 2010, 19:38

Ah, thank you for pointing me to that!

manolito

17th February 2010, 20:43

A big THANKS for the new beta!

Already tested it with the problematic source from this post: http://forum.doom9.org/showthread.php?p=1365549#post1365549
and the artifacts are gone.

So whatever changes you applied to 1-pass VBR mode, they sure did the trick.

Thanks again
manolito

mikenadia

18th February 2010, 02:33

1-pass VRB with 30 000 frames (sample size is not an issue).
1 pass CQ=9 (usual for the Extras of a DVD) with ip_ratio=1 and ib_ratio=1.2 leads to a 1600 kbps bitrate.
Did one pass VBR with 1600kbps . Got 1601 kbps.
Did the same thing with 1530 kbps (the max bitrate for which I have the warning "Very high quantizer"). Got 1340 kbps (the initial Q jumps by more than 25 % between 1530 kbps (warning) and 1540kbps (no warning).
Also, it is very sensitive to "sampling".
10 031 frames (1899 kbps for a targeted 2000 kbps : no High quantizer warning: initial Q:10.22).If I add 1 frame ,10 032 frames. right on target, 1993 kbps :initial Q:9.73).It looks it is assuming that the initial Q of the sample is the right one and does not adjust to it if the encoding shows differently ( if a coin always goes to "tail", after N trials, you have to believe the coin is biaised and bet accordingly even if your initial assumption was that the coin was not biased).
And the issue is bitrate dependent. If I increase bitrate to 4000 kbps, both clips (10 031 and 10032 frames) give me undersized enodes (both around 3525 kbps).On the 30 000 frames, no problem at 2000 kbps and 4000 kbps.

Edit: I do not know how sampling is done but rb-opt is doing it with
SelectRange Every(299,15) for 5% sampling (15 could be replaced by Max GOP length). This might enable them to estimate also ip_ratio and ib_ratio and may be to have a more accurate initial Q (when encoding with 1-pass VBR, I can often see 25% variation in I-frame quantizer).

Edit 2: a pause and resume during sampling phase leads to negative and then above normal average fps.

txporter

19th February 2010, 18:42

Question about re-encoding 1080i MPEG2 captures: I am seeing low CPU utilization (~30-40%) when re-encoding 1080i captures to remove the pulldown and resize to 1280x720. Is this due to decode performance? I don't think I see this with 720p material, but I haven't looked really hard there because I was getting ~2X the encode speed (~11 fps for 1080i and 22-23 fps for 720p). This is on a Q6600, 4gb ram, Vista64. My avs files are pretty simple. I am encoding at constant quant with the latest 0.24beta.

1080i
loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test.d2v",cpu=3)
TFM().TDecimate()
Undot()
Bicubicresize(1280, 720)
TextSub("test.srt")

720p
loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test2.d2v",cpu=3)
SelectEven().TDecimate()
Undot()
TextSub("test2.srt")

I have tried the 1080i with setmtmode 1 and 2. I see maybe 5-10% improvement in encode speed, but I am still hovering between 30-40% cpu utilization. What am I doing wrong? Where to look to increase the utilization and hopefully framerate?

Boulder

19th February 2010, 19:25

Add Distributor() as the last item in your script (if using SetMTMode())

txporter

19th February 2010, 20:22

Add Distributor() as the last item in your script (if using SetMTMode())

Ok, will try that. I totally missed that before! Do I need to call the MT plugin for that, or is that an internal filter for the MT avisynth.dll?

update: Ok, I confirmed on CoreDuo T2400 laptop that I was doing absolutely nothing with my setmtmode call without distributor(). ;) (Thanks, Boulder) CPU utilization on dual core was already fairly close to max'd, so might see more improvement on quad core at home. Also, setmtmode(1) crashes with my script.

Dual core numbers:
1080i-->1280x720 no setmt - 5.1 fps (~70% cpu util.)
1080i-->1280x720 setmt2 - 5.4 fps (~70-75% cpu util.)
1080i-->no resize no setmt - 4.5 fps (~90% cpu util.)
1080i-->no resize setmt2 - 4.8 fps (~90-95% cpu util.)

Also, am I right that I should resize after IVTC and other filters? Or should I resize before all of that?

Boulder

19th February 2010, 21:19

It's Avisynth's internal functions so no need for mt.dll.

hank315

19th February 2010, 22:20

@mikenadia
To sample a (15 frame) GOP, 3 frames are encoded: one I, one P and one B-frame, the total GOP size is then calculated:
GOPsize=Isize+4*Psize+10*Bsize
The amount of sampling depends on the movie size but it is small, for a 10000 frame clip, 100 frames are actually encoded.
Sampling is done for Qvalues 2.0, 5.0 and 10.0, after that a second order polynomial is fitted on this data to determine the final Qvalue which will (theoretically) hit the desired bitrate.
With this Qvalue a compression curve is created (stage 2), this curve is smoothed and scaled.

So it's important the Qvalue lies between 2.0 and 10.0 otherwise the value will be extrapolated from the fitted data.

Edit 2: a pause and resume during sampling phase leads to negative and then above normal average fps.
Cool, people always try things I didnt test :)
Think I will simply gray out the pause button during sampling.