HCenc 023 released

[KCE]

Hi. There's a problem with one of the parameters. Passing -filesize input at the command line results in a very low calculated average bitrate used. For example at the command line, I pass an avs file, an ini file (has bitrate setting) and then a single -filesize argument which has a reasonable number such as 4395283 kilobytes for 1 hr 36 min 23.976 movie, which HC should calculate to around ~6000 kbs but instead HC calculates ~250 kbs when it starts encoding.

Also could I get some more information on how you calculate the file length/average bitrate in the gui window? I'm trying to automate this for a large number of files, which is why I'm passing filesize input. The formula I use is to just subtract the audio file size from the total disc capacity [no menu/subs/etc. and includes author overhead] to get the resulting space left entirely for video. I then input that filesize into the HC GUI and get the average bitrate. Actually if I save the ini file and reload it, then the filesize shrinks by a few megabytes which I'm guessing is HC approximating to the nearest possible filesize it can accomidate. I have a feeling I'm missing something really simple in my calculations...

I also use this formula or videohelp calc to get the average bitrate but the average bitrate I calculate tends to be lower by ~50 than the one produced by HC when I manually input the filesize in the gui window. However since the filesize argument doesn't work, it ends up using my calculated bitrate in the ini file. Then I'm left with about 25 MB of extra space on the disc which could go towards the video (every little "bit" counts )

The second problem is lossless file option does not work with autogop of 12. I forgot some of the other options I tried at the moment...but it would just crash a second after I start encoding.

[Emulgator]

manolito:
Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

Yes, was in my mind too.

[Emulgator]

(using HC autodetect)

txporter:
ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.
Are the ReStream parameters what I should expect or should I be seeing something different?

Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

[midnightsun]

KCE, if you really want to maximize video size and you say you fall X MB short, add [(X*8*1000)/(running time in seconds)]kbps to the pre-calculated bitrate.
Be aware that for 25MB and 1:30:00 worth of footage, the bitrate goes up by just 37kb/s.

[txporter]

Originally Posted by Emulgator

(using HC autodetect)

Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

Just as a follow-up, I went ahead and created two clips of 3:2 pulldown material. Both were sent through TFM.TDecimate. One was encoded using autodetect and the other being forced as progressive.

1) Auto clip - experienced strobing on Tivo
2) Prog clip - smooth playback
3) Auto + ReStream set to BFF - jerky forward/back motion indicative of setting the wrong field order
4) Auto + ReStream set to Progressive sequence - smooth playback
5) Auto + ReStream set to BFF and Progressive sequence - smooth playback

I guess it is just how the tivo handles MPEG2 video. It's easy enough to fix by just forcing progressive encoding when I need it.

[KCE]

Originally Posted by midnightsun

KCE, if you really want to maximize video size and you say you fall X MB short, add [(X*8*1000)/(running time in seconds)]kbps to the pre-calculated bitrate.
Be aware that for 25MB and 1:30:00 worth of footage, the bitrate goes up by just 37kb/s.

Thanks. Actually I thought it was the same for each file but it's not though I can live with a *small amount of space left over. Anyhow, the values in HC don't seem to follow the standard formulas. Maybe hank can give some insight?

[midnightsun]

It has to be the other way around actually, i.e. the formulae are not 100% correct in calculating DVD overhead (another explanation could be that they're tuned to leave you a little bit of "safe area", 20-25MB you're finding out to be unused, so you don't end up with a slightly oversized DVD).
If that's of any help and my memory serves me right, I empirically noticed that DVD overhead is about 1.8% of the combined size of video+audio streams for a typical 1:30:00-2:00:00 DVD (give or take 0.1%). If you use 2.2% you should *always* be alright:

[(4,700,000*8*0.978)/(running time in seconds)]-[total bitrate of audio/subtitle streams] kb/s

(4,700,000 is the size of a DVD-R in kbytes as you know, DVD+R is slightly larger at 4,706,000)

For 1:40:00 of video+(192kb/s)audio you get 5937kb/s, while videohelp calculator yields 5903kb/s.

If you feel like it, try it out on a project of yours to see if you come closer to full capacity while not going oversize.

[KCE]

Actually I've seen a suggestion of 1.1% of Length(sec) which seems pretty good to me...don't have the link on me but it was somewhere on videohelp. I understand it's tricky calculating, predicting, and actually hitting a target filesize.

I just did 11 encodes of popular movies and two of them (Pirates of The Caribbean II/III) were oversized by ~10 MB (the final authored DVD). The others were undersized from 0.13 MB (Pirates I in fact) to 22 MB. All audio is AC-3 192. Compared to the other movies encoded, Pirates is a lot longer and thus more frames, and of course the average bitrate used was relatively lower. Here's something I noticed

MB (remaining)|Length(sec)
22.1	5340
16.6	6240
10.8	6720
8.5	7020
7.2	7200
4.3	7860
0.125	8580

Based on DVD capacity of 4596992 kilobytes (~DVD-5). Bitrate and MB are also correlated - high mb = high bitrate. Other than the obvious, I'm not sure what other conclusion to draw from this? There seems to be a cut-off point (in seconds?) at which the file becomes too big. At that same point is where my calculations aren't useful anymore - my target average bitrate is too high. The m2vs produced by HC enc are right under the targeted filesize so the problem lies with authoring overhead.

I calculated average bitrate inputted into HC with the videohelp formula:

(Size - (Audio x Length )) / Length = Video Bitrate

or simpler

(Size / Length) - Audio Bitrate = Video Bitrate

I didn't include overhead into the "Size", mainly because videohelp didn't either so I figured okay. Had I did, the Pirates movies would probably fit and the others would be a bit more undersized, but like I said I can live with that. At the same time the 22.1MB movie could have used an increase in average bitrate because of the free space. So I guess that's what I'm looking for - that bitrate offset - or a better formula. The thing is I need the file size that HC generates to know how much more to add/subtract.

Actually I'm writing a program to automate multiple files->DVD, which I would describe as a robust scripting-made-easy program, so I'm looking for more of a general formula to go by. If it works well, I'll release it here. I'm sure we could use another conversion program...

Okay I'm getting off-topic now. Back to HC...

[mikenadia]

Looks solved in Edit: For a specific DVD (Taxi2), HC024 27-11 beta (HC023 too) keeps computing but hangs (nb of frames encoded do not move, time left is blocked..) but on Windows Task manager, it seems the application is still working (workload moving from 40% to 99%). It happens to me only twice.
in a 100 000 frames files; Trim(1,60000) works fine (using HC with an avs file produced by DVD-RB) . With identical Trim, it seems (I checked for n<12) that it hangs for every n>=7 if I add to the DVD-RB avs file SelectRangeEvery(n,1) and does not hang for every n<=6. I thought it was related to GOP length but it does not look like (I tried a lot of settings in HC.ini to change the behaviour for n=6 or 7 but to no avail). Also, the encoding speed is ,from frame 1, much quicker when it will not hang. Even more
Also (7,1) hangs ; (7,2) does not and (8;2) does. So the issue seems to be (n1-n2)>=6 when using SelectRangeEvery(n1,n2). If I encode the DVD-RB indexed file with Trim only, reindex the output with DGindex 1.5.7 , everything is fine. I can use any SelectRangeEvery(n1,n2).

For other info, the file was preprocessed with VOBblanker ( I may have split some cells but not at this location) but I still do not understand why HC only hangs with some SelectRangeEvery (probably, SelectRangeEvery cannot return a valid frame and HC may loop) .
I was wondering if the version of DGdecode.dll could be the problem.
http://forum.doom9.org/showthread.ph...51#post1370451

Thanks in advance.

Edit: It is probably the DGdecode.dll used by DVD-RB. Using DVD-Decrypter in IFO mode (no File Splitting) with the preprocessed file, HC is able to process the file with any SelectRangeEvery with the file indexed with DGindex 1.5.7. End of story.

Edit 2: The issue with SelectRangeEvery may for small values of n1-n2 be fixed with the 10 frame window of ChangeFPS as explained by IanB (way above my head).
http://forum.doom9.org/showthread.ph...23#post1370823
It is possible that the MPEG2Source used by DVD-RB is not completely frame exact (and that my source file was not linear in one way) but HC indirectly was able to find that.
http://forum.doom9.org/showthread.ph...88#post1370788
It may be an issue with 1-pass VBR because I suspect that HC expects a frame exact source.
Edit3: Surprise,surprise: Was able to get the "hang" on the unprocessed DVD (straight from the rip).The movie is 29.97 fps (for DGindex 1.5.7) .

[Emulgator]

I still owed an answer to manolito:

Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way,
but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Now that I got back to my old test source, I could make a encode comparison
between a straight version (2-Pass VBR 5000kbps, 800 frames), deadzones=off vs. deadzones=auto
and a GradfunDB regrained version (2-Pass VBR 5000kbps, the same 800 frames) deadzones=off vs. deadzones=auto.

I found the same as manolito did, in both cases, straight and GradfunDB:

No visible differences between deadzones off and auto under my TFT testing conditions,

So it looks as if qualitywise deadzones=auto is safe to use instead of deadzones=off.

Filesizes came out identical to the single byte, bitrate distribution identical as well,
so HC 0.24 beta 27-11-2009 must have picked the same quantization for auto as for deadzones (0,0).

File size identical... this smelt a bit suspicious to me.
Because some of the settings in 0.24 GUI are not taken over on encoding time,
(you have to save the .ini first, close HC and reopen to be safe),
I repeated the encodes, but results stayed the same.

[hank315]

New HCenc024beta

- 1pass VBR bugs fixed
- asm optimizations, a bit more speed
- some cosmetics

Also there's a special version included which doesn't use CPU dispatching which runs 2-3 % faster on my Intel Q9450.
Any CPU which can do SSSE3 will run it, it might also speed up AMD systems, Intel compiler options: /O3 /Og /QaxT /QxT
(There are rumours CPU dispatching by Intel compilers doesn't do that well on AMD ).
I don't have an AMD system at home or at work so I can't test it.

[hydra3333]

Thanks ! What is CPU dispatching and will I miss out on anything by using it (I also have a q9450).

edit: oh. http://www.agner.org/optimize/blog/read.php?i=25

[hydra3333]

Hmm, thanks for trying.

*INFILE D:\DVD\test.avs 
*OUTFILE D:\DVD\test.mpv 
*LOGFILE D:\DVD\test-HC.log 
*PROGRESSIVE 
*AVSMEMORY 256 
*SMP 
*1PASS  
*LOSSLESS 
*DBPATH  G:\TEMP 
*LLPATH  G:\TEMP 
*PROFILE BEST 
*ASPECT 16:9 
*LASTIFRAME 
*AUTOGOP 15 
*CLOSEDGOPS 
*PULLDOWN 
*DC_PREC 10 
*MATRIX FOX1 
*AQ 0 
*BIAS 75 
*LUMGAIN 1 
*BITRATE 9200 
*MAXBITRATE 9400 
*CQ_MAXBITRATE 2

"C:\software\HC\HC024beta\HCenc_024_QaxT.exe" -ini "D:\DVD\test-HC.ini"

Yielded a windows error window "MPEG2 encoder has encountered a problem and needs to close ..."

(edit: i'd put in the *1PASS as a new option because i'd just spotted it, so it's my fault) It seemed to spend a minute or so "Sampling", before crashing, which I've never seen before and could potentially be very long for a glacial script ( even on a q9450 ). Commenting out the *LOSSLESS made no difference to crashing. Now I have to undo updating about 10 scripts ...

edit: same script and .ini (with and without lossless) worked just fine in v023 with and without any "Sampling" message thing whatever that was.

edit2: took out the *1pass and the "Sampling" message thing went away and it appears to be working. Looks like I don't understand what *1PASS does as compared to *CQ_MAXBITRATE which it appeared to over-ride. Any hints on the difference ?

PS nice work, with HC in videoredo

[Emulgator]

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.


This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4

What do you think ?

[Gromozeka]

hank315
Thanks, may be you add yuv input in next release

[hank315]

Originally Posted by Emulgator

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.


This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)

This is easy to implement but DGPulldown can do all this AFAIK.

Originally Posted by Emulgator

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4

This can be done with progressive pulldown, repeating frames instead of mixing fields.
Progressive sequence = 1, TFF = 0
RFF = 0 play frame 1 times
RFF = 1 play frame 2 times

16 --> 25: 1:2:1:2:1:2:1:2:2:2:1:2:1:2:1:2
16 --> 30: 1:2:2:2:2:2:2:2:1:2:2:2:2:2:2:2
18 --> 30: 1:2:2:2:1:2:1:2:2:2:1:2:1:2:2:2:1:2

Don't know if HW/SW players actually support this.

[Emulgator]

Ah, thank you for pointing me to that!

[manolito]

A big THANKS for the new beta!

Already tested it with the problematic source from this post: http://forum.doom9.org/showthread.ph...49#post1365549
and the artifacts are gone.

So whatever changes you applied to 1-pass VBR mode, they sure did the trick.

Thanks again
manolito

[mikenadia]

1-pass VRB with 30 000 frames (sample size is not an issue).
1 pass CQ=9 (usual for the Extras of a DVD) with ip_ratio=1 and ib_ratio=1.2 leads to a 1600 kbps bitrate.
Did one pass VBR with 1600kbps . Got 1601 kbps.
Did the same thing with 1530 kbps (the max bitrate for which I have the warning "Very high quantizer"). Got 1340 kbps (the initial Q jumps by more than 25 % between 1530 kbps (warning) and 1540kbps (no warning).
Also, it is very sensitive to "sampling".
10 031 frames (1899 kbps for a targeted 2000 kbps : no High quantizer warning: initial Q:10.22).If I add 1 frame ,10 032 frames. right on target, 1993 kbps :initial Q:9.73).It looks it is assuming that the initial Q of the sample is the right one and does not adjust to it if the encoding shows differently ( if a coin always goes to "tail", after N trials, you have to believe the coin is biaised and bet accordingly even if your initial assumption was that the coin was not biased).
And the issue is bitrate dependent. If I increase bitrate to 4000 kbps, both clips (10 031 and 10032 frames) give me undersized enodes (both around 3525 kbps).On the 30 000 frames, no problem at 2000 kbps and 4000 kbps.

Edit: I do not know how sampling is done but rb-opt is doing it with
SelectRange Every(299,15) for 5% sampling (15 could be replaced by Max GOP length). This might enable them to estimate also ip_ratio and ib_ratio and may be to have a more accurate initial Q (when encoding with 1-pass VBR, I can often see 25% variation in I-frame quantizer).

Edit 2: a pause and resume during sampling phase leads to negative and then above normal average fps.

[txporter]

Question about re-encoding 1080i MPEG2 captures: I am seeing low CPU utilization (~30-40%) when re-encoding 1080i captures to remove the pulldown and resize to 1280x720. Is this due to decode performance? I don't think I see this with 720p material, but I haven't looked really hard there because I was getting ~2X the encode speed (~11 fps for 1080i and 22-23 fps for 720p). This is on a Q6600, 4gb ram, Vista64. My avs files are pretty simple. I am encoding at constant quant with the latest 0.24beta.

1080i

loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test.d2v",cpu=3)
TFM().TDecimate()
Undot()
Bicubicresize(1280, 720)
TextSub("test.srt")

720p

loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test2.d2v",cpu=3)
SelectEven().TDecimate()
Undot()
TextSub("test2.srt")

I have tried the 1080i with setmtmode 1 and 2. I see maybe 5-10% improvement in encode speed, but I am still hovering between 30-40% cpu utilization. What am I doing wrong? Where to look to increase the utilization and hopefully framerate?