HCenc 023 released [Archive]

manolito

16th May 2008, 20:23

I did a couple of test encodes using either *AQ 3 or *LUMGAIN 3. The source was quite dark, average bitrate a little over 5000 kbps. The matrix I used comes from manono:
8, 8, 8, 9,11,13,14,17,
8, 8, 9,11,13,13,14,17,
8, 8,11,12,13,14,17,94,
9,11,13,13,14,17,17,94,
11,11,13,13,14,17,94,94,
13,13,14,16,17,20,94,94,
13,13,14,17,94,94,94,94,
13,14,17,94,94,94,94,94

12,12,13,14,15,16,22,26,
12,13,14,15,16,22,26,32,
13,14,15,16,22,26,32,41,
14,15,16,22,26,32,41,53,
15,16,22,26,32,41,53,94,
16,22,26,32,41,53,70,94,
22,26,32,41,53,70,94,94,
26,32,41,53,94,94,94,94

To make it short: I could not detect any differences between the two encodes. Maybe the bitrate was too high. AQ 3 did compress slightly better than LUMGAIN 3, but this probably depends on the source.

So more testing needed, I guess...

Cheers
manolito

Boulder

19th May 2008, 17:17

Try sources with flat areas versus sharper edges..the bits should be shifted from the edges towards the more flat areas of the frames. Also scenes with grass or similar texture are quite good.

blutach

21st May 2008, 08:19

As usual, I am late with my thanks Hank. :)

Regards

asarian

21st May 2008, 09:43

Outstanding piece of work, that HC encoder of yours!

:thanks:

phXql

26th May 2008, 06:59

Great piece of software. Thank you!

six13

30th May 2008, 05:27

Can someone please explain the 4 settings in AQ vs the 4 settings in LUMGAIN? I am trying to understand what to look for. I have previously used LUMGAIN 4 and liked the video. AQ has me puzzled, I saw the comments of Boulder but am still puzzled. I mainly compress/encode TV captures movies at a bitrate of 4K/S with small compression % like 82% with DVD-RB. I do notice that the detail on a face isn't real good it looks kinda flat. I am learning and would welcome opinions. Thanks in advance.

Dark Shikari

30th May 2008, 05:33

I do notice that the detail on a face isn't real good it looks kinda flat.Unless I'm mistaken, HC uses luminance-masking adaptive quantization, which won't help in such a case (one needs complexity masking like x264 has).

Boulder

30th May 2008, 06:09

AQ adapts the quantization, LUMGAIN affects the quantization matrix. hank posted the code related to LUMGAIN in the HC Encoder thread which shows how it works. Basically LUMGAIN lowers the coefficients of the base quantization matrix depending on the luminance levels. The strength affects the factor used in determining the new coefficients.

If I'm not entirely mistaken, AQ is meant for shifting bits inside one frame - from sharp edges to the more flat areas. LUMGAIN is supposed to shift bits to low-luminance frames from the brighter ones to prevent blocking in dark frames.

six13

31st May 2008, 01:26

So Boulder, then is it OK to use them at the same time or is it best to pick 1 or the other? In the past you have sugested I use LUMGAIN 4 and i like the output but was wondering if AQ would produce better results then LUMGAIN. What I recently encoded didn't have many dark sections. What do you use to make the decision as to use LUMGAIN or AQ?

Boulder

31st May 2008, 17:52

I use 3 for both AQ and LUMGAIN. They are two different things so they can be used together.

hank315

31st May 2008, 23:42

AQ is meant for shifting bits inside one frame - from sharp edges to the more flat areas. LUMGAIN is supposed to shift bits to low-luminance frames from the brighter ones to prevent blocking in dark frames.
Yes, that's what it does. The final quant for each MacroBlock: Q(i,j) = QMB * Qmatrix(i,j).
QMB is adjusted by the variance of the MacroBlock --> AQ.
Qmatrix(i,j) is adjusted by the average luminance of the frames in a GOP --> LUMGAIN.

Unless I'm mistaken, HC uses luminance-masking adaptive quantization, which won't help in such a case (one needs complexity masking like x264 has).
HC doesn't use luminance-masking adaptive quantization, the quantization is adapted by the variance of the MB so it's some kind of a complexity mask.
This picture (http://i25.tinypic.com/2j67594.png) shows the Quants of the same frame for different AQ values.

Dark Shikari

31st May 2008, 23:50

HC doesn't use luminance-masking adaptive quantization, the quantization is adapted by the variance of the MB so it's some kind of a complexity mask.
This picture (http://i25.tinypic.com/2j67594.png) shows the Quants of the same frame for different AQ values.Oh, so its like VAQ, complexity masking. Good to know that you've implemented a useful AQ method :)

six13

1st June 2008, 04:01

Thanks Boulder, which matrix do you use with those settings in HCenc?

Boulder

1st June 2008, 08:18

The matrix depends entirely on the source. Use the search, there are quite a few threads about this issue.

zeropc

7th June 2008, 14:33

hank, first let me say this is a very nice tool. i just started using it and the results are so far pretty good. but there are some open question for me in fully understanding the usage and some minor issues. i hope you or somebody else can help with this.

1. i currently set aq to 0 and the luminance gain to 0 , is this good or should i raise it? i encode from br/hd-dvd sources at 18mbps
2. i experienced that hcenc seams to lower the colors by a bit. i re-encoded the simpsons movie and saw how the skin colors faded a bit, which kinda bugs me
3. is there a way to scale hd sources to dvd res (pal and ntsc)?
4. is it possible to import .dga files from dgavcindex besides using avisynth?

gizzin

7th June 2008, 22:50

1. I say leave it on, especially at the bitrate you are using.
2. I never heard of that before, but maybe, how/what program are you using, what kind of file (DVD,XVID), essentially what I'm saying is you need to provide more information.
3. Yes
4. Not sure. I wouldn't think so.

Hank, I get this mismatch error with v23. "Error, large source mismatch found in pass 2 starting at frame:0, count :1 frames." I'm using FAVC 1.06. Just thought I'd point that out. It seems to be only this program that experiences this problem, as I use DVD Rebuilder and it never happened.

Graigddu

7th June 2008, 23:35

gizzin i remember that i came across this error before or something similar and after a bit of searching found that ticking reload avisynth during second pass under the settings 3 tab in the HCGUI fixed the problem, remember to save HC Ini in main tab or any settings changes won't take effect

Revgen

8th June 2008, 00:34

Quality is good as long as I keep AQ off. Not much different than HC022.

BDrift

8th June 2008, 03:26

Zones don't seem to work for me :(

*ZONE 3
1 0.2
1200 1.0
163500 0.2

Thats what I use in hc.ini, but when I compare it with a encode without zones, there is close to zero difference visible, not even in a frame by frame comparison. Am I missing something? Shouldn't frames 1-1200 and frames 163500-end have a ~80% lower average bitrate?

For testing I've set end frame to 2000 ... could that be causing trouble?

lithoc

13th June 2008, 18:28

I've found a bug that cause Ifoedit to crash when muxing.

I encoded some 352x240 video. I found out that Ifoedit will crashed at the beginning if the video bitrate is below ~256kbps.

I notice video having a black fade in. When the black is usually low bitrate that's where the Ifoedit crashed.

If I encode the video starting frames of 100 where there's no black screen. Ifoedit mux just fine when the bitrate is > 256kbps.

Can HCenc have minimum bitrate settings?

totya

15th June 2008, 14:56

Thank you (developer) for this great application!

magic75

2nd July 2008, 20:07

Rogi

2nd July 2008, 21:41

I am having problems getting the aspect ratio choice to "stick". With 0.22 I can choose 4:3, hit save hc.ini, and the next time I start up hcenc the aspect ratio is set to 4:3. With 0.23 it seems to "reset" to 16:9 no matter what.

I don't know what is the problem but on my computer HC 0.23 works well and remember my aspect ratio choice.

krosswindz

4th July 2008, 17:24

I have been using HC enc for sometime. I am trying to use HC encoder inside a VMWare. I tried HC 022 and 023 both crash immediately opening the send report dialog. I would really like to be able to run HC encoder in the VMWare. Any help in solving this would be appreciated.

krosswindz

4th July 2008, 18:10

I looked further into this problem seems like HC Enc and MT_Masktools-26.dll dont like each other. Is this an known issue or something, I am seeing for the first time. The vesion of MaskTools is 2.0.35.0

rendez2k

10th July 2008, 18:35

For the first time, I noticed an error with HCEnc yesterday, ERROR, source mismatch in pass 2 starting at frame: 0 - never seen this before but I'm not sure if its a HCEnc 023 issue or FAVC? What does the error actually mean?

Rumbah

14th July 2008, 14:23

It's probably your Direct Show decoder that does not support frame accurate seeking as the error message means that the first frame in the first pass is different form the first frame in the second pass.

You could try to fix it by enabling the reload Avisynth switch in HCEnc. It was made for these kinds of errors.

Boulder

14th July 2008, 14:30

You could try to fix it by enabling the reload Avisynth switch in HCEnc. It was made for these kinds of errors.You could also try enabling the lossless file option if you have the extra HD space it requires.

rendez2k

14th July 2008, 14:53

Rumbah

15th July 2008, 12:22

If you use reload Avisynth and you don't get an error everything should be fine.

rendez2k

15th July 2008, 12:30

If you use reload Avisynth and you don't get an error everything should be fine.

The trouble is, in some apps that use HCEnc, you can't edit HC.ini to force reload. I'm wondering if using HC 0.22 would help?

Boulder

15th July 2008, 14:15

Search for HCWrap, it'll allow you to enter custom ini commands.

halsboss

16th July 2008, 13:00

Error "can't open dbs file, invalid path?" before 1st pass starts, from

*INFILE G:\DVD\testi\test1080iHD-CQ2.5.avs
*OUTFILE G:\DVD\testi\test1080iHD.mpv
*LOGFILE G:\DVD\testi\test1080iHD-HC-CQ2.5.log
*INTERLACED
*TFF
*PROFILE BEST
*ASPECT 16:9
*BIAS 50
*BITRATE 9300
*MAXBITRATE 9400
*LASTIFRAME
*AUTOGOP 15
*PULLDOWN
*CLOSEDGOPS
*DC_PREC 10
*MATRIX FOX3
*LOSSLESS
*DBPATH G:\DVD\testi
*LLPATH G:\DVD\testi

Any idea what I'm doing wrong ?

Darksoul71

16th July 2008, 13:47

Not really..just two questions:
1) Does G:\DVD\testi exist ?
2) Any special reason why you specify the DB path ?

halsboss

16th July 2008, 15:01

Thanks for looking; yes it's the same folder as the source, no reason other than I'm not sure what it defaults to and wanted to easily find and get rid of the files ...

edit removed the path lines and it works ... now where did those files get put ?

Darksoul71

16th July 2008, 19:05

My guess for the lossless file:
It goes to the same directory as the DB -> the directory where HCEnc resides when you call it.

msdossys

17th July 2008, 08:08

hank315, first let me say thank you for this very excellent program. Now, that being said, I have one feature request that seems to be in the GUI but not the encoder. When the output of the encode is set to a file that already exists, the GUI will ask you if you want to overwrite it. However in my haste after editing the source in my AVS, I double-clicked on the encoder instead of the GUI, and it just started encoding along on its merry way, trashing my 1hr encode that was just completed. Sooooo, is this a possibility to put in to save stupid people like me?

Darksoul71

18th July 2008, 12:50

@hank:
Just two short suggestions/feature requests:
1) Please add an option to set the DB filename. This would help to use HCEnc^n with multiple instances of HCEnc without generating a separate dir for every HC instance.
Currently *DBPATH doesn´t really help since I would have to create a separate DB dir for each HCEnc instance. This makes no big difference to my prior approach to copy the HCEnc.exe to a separate directory.

So something like *DBFILE c:\mypath\my_database.db would be extremely nice.

2) Please add an option to output a DB file during CQ encoding. As far as I´m aware of a DB file is only written in 2 pass mode, right ?
Nevertheless it would be cool to run a CQ encoding and have the option to do a second pass (non-CQ) if the resulting file is way off.
This is pretty similar to CCE which provided an option to write the VAF file during any encoding (be it CQ, CBR or VBR).

TIA,
D$

Chumbo

20th July 2008, 17:47

I wanted to report an issue I found yesterday. I was doing a reencode using a .d2v file. I normally just run the command line and feed it an .ini file. When I did this, the command line brought up the encoder status dialog which then reported that it cannot find an mpeg encoder. However, if I bring up the GUI, load the .ini file and run the encode, it works just fine. I searched this thread for this problem but didn't find anything as I didn't want to report a known issue.

This is with version 023.

Nomolu

21st July 2008, 02:22

I am having problems getting the aspect ratio choice to "stick". With 0.22 I can choose 4:3, hit save hc.ini, and the next time I start up hcenc the aspect ratio is set to 4:3. With 0.23 it seems to "reset" to 16:9 no matter what.
I'm seeing the same issue. Other options, such as BITRATE and MAXBITRATE, are loaded from the HC.INI file when HCGUI is launched. But "*ASPECT 4:3" is ignored. The ASPECT option only seems to work if I first load an AVS file, then manually load the HC.INI file.

msdossys

22nd July 2008, 05:56

mikeytown2

23rd July 2008, 10:53

Kinda merging 2 ideas here...

You dont need HCEnc^n anymore... since 022 HCEnc has SMP built in :)
HCenc^n splits up the video and then processes it.

Over in the AviSynth Dev Thread...
How about that idea: split a video into X parts, where X is the number of cores, let the parts overlap by the amount of frames, used by e.g. MVTools, process all parts simultaneously and rejoin them, removing the overlap?...

I could see HCenc splitting the avs into n different files since temp lossless is an option in 2 pass mode. For the second pass, HCenc would reference the multiple lossless files. This would only be useful if the script is a major CPU eater, thus it will not be beneficial at all times. I'm sure there might be filters out there that have potential issues when used simultaneously (GPU). But with more CPU's having more cores, and until a 2.6RC is done, this could be a big boost for people with heavy scripts and lots of cores. Don't know if it's worth the time and effort though...

Gromozeka

23rd July 2008, 13:13

hank315
You may add trellis quantization and to increase length Gop up to 300 in HCencoder 0.24? :)
It is good for quality on low bitrate (and the autoring)

Darksoul71

24th July 2008, 20:41

Hi all,

@msdossys:
1st, I think it´s kinda unpolite to comment a feature request in this way.
No matter if the comment shows a smiley or not.

2nd, I agree that the MT implementation of HCenc is the best I´ve seen around compared to other freeware MPEG2 implementations (Mencoder, AQE) but I do not know any details in regard to approach Hank has choosen. In a lot of my tests HCEnc MT is as fast as HCEnc^n is. Sometimes HCEnc even beats HCEnc^n since HCEnc^n does the merging of the M2V segments and correction of the timecodes afterwards. Nevertheless I like HCEnc^n over HCEnc because I have the control over it. In HCenc^n I can e.g. limit HCEnc to two cores and leave to other cores free for AVISynth filtering. This could be a nice option for QuadCore owners.

All in all I do not see HCEnc^n and the MT implementation of HCenc as redundant. HCEnc MT is integrated in the encoder and more stable. HCEnc^n is external and allows the user more modifications. Esp. since I include the AutoIt sources with it.

@mikeytown2:
I really enjoy your idea. Of course this approach is limited to AVISynth filters which are not dependend to a single resource (GPU as you already named) but for complex scripts this could really help.
Just as my example above:
QuadCore CPU and only use 2 cores for HCEnc encoding and the other two CPUs free for AVS filtering.

Don't know if it's worth the time and effort though...
Hm, the effort is quite low. I simply had to hack in two more lines of AutoIt which add the *LOSSLESS and *LLPATH parameter.
I think I´ll make this controlable via a small "lossless" switch within the ini-file of HCEnc^n.

I´ll upload this version tomorrow after some testing but things are looking good.

@hank:
I agree with Gromozeka:
It would be nice to be able to set GOPs longer than 36. I used long gops on (k)SVCD and they always played back fine on my SAP.

Stay tuned,
D$

Darksoul71

24th July 2008, 20:55

Hi Hank,

could it be possible that HCEnc has a problem when parsing the ini-file and you´ve separated the command from the value by using a tab ?

I first added the *LLPATH and *DBPATH to the HCEnc.ini with a tab as separator in between.

Example
*LLPATH TAB C:\MyPath

This gave me the same problems as halsboss is describing here:
http://forum.doom9.org/showpost.php?p=1159595&postcount=53

HCEnc says ...error "can't open dbs file, invalid path?"

When I separate the command and the value by using blanks everything is working fine though.

Could you have a look at this and confirm it ?
:thanks:

I have no big problem to use blanks instead of tabs though :cool:

Later,
D$

Darksoul71

24th July 2008, 21:58

Yes, I know... it´s my third posting in this thread but merging them makes no sense to me since they are way to different.

I´ve (re)started to write a HCEnc prediction tool which acts as a wrapper (similar to HCWrap and HCEnc^n).
The idea is similar to CCEFront:
Make a simple commandline tool which does all the prediction and encoding itself. I´ll use the bitrate
specified and the movie length to calculate the target size. After this a simple prediction will take place (using
Inc´s old Ping Pong approach) and the final encoding will be done in CQ mode.

You could simply call the tool like HCEnc itself:
HCPred -ini "C:\my_movie.ini"
and the rest will be done automatically.

CQ encoding with HCEnc can often end up in mixed results. Sometimes a good 10 to 20% oversize are not that uncommon
(depending on your source).
CCEFront (as well as D2SRoba) had two nice features available for CCE:
1) Do a second pass in VBR mode if the CQ file is way too oversized
2) Shrink the resulting M2V with Rejig / Requant if the file is just a few percent too big.

I asked hank to output a DBS file even if you do only CQ mode. It would be even nicer if we could output a lossless file as well.

How it could work:
- Tool starts prediction
- HCEnc runs in CQ mode but outputs a lls file and a db file as well (specified by the tool)

If the resulting file overshoots the target size it could be either shrinked with Rejig / Requant (for up to 5% may be) or be transcoded in a "second pass" using the lossless file from the previous run as source.

What do you think ?

45tripp

28th July 2008, 18:20

bug:

from the log.

total CPU time: 0:07:07 (426.84 s)
total elapsed time: 0:04:12 (252.47 s)

total elapsed tme seems to be cpu time,
and cpu time is completely off.
hc 21 and 22 seemed fine.

request:

i'd like drag and drop for source input (of already valid inputs) into the gui.
and drag and drop wherever else it's valid, like ini input for example.

ty
tripp

florinandrei

30th August 2008, 21:09

HCenc 0.23 scales very well with the number of CPUs.
I'm transcoding a standard-def progressive MPEG2 source. I'm using the best profile. Previously, on a dual-core AMD (2600 MHz clock) I would get around 70 fps.
On my new Phenom quad-core (2500 MHz clock) I get around 140 fps. That's just awesome. :thanks:

halsboss

30th August 2008, 23:18

Fantastic product !!!

Couple of minor suggestions:

a) when trying to run 2 HC's with lossless set to the same path the 2nd crashes I guess since the filenames may be "fixed" ? Could the lossless files use say random names or the same as the .ini or output except with a different extension ?

b) instances all have the same name in task manager ... maybe the titles could reflect the file being processed ?

Just a thought. Thanks !!!

Adub

30th August 2008, 23:28

May I ask why you are running two instances of HC at the same time? This wonderful encoder was multithreaded ages ago, so there should be no reason to run to instances as the same time.

halsboss

30th August 2008, 23:47

Good question. 4 instances each with avisynth SetMTmode(mode=2,threads=2) and 256Mb setmemorymax. (q9450 4-core, 4gig memory, 750gb disk)

Consider the case of a PAL HDTV 1080i-> DVD 576i 4-quarter football match, with 4 per-quarter independent videos.

Each quarter edited independently, down-converted/HC'd independently, targetted to it's own DVD so as to use maximum DVD spec bitrate and hence highest possible quality, given it's 36 players in a very large grassed arena with very fast action.

Currently uses 2-pass profile=best 224k audio 9350k video fox1 matrix (happy to receive other suggestions as even then it doesn't max out the bitrate, hovering between 8800k and 9000k even though *AQ 0 *BIAS 70 *LUMGAIN 0 *BITRATE 9350 *MAXBITRATE 9350).

blutach

31st August 2008, 00:46

@halsboss,

Unless you're a Geelong supporter, don't worry :D :D :D

(Sorry for OT everybody).

Regards

halsboss

31st August 2008, 01:10

For some reason, parts of the grassed background seem to "blur/float" with the long-shot panning action ?? The source seems to have bits of it like that... would fox2 be slightly "gentler" with that effect do you think ?

@blutach :) Yes the Cats are a team of Big fast tough Men... Craigy's Crowbots are well drilled and never ever ever give up though :D Duracells, every lovely one of 'em (they keep on going).

Boulder

31st August 2008, 09:47

I'd say that MPEG2 at standard DVD bitrates is just not enough for such material :(

halsboss

7th September 2008, 03:05

Thanks, yes it seem I expect too much from the bitrate / resolution.

Oh well, my beloved footy team (the Crows, possibly to be renamed the Chooks) are now out of the finals, so I have about 6 months to fiddle and squeeze the most I can out of reasonable encoding times for PAL hdtv->dvd.

I did try *BIAS 100 and it still didn't max out the bitrate, under by 300k-700k, so I'm a bit unclear on what else to do. I'm using this resizing method http://forum.doom9.org/showthread.php?p=1177122#post1177122 rather than TDEINT&NNEDI deinterlacing/resizing/reinterlacing on the basis it looked the same or better visually and was faster ... open to suggestions on any front... since I have 6 months (it's a long long off season).

AlanHK

29th September 2008, 04:28

smok3

29th September 2008, 08:10

May I ask when to use the *LASTIFRAME option?
Dose any editing/authoring software require the last frame to be an I-frame under some circumstances?

i wonder that too.

pandy

29th September 2008, 10:20

i wonder that too.

easier editing? - especially when joining few ES together or create some looped video - this make possibility to clean cut/join of ES - especially when few ES have different parameters (i.e. resolution, color space, interlaced vs progressive etc etc)

For me this give possibility to make seamless TS for playing in loop, can be useful for IP broadcast from playlists

Chumbo

29th September 2008, 16:46

I was having the message :
ERROR, source mismatch in pass 2 starting at frame...
and found that setting "-avsreload" in the command line fixed this.

Is there any reason not to always use this, just as a precaution? There seems no noticeable delay.
Reloading AVS (available in the UI too btw) will not always fix this issue. I encountered this issue regardless of this setting, so make sure to check your log after an encode to be sure.

What is 100% is using the lossless option and it's faster. However, you'll need a LOT of available hard drive space depending on what you're encoding.

krosswindz

20th October 2008, 05:36

mikeytown2

20th October 2008, 06:47

What are you using to load your source? It needs to be frame accurate.

krosswindz

20th October 2008, 14:23

^ I am using HCEnc Gui to load the avs file.

Chumbo

20th October 2008, 16:48

I am also encountering the Error, source mismatch in pass 2 starting at frame... error. I have the option of reloading avisynth selected. I dont have a lot of space so dont have the option of running the lossless option. Any reason why this might be happening?
You should really consider getting an external drive just for this if you do a lot of encoding. It'll pay for itself in time spent doing this. Drives are inexpensive and all you probably need is a 250-300Gig drive. Good luck. :)

mikeytown2

20th October 2008, 19:29

^ I am using HCEnc Gui to load the avs file.

What are the files you are trying to load? AVI, FLV, ...? My guess is its a file that has a variable frame rate; which means you need to load it with the correct tools, not DirectShowSource().
http://avisynth.org/mediawiki/External_filters#Source_Filters

krosswindz

21st October 2008, 00:30

@chumbo yeah I have been thinking about getting an external drive. The problem being that I am running windows XP inside a VM in linux. I am not sure if I have to use fat32 on the external drive or if I can use ntfs.

@mikeytown2 I am trying to load DVD VOB files using DGDECODE_MPEG2source. I have pmed you the script I am using.

Darksoul71

21st October 2008, 08:19

@others: Sorry for OT

@krosswindz:
The problem being that I am running windows XP inside a VM in linux. I am not sure if I have to use fat32 on the external drive or if I can use ntfs.
Normally you should be able to use ntfs on the external drive. I never experienced any problems when accessing ntfs in read/write access from linux. Another approach might be using a linux FS on the external drive (ext3 for example) and publish it via Samba. From the Windows VMs perspective the underlying FS shouldn´t matter then.

Depending on your Virtualisation solution you might also be able to attach the USB drive to the VM directly. This is something which for example Sun´s VirtualBox supports. I haven´t been able to get this running on my Debian Host with a VBox WinXP Guest though.

Chumbo

27th December 2008, 17:04

halsboss

28th December 2008, 01:56

Have you put 2 or more spaces between the option and the path/file in the .ini file ? :)

Chumbo

28th December 2008, 04:27

Have you put 2 or more spaces between the option and the path/file in the .ini file ? :)
On this PC, I happen to be using the GUI. I don't have this issue on my other computer either via the UI or if I run it from the CLI. But either way, I always create the ini file via the GUI.

Boulder

28th December 2008, 10:41

@hank,
I've been trying to encode using the Lossless option on one of my computers but it has been crashing when checking this option. Seems to work fine with reload AVS, but I'd prefer to use the lossless file option.

The crash is just the standard application crash per the snapshot below. I tried all sorts of combinations of same and different paths and drives for the locations of both the lossless and db path, in case of some sort of clash, but nothing works. Please let me know if you want any other details. Thank you.
http://img360.imageshack.us/img360/7265/hclosslesscrashbe4.jpgI wonder if that crash creates a dump file which you can analyse. The next time that happens, you could leave the crash report window open and see whether your Windows temp folder (I think it's usually c:\documents and settings\username\local settings\temp) contains a .dmp file. Copy that file somewhere safe and upload it somewhere so we can take a look at it. You have to leave the crash report window open because the dump files will be removed after the window is closed. Another place to look for the dump files is at c:\windows\minidump.

Chumbo

28th December 2008, 17:57

I wonder if that crash creates a dump file which you can analyse. The next time that happens, you could leave the crash report window open and see whether your Windows temp folder (I think it's usually c:\documents and settings\username\local settings\temp) contains a .dmp file. Copy that file somewhere safe and upload it somewhere so we can take a look at it. You have to leave the crash report window open because the dump files will be removed after the window is closed. Another place to look for the dump files is at c:\windows\minidump.
I sent the crash info via PM to you Boulder.

btw, I happen to test this on an mpeg2 source and it works just fine. It only crashes when the source is an AVC. I normally use the .dga file as the source in the input avs script, but it crashes if I use either AVCSource or directshowsource as long as the source is AVC. Maybe that'll help narrow down things.

I tested this on my "working" system and sure enough it crashed there too. What I didn't remember is that when it worked, my sources were all mpeg2. Sigh...

Let me know if you need any other info.

moviemaven

1st January 2009, 08:18

I have had the same exact problem here also. If I remove *LOSSLESS, it works okay, but crashes with it in.

MM

Sharc

6th January 2009, 00:30

Guest

6th January 2009, 00:37

Works fine for me.

Chumbo

6th January 2009, 01:54

I have had the same exact problem here also. If I remove *LOSSLESS, it works okay, but crashes with it in.

MM
You can't just use this option by itself. You have to use it with a location. See the GUI for the option to set that. :)

I've moved my issue to the 024 beta thread, so it would be good to post any bugs you all have there.

moviemaven

6th January 2009, 04:30

You can't just use this option by itself. You have to use it with a location. See the GUI for the option to set that. :)

I've moved my issue to the 024 beta thread, so it would be good to post any bugs you all have there.

Actually, you do not need the location (LLPATH) so long as it is set in the program. I think LLPATH would override your default setting. Funny thing, I cannot reproduce the crash anymore, so it must have been only one kind of file.

Glenn

Chumbo

6th January 2009, 16:07

Actually, you do not need the location (LLPATH) so long as it is set in the program. I think LLPATH would override your default setting. Funny thing, I cannot reproduce the crash anymore, so it must have been only one kind of file.

Glenn
Oh yeah, I forgot. You're right. :) Is it working with your source being AVC?

Sharc

6th January 2009, 20:12

Works fine for me.
Ooops ... works perfectly when I
- Do the encoder settings in HCgui_023 and save the HCini
- Exit HCgui and fire up HCenc_023.exe separately
Looks as if the HCgui cannot start the encoder properly.
Scratching my head .... perhaps something is borked here ....

moviemaven

7th January 2009, 04:59

Oh yeah, I forgot. You're right. :) Is it working with your source being AVC?

Am I using AVCs?? I honestly don't know. I was under the impression that those are associated with h264 files. ALl my inputs have been AVIs.

~bT~

7th January 2009, 10:08

I created an index file .dga of an .m2ts (VC-1 encoded) and an .avs using DGVC1IndexNV. I loaded the .avs successfuly in HCgui 0.23, and I can preview the movie in the "Preview/Chapter" Tab of HCgui, but when pressing "encode" nothing happens ...
Is HCenc not compatible with .dgv index files?

exact same problem with same type of input file. hc 0.22 worked flawlessly.

pls let me know what info u need to rectify this if poss.

Sharc

7th January 2009, 22:00

@bt:
Not a solution, but perhaps you have seen my workaround in post #99.

~bT~

8th January 2009, 00:50

Sharc

13th January 2009, 21:22

@hank315:
When I convert a blu-ray clip (Film@23.976 fps) to PAL DVD format and press "make DVD compliant", HCgui inserts *pulldown in HC.ini, which results in a jerky playback.
This may be ok for NTSC, however for PAL I believe it should insert "*assumeFPS(25)" in order to do the PAL speedup.
(It works fine when I put assumeFPS(25) into the script.)

um3k

13th January 2009, 21:55

The speedup is your job, not the encoder's.

~bT~

29th January 2009, 15:56

Chumbo

14th February 2009, 15:12

...
btw, I happen to test this on an mpeg2 source and it works just fine. It only crashes when the source is an AVC. I normally use the .dga file as the source in the input avs script, but it crashes if I use either AVCSource or directshowsource as long as the source is AVC. Maybe that'll help narrow down things.

I tested this on my "working" system and sure enough it crashed there too. What I didn't remember is that when it worked, my sources were all mpeg2. Sigh...
@hank,
I guess the source being mpeg2 isn't really the issue here. I just ran into this yesterday and have narrowed it down to the script having the LanczosResize or Lanczos4Resize command. Without this it works fine with the lossless option, but once I add it, bang! HCenc crashes. I tried putting the resize before and after the convert... function with the same results.

Here's my script, it's very basic.LoadPlugin("c:\PROGRA~1\AutoGK\DGMPGDec\DGDecode.dll")

mpeg2source("H:\Media\movies\DVDs\SD\movie\movie.d2v")
LanczosResize(720,480)
converttoyv12()

davexnet

18th February 2009, 22:02

General HCenc questions thread?

I was wondering - are any of the built in matrix's suitable for animation ?

I tried the "custom matrix" and typed the numbers from Tmpgenc 2.5 animation/CG matrix, but the results were so-so.

TIA for any info.

halsboss

2nd March 2009, 12:55

Just on matrices, I tend to default to FOX1 as I gather it retains the most detail... is that right ? I admit to not understanding matrices.

If it goes over 1 DVD, I then use Nero Recode to get it back down. Is that generally an OK thing to do ?

Chumbo

2nd March 2009, 16:22

...If it goes over 1 DVD, I then use Nero Recode to get it back down. Is that generally an OK thing to do ?
Why don't you just use the target size and let HC estimate the bitrate for you? When you recode what you've already encoded, you'll lose more quality.

Depending on the audio track size, you can estimate the size with a bitrate calculator. I just multiply 3.8 x 1024 x 1024 and enter the rounded whole number into the file size. I always get exactly what I need. ;) Plus, you don't want to use every bit on the DVD as they tend to wear down on the outer edge first.

AlanHK

3rd March 2009, 01:57

If it goes over 1 DVD, I then use Nero Recode to get it back down. Is that generally an OK thing to do ?
If you use two-pass encoding, you can set the exact file size in HC.
(In HCgui, under "bitrate", switch from "average bitrate" to "file size". Then you can switch back to see the corresponding bitrate.)

If I'm encoding several files for one disc, I make an omnibus AVS file that joins them all together, use HCgui to calculate the bitrate to make that 4000000 kB, then use that rate for each individual encode

eg:

Import("09.avs")
c=last
Import("10.avs")
c=c++last
Import("11.avs")
c=c++last
Import("12.avs")
c=c++last
Import("13.avs")
c=c++last
c

~bT~

3rd March 2009, 02:19

If you use two-pass encoding, you can set the exact file size in HC.
(In HCgui, under "bitrate", switch from "average bitrate" to "file size". Then you can switch back to see the corresponding bitrate.)

does that take audio & subs into consideration? i doubt it.

qyot27

3rd March 2009, 02:30

does that take audio & subs into consideration? i doubt it.
Why would it? HCenc only encodes video.

However, you can fudge it by making the size equal to the final video+audio size (subs take up virtually nil most times, just subtract the size from what you want to DVD itself to be, and then use the remaining number as the video+audio size), get the combined bitrate, subtract what bitrate you plan to encode the audio to from the combined bitrate, and voila.

AlanHK

3rd March 2009, 02:36

does that take audio & subs into consideration? i doubt it.

4 GB for video
2-300 MB for audio (AC3 @ 128k)
subtitles negligible, 5-10 MB if I do them.
Menu, mux overhead about 20 MB (menu, no motion or sound).
Leaves usually 2-300 MB spare.

If you do high quality 5.1 sound or motion menus obviously you need to allow more space.

~bT~

3rd March 2009, 10:32

^ well, some of the replies to halsboss were misleading..

obviously it is an encoder which can only work out the video bitrate via size.

what halsboss really needs is a calc: http://www.videohelp.com/calc.htm

halsboss

3rd March 2009, 12:08

Ah, thankyou all. Wasn't aware of the filesize thing in HC , just wanted to max it out with the most detail as 1st choice. The calc will come in handy. Noone else uses FOX1 then ?

~bT~

3rd March 2009, 12:12

Noone else uses FOX1 then ?

avamat6 or avamat7 is all i've needed.

pdadi

19th March 2009, 20:33

qyot27

19th March 2009, 21:05

Great encoder . Thanks.

I have a NTSC DVD from PAL master. In my avs script I bring it back to 24.975fps (this looks best with all dupes gone).

After loading it in HC it is warning that it is not DVD complaint.

My plan is to do DGpulldown after HC encode is done to make it NTSC complaint.

Any solutions?
24.975 isn't a proper MPEG-2 framerate, or at the very least it's not a PAL compliant fps. PAL actually is 25fps, it's not a situation like 24fps and 30fps often being used as aliases to 23.976fps and 29.97fps in NTSC land.

Use AssumeFPS(25,sync_audio=true).SSRC(48000,fast=false)* to correct for the audio difference (I'm assuming that your script has the audio correctly synced at 24.975), and then encode the 720x480, 25fps stream with HCenc. Afterward, you can use DGPulldown to set the right flags, and use mplex from the MJPEGTools project to mux the video and audio together. The .BAT file would look like this (I actually have a profile I use for 25fps material):

hcenc_023 -i "input.avs" -o "output.m2v" -ini "16-9_25fps-6000.ini"
dgpulldown "output.m2v" -inplace -srcfps 25 -destfps 29.97
mplex -f3 -V -o "finaloutput.mpg" "output.m2v" "audiofile.ac3"
I use mplex because it works. I'm not too up on MPEG-2 muxers, but I seem to remember that it was the only one I tried that handled 25->29.97 pulldown flags correctly.

*Sometimes that will fail, and can be fixed by sticking another SSRC(48000,fast=false) command before the AssumeFPS call. And sometimes even that will fail, requiring the before-AssumeFPS SSRC samplerate to be raised slightly: 49000, 50000, 51000, or 52000 usually work - although going over that tends to result in access violations (at least on my computer it does).

pdadi

19th March 2009, 21:33

thanks for the info qyot27. with 24.975 audio is always in in synch. just adding assumefps(25) at the end and encode it in HC and do a dgpulldown 24.975 to 29.97 might work

QuadraQ

28th March 2009, 04:37

Ooops ... works perfectly when I
- Do the encoder settings in HCgui_023 and save the HCini
- Exit HCgui and fire up HCenc_023.exe separately
Looks as if the HCgui cannot start the encoder properly.
Scratching my head .... perhaps something is borked here ....

Just wanted to add that I ran into the exact same issue. I create a dga from a film avc mkv file, and then create an avs file using meGUI. Open it up in HCgui 0.23 fine, but when I press encode it's stuck. Tried opening up the HCenc 0.23 directly as suggested and it runs through the first pass fine, but then freezes on the second pass. Suspend then resume doesn't help. :-(

Here's my AVS script:
AVCSource("C:\Users\Isaac\Videos\720p AVCHD\Test.dga")
#deinterlace
#crop
Spline36Resize(720,480) # Spline36 (Neutral)
#denoise

Here's my HC.ini file:
*INFILE c:\users\isaac\videos\720p avchd\Test.avs
*OUTFILE C:\Users\Isaac\Videos\Temp Files\Test.m2v
*LOGFILE C:\Users\Isaac\Videos\Temp Files\Test.log
*BITRATE 4000
*MAXBITRATE 8000
*FRAMES 0 71154
*AUTOGOP 12
*PROGRESSIVE
*PULLDOWN
*MATRIX mpeg

I've tried the 0.24 beta as well with the same results. I'm tearing my hear out here!

Guest

30th March 2009, 05:44

Hank never gave me a solution for this and without the source code I am powerless.

QuadraQ

30th March 2009, 06:14

I should have stated that this only seems to be a problem with the NV version of DGAVCDec. When I tried it with the "normal" DGAVCDec it works perfectly fine.

hank315

31st March 2009, 22:33

It will be solved in the final 0.24 release, I can replicate the issue using DGAVCDecNV.
It's a complete lack of time ATM to get a new release out :o

G_M_C

21st April 2009, 12:09

Finnaly found the time to do a BD -> DVD conversion, using HCEnc & DVD-lab. The encoder works perfectly, as usual; But is was kind of funny.

It's been some time that i've done such a project, and i've upgraded to a Core 2 Quad Xtreme 9650, wich runs at a conservative 3200 MHz (400 x 8). The funny thing is that the whole 2-pass encoding process of a 2 hrs movie took less than the 2 hours the movie is long. It ran at more than 150 fps during encode :eek:

I remember times that encoding went @ 5 or 6 fps. That goes to show how technoligy has evolved in the years, but also that HCEnc seems to have matured considerably during that time as well. And that not oly gos for the speed of the encoder, but the quality it produces as well; Thats very very good (lum_gain=1 and AQ=1 seems to be a very good combo). So thanx again for all the work and time you have invested in the years !

halsboss

16th May 2009, 05:23

create a dga from a film avc mkv file, and then create an avs file ... Open it up in HCgui 0.23 fine, but when I press encode it's stuck. Tried opening up the HCenc 0.23 directly as suggested and it runs through the first pass fine, but then freezes on the second pass.
Oh NO !
It will be solved in the final 0.24 release, I can replicate the issue using DGAVCDecNV.
It's a complete lack of time ATM to get a new release out :o
Oh Good. Thank goodness, this is significant, DGAVCDecNV is of no use to me otherwise. Is it still a problem when using the lossless intermediate file ?

Any timeframe on it as yet ?

Guest

16th May 2009, 05:25

You can use version 0.22 as I do.

Unfortunately it's an HCEnc issue. I tried to work around it but there is no way.

If Hank wants to lend me the source code I can try to fix it, otherwise we are totally at his mercy.

Don

halsboss

16th May 2009, 06:11

Thanks Neuron2. OK.

It it works for a 1st pass for QuadraQ, I wonder if it'll work when an intermediate lossless file is used for the 2nd pass to "work off". Will clutch at that straw tomorrow :)

Hank, you into sharing, matey ?

hank315

17th May 2009, 19:56

Boulder

17th May 2009, 20:08

Thanks a lot, hank :) It feels weird that v023 was released a year ago..time flies so fast!

Sharc

17th May 2009, 21:33

- fixed some issues using AVCsource input
This will fix the problems using DGAVCDecNV.

Thanks, hank!

halsboss

18th May 2009, 11:13

Hank, you are a Legend, mate.

manolito

18th May 2009, 17:29

Thanks a lot, Hank!

I can confirm that the -1pass and -2pass command line parameters are now working...:thanks:

Cheers
manolito

Fr4nz

18th May 2009, 20:48

Thanks Hank for your efforts!

Excuse me for this question: did you see this (http://forum.doom9.org/showthread.php?t=146537) thread? Are you planning something in the upcoming 0.24, in order to improve encoding in "dark areas"?

kumi

19th May 2009, 00:09

Thanks for the new beta!

Can hank315 or someone else explain in layman's terms what INTRAVLC is, and what it's good for?

hank315

19th May 2009, 08:36

About IntraVLC...

MacroBlocks can be encoded in two different ways:

-------------------------------------
| Intra VLC | 0 | 1 |
|------------------------------------
| Intra blocks | B14 | B15 |
|------------------------------------
| Non-intra blocks | B14 | B14 |
-------------------------------------

B14 and B15 are two tables which are used to perform the VLC: Variable Length Coding.
The outcome of this will be written in the video bitstream.
Table B14 is more efficient when there are only a few values in the 8x8 block, table B15 is more efficient when there are lot of values in the 8x8 block.
So in general table B14 should be used for low bitrates, B15 for higher bitrates.
IntraVLC can be changed for every frame.
So *INTRAVLC 0 means: always use table B14, *INTRAVLC 1 means always use table B15, *INTRAVLC 2 will use the VLC method which gives the lowest nr of bits for that frame, *INTRAVLC 2 simply does both and picks the table which generates the lowest nr of bits to represent the encoded block.

scharfis_brain

19th May 2009, 09:23

How much gain in bitrate do you estimate by Automatic IntraVLC compared to fixed IntraVLC?

hank315

25th May 2009, 18:27

The table shows file sizes of a testclip for different INTRAVLC settings using different Quantizers.
All other settings are the same so it shows the efficiency of the INTRAVLC setting only.

---------------------------------------------
| CQ | intravlc 0 | intravlc 1 | intravlc 2 |
|----|------------|------------|------------|
| 2 | 101674 | 100606 | 100576 |
| 3 | 53550 | 53140 | 53086 |
| 4 | 45531 | 45247 | 45182 |
| 5 | 33787 | 33782 | 33629 |
| 6 | 28421 | 28541 | 28330 |
| 8 | 23530 | 23844 | 23507 |
| 10 | 16561 | 17036 | 16561 |
---------------------------------------------
It shows table 14 (intravlc 0) is better at low bitrates (high Q) and table 15 (intravlc 1) is better at high bitrates (low Q).
Intravlc 2 (auto mode) always produces the smallest file size and converges at low bitrates to table 14.

So the gain using auto mode is pretty small... but it comes at almost no extra cost, about 1% extra encoding time.

doxville

27th May 2009, 22:12

Hi hank315, many thanks for all the time you spend on this great encoder!

Since 0.24 is a beta, I wanted to ask you, if you had the time to look over this issue (http://forum.doom9.org/showpost.php?p=1161337&postcount=60), because I found the same behaviour as in 0.23. Would be great if that could be fixed in the final release (i am still sticking to 4:3 encodings), even if it's not a big thing to change AR yourself.

:thanks:

cweb

28th May 2009, 07:09

Just to thank you Hank for your new beta release..

zack_ind

28th May 2009, 12:03

Here's a new HCenc 0.24 beta (http://www.mediafire.com/?kuzzgimg2um):

- added *INTRAVLC command
- added 4:2:2 input / output
- fixed some issues using AVCsource input

This will fix the problems using DGAVCDecNV.

Adaptive Quantization even when set to 0 doesnt work. In the ini file its not there but once we encode the log file shows it as 2.
This is in 0.24 beta , 1 pass vbr is quite brilliant

manolito

30th May 2009, 17:21

Small Bug in the current 0.24 beta:

Contrary to the documentation the default INTRAVLC mode is 0 and not 2. To enable the INTRAVLC auto mode you have to add the "*INTRAVLC 2" command to the HC.ini file.

Cheers
manolito

Guest

1st June 2009, 00:52

This will fix the problems using DGAVCDecNV. I only just noticed this. :) Thanks!

I'm curious as to the nature of the bug that caused that. Can you give a quick explanation, please?

hank315

1st June 2009, 20:58

@neuron2
Previous versions did something like this:

avsDLL = LoadLibrary("avisynth.dll");
env = CreateScriptEnvironment(AVISYNTH_INTERFACE_VERSION);
...
//get video info
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get movie info...
...
//pass1
delete Video;
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get frames and encode...
...
//pass2
delete Video;
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get frames and encode...

HCEnc 024 does something like this:

avsDLL = LoadLibrary("avisynth.dll");
env = CreateScriptEnvironment(AVISYNTH_INTERFACE_VERSION);
...
//get video info
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get movie info...
...
//pass1
delete Video;
iret = FreeLibrary(avsDLL);
avsDLL = LoadLibrary("avisynth.dll");
env = CreateScriptEnvironment(AVISYNTH_INTERFACE_VERSION);
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get frames and encode...
...
//pass2
delete Video;
iret = FreeLibrary(avsDLL);
avsDLL = LoadLibrary("avisynth.dll");
env = CreateScriptEnvironment(AVISYNTH_INTERFACE_VERSION);
Video = new PClip();
*Video = env->Invoke("Import", AVSValue(arg,1)).AsClip();
get frames and encode...
Apparently CUDA doesn't like the old method, also the 024 GUI now releases all AVS stuff before starting the encoder.
Until now it passed all my tests using DGAVCDecNV but running multiple instances of HCEnc with CUDA won't work.

@manolito
About the intraVLC, you're right, default is now mode 0, should be 2 (auto), fixed... thanks for the report.

Boulder

1st June 2009, 21:20

Hank, I keep getting a Windows crash report message every time HC023 or 024 beta finishes encoding. This happens even with a simple script with only MPEG2Source in it (the latest DGIndex/DGDecode.dll used). I recall seeing the issue in DVD-RB some time ago, but jdobbs did something to fix it in his application. I uploaded the crash dump and a debugging log, here's the link: http://www.mediafire.com/?sharekey=e1621205ec925d0f4c17ca8801618ef7e04e75f6e8ebb871

I have not made any massive changes in my system, also different Avisynth versions do not help nor does emptying the plugins folder. It was not a long time ago that I was able to run multiple HC encodes in a row from a batch file so I'm quite puzzled.

AlanHK

22nd June 2009, 09:20

I get random crashes with the message in the log:
*** ERROR, Avisynth message: invoke environment error - pass 2

AVS script:
DirectShowSource("17.m4v",fps=23.976,ConvertFPS=True)
EnsureVBRMP3sync()
ConvertToYV12()
AmplifydB(11.07)
LanczosResize(688,448)
AddBorders(16,16,16,16)

Source file is M4V, 720x544

HC invoked with this batch:
set rate=2164
set ar=4:3
set hc=P:\HCenc\HCenc_023
set mpgd=s:\mpg\
for %%A in (
16.avs
17.avs
18.avs
) do (
%hc% -i %%~fA -o %mpgd%%%~nA.m2v -b %rate% -aspectratio %ar% -pulldown -profile best -matrix qlb -frames all -noini -2pass -maxbitrate 8000 -log %%~dpAhenc.log -avsreload
)

File 16 encoded okay.
All files open in VDub, and I can extract audio with wavi with no issues.

ffdshow is being used by Avisynth.

I restarted the encode, and had Process Explorer open to observe when pass 1 was finished.
This time it crashed Windows (Win2k) and the the computer rebooted.

I tried again, and the third time it encoded perfectly ?!
When running on this HC is using about 360 MB RAM, out of 1 GB total. Nothing else running

I use the avsreload option hoping it would assist here, but obviously not.

Any point trying v 0.24? It isn't on the normal download page, so is it ready for general use?
I don't need any new features, but if this bug is fixed I'd obviously try.

EDIT:
With nothing to lose, I did try 0.24.
So far it's done the file that crashed and two others.
A few other nice enhancements too.
As I said, these crashes are erratic, too early to pronounce it solved, but it looks good.

Sharc

15th August 2009, 16:44

There still seems to be an issue with the interoperability of DG...NV tools (beta7) and HCenc 0.23 / 0.24 beta:
After running pass 1 successfully, pass 2 of HCenc just does not start. Any remedy for it?

hank315

22nd September 2009, 20:51

A new beta of HCenc 024 is out.

Major changes:
- added *INTRAVLC command
- added *MINBRFAC command
- added 4:2:2 input/output
- fixed some issues using AVCsource input
- better fade detection/handling

and some other minor changes and bugfixes.

INTRAVLC 2 is now default, every frame uses the optimal intra VLC table.
The *MINBRFAC command shifts the lower part of the compression curve.

Link: HCenc024 beta (http://hank315.nl/files/HC024_beta_22-09-2009.zip)

Boulder

22nd September 2009, 21:21

Thanks hank :)

cweb

22nd September 2009, 21:25

thanks from myself too :)

Dkruskie

22nd September 2009, 22:42

Thanks hank

Sharc

22nd September 2009, 22:44

Thank you hank.

Revgen

23rd September 2009, 00:24

Good work Hank. I'll try it later tonight.

mikenadia

23rd September 2009, 03:10

Thank you , Hank, for the new beta.
HC still crashes in Lossless mode with AUTOGOP 12 ( not with AUTOGOP 13).
http://forum.doom9.org/showthread.php?t=140361 (post #15).
Thanks in advance.

Fishman0919

23rd September 2009, 05:20

Thank You Hank

Revgen

23rd September 2009, 13:42

Well, there's a problem.

I just encoded a tv capture and HCEnc 0.24 (latest beta) is adjusting the bitrate by -4.46%. While HCEnc 0.23 has done this at times in the past, it never went over 1%. 4.5% in any direction is way too much.

Here's the .ini file settings.

*BITRATE 6900
*MAXBITRATE 9500
*FRAMES 0 123672
*PROFILE best
*ASPECT 4:3
*AUTOGOP 12
*AQ 0
*DC_PREC 10
*PROGRESSIVE
*PULLDOWN
*INTRAVLC 2
*MATRIX mpeg
*COLOUR 4

I'll try encoding this with HC 0.23 tomorrow and see if the same behavior happens.

EDIT 1:

Other TV caps are experiencing 10% bitrate deductions.

EDIT 2:

Just finished encoding with HC 0.23. Encode size is what it should be. So something is definitely wrong with the way HC 0.24 calculates bitrate. Later today, I'll see if I can replicate the problem on a smaller sample.

hank315

23rd September 2009, 20:22

@Revgen
Probably this new beta will saturate easier.
The automatic intravlc is pretty effective, also quantization with a small deadzone is now used.
Please try adding the next line:

*DEADZONE 0 0

This will disable the deadzone quantization, just to see if it makes a difference.
Also a visible comparison between the two would be interesting.

Thank you , Hank, for the new beta.
HC still crashes in Lossless mode with AUTOGOP 12 ( not with AUTOGOP 13).
http://forum.doom9.org/showthread.php?t=140361 (post #15).
Thanks in advance.
I can't replicate this, does it crash at every segment of the movie?

mikenadia

23rd September 2009, 21:06

Hank, this is the ini file.
*INFILE c:\working\d2vavs\v05000100001002.avs
*OUTFILE C:\working\D2VAVS\V05000000001001.m2v
*DBPATH C:\working
*LLPATH C:\working
*BITRATE 4000
*MAXBITRATE 8000
*FRAMES 0 7571
*NOSMP
*PROFILE fast
*AUTOGOP 12
*NOSCD
*LOSSLESS
*INTRAVLC 2
*MATRIX mpeg

the avs is loaded (in info:input using ini file: allocated memory).
It happens in all avs file (generated by DVD-RB) of different movies.

#------------------
# AVS File Created by DVD Rebuilder
# VOBID:01, CELLID:01
#------------------
LoadPlugin("C:\Program Files\DVD Rip Prog\DVD-RB\DGDecode.dll")
mpeg2source("C:\WORKING\D2VAVS\V05.D2V")
trim(0,7571)
ConvertToYV12()

If I shift the AUTOGOP number from 12 to 13 back and forth, it is, on my setup, consistent behaviour.
Thanks for your help.

Revgen

24th September 2009, 06:43

The deadzones workaround worked.

Here are bitrate differences for the -10% video. Turns out it actually lost -15% bitrate.

http://img34.imageshack.us/img34/8497/hc24bitrate.th.jpg (http://img34.imageshack.us/i/hc24bitrate.jpg/)

The bitrate I entered was 6300, which HC23 and HC24 with deadzones seem to match closely. HC23 is about 13kbps short, but it's not too bad. HC24 with no deadzones is really bad, and it's doing this across the whole video, not just certain sections.

This movie is a b&w movie about miners and there are several dark mine scenes. The very low bitrate section on the far right is one of those darker scenes.

As far as how the image looks, the movie is b&w plus it's pretty dark at times, so I actually have to look for the blocks to see them, but when I do I can see the difference.

Emulgator

24th September 2009, 09:40

Many thanks to Hank again !
I was checking with poor, dark and grainy PAL-SD DV50 source @ 4950kbps.

Qualitywise you are so close to CCE @ comparable settings
that I have a hard time to spot any difference.

Because this grainy stuff tears the most regarding this half-starved bitrate
both encoders exhibit the same grain-modulated blockstuctures if stepped frame-wise.
OK on normal playback.

Very good work !

My 2 test encodes so far hit the bitrate.
First (LUMGAIN=4) @ 4900kbps hit the 4900.
Second encode LUMGAIN=0,AQ=3 @ 4950kbps hit 4951.

(Konran's BitrateViewer in GOP enhanced mode used)

Using AQ=3 both bitrate distributions (HC vs. CCE) have been very similar and useful.
The last part of the source containes smoothed and deinterlaced material
and both encoders managed to save on bitrate here.

weaker

24th September 2009, 11:08

Thank you Hank for further continuing with HCenc development! And thanks to those who beta-test it.

manolito

26th September 2009, 14:35

Thanks a lot Hank for the new version. Did a couple of tests in 1-pass VBR mode, works great!

I am curious about the new features. Is the *MINBRFAC command an equivalent to a Min Bitrate parameter? Is raising the value above the default 1.0 recommended to help avoid artifacts in dark scenes?

And more questions about the undocumented commands:
Even though you probably do not want the average user to play with these settings, could you elaborate a little about them?

I did google for Deadzone Quantization, but frankly this stuff is way above my head. Is it generally a good thing to use it, or are there situations where it should be turned off?

Other undocumented commands I found are
*MASK_VALUE
*MPEGLEVEL
*NOSMPDECODE
*QROUND
*PSYOFF

Could you explain just a little what these commands do?

Thanks again,
Cheers
manolito

hank315

4th October 2009, 23:00

@Revgen
Thanks for testing.
Your source is of course not the usual kind but it shows the settings are not optimal yet for all conditions, I will make it adaptive for different sources, bitrates etc.

I am curious about the new features. Is the *MINBRFAC command an equivalent to a Min Bitrate parameter? Is raising the value above the default 1.0 recommended to help avoid artifacts in dark scenes?
It manipulates the lower part of the compression curve, so yes higher values will prevent artifacts in dark scenes.

I did google for Deadzone Quantization, but frankly this stuff is way above my head. Is it generally a good thing to use it, or are there situations where it should be turned off?
Quantization is only a mapping of DCT values to a (smaller) set of discrete values, using a deadzone means small values are easier quantized towards zero.
So some noise will be quantized to zero, useful for low bitrate encodes but for high bitrate encodes it will saturate easier.

There are some hidden commands meant for testing purposes:

*MASK_VALUE Ymask Umask Vmask
If *MASK_SHIFT is used, these values are used for masking the top/bottom bars.
Default: Ymask=16, Umask=-128,Vmask=-128

*MPEGLEVEL level
HCenc will automatically set the level according to the resolution of the movie, with this command you can force the MPG level.
Possible values for level: MP@ML, MP@H-14, MP@HL.

*NOSMPDECODE
Normally HCenc encodes frames for GOP n and reads/processes input frames for GOP n+1 (look ahead) at the same time in different threads, this command just disables the parallel encoding/reading, it will not disable parallel encoding.

*QROUND R1 R2
The MPEG standard exactly describes how to decode an MPEG stream but a programmer is free how to quantize the frequency domain data (after forward DCT).
Part of the intra quantizing: (3 * Q + 2) >> 2, HCenc uses (R1 * Q + R2) >> 6, default: R1 = 48, R2 = 32, best to leave it at the default values.

*PSYOFF
This command disables fade detection and some other luminance on dark scenes adjustments.

manolito

5th October 2009, 11:57

Thanks Hank for your detailed explanations.
Very useful for me :)

Cheers
manolito

Emulgator

7th October 2009, 21:43

Many thanks from me too and I as well will continue to test
and maybe utter some helpful thoughts to bring HC to the top...

MilesAhead

20th October 2009, 02:19

Are there sort of "nightly builds" or stuff newer than Sept. beta for those of us who want to mess around "updating" converter suites, to download? Sept. beta is the newest I can find.

hank315

22nd October 2009, 23:13

Are there sort of "nightly builds" or stuff newer than Sept. beta for those of us who want to mess around "updating" converter suites, to download? Sept. beta is the newest I can find.Here's a new one.
Deadzone quantization is now adaptive, see HCgui tab settings 1.

link: HCenc024 beta (20-10-2009) (http://hank315.nl/files/HC024_beta_20-10-2009.zip)

Chumbo

22nd October 2009, 23:26

@hank,
I just started using hc023 again for some encodes I needed, but this time under Windows 7. The program works just fine, however, when the encoder closes after it's done and when you attempt to exit the GUI, it causes a program "crash." This happens consistently. When the encoder GUI finishes, you get something like "the mpeg2 encoder has stopped working" and when you click Exit on the normal UI, you get something like "HCgui 0.23 stopped working" and so on.

You can then click on Cancel or Close Program to get out. I've tried running in Windows XP and Windows 2000 modes but I get the same thing. It may be that I'm running under the 64-bit version? I thought you may want to know. Is there any chance to get a 64-bit version of HC023?

[EDIT] I tried the latest 024 beta from the link above and the GUI exists cleanly which is nice.

[EDIT2] Well, I was wrong. It looks like the exit has to do with whether an hc.ini file exists or not. After I finished an encode with 024 beta, the same thing happened with the encoder and then with the GUI. After I removed hc.ini, the GUI exits without the "crash."

MilesAhead

23rd October 2009, 07:31

Here's a new one.
Deadzone quantization is now adaptive, see HCgui tab settings 1.

link: HCenc024 beta (20-10-2009) (http://hank315.nl/files/HC024_beta_20-10-2009.zip)

:thanks:

Dkruskie

24th October 2009, 03:39

Thanks Hank

Blue_MiSfit

26th October 2009, 18:36

Nice! Adaptive deadzones huh?

HCenc is quite marvelous ;)

~MiSfit

stenews

26th October 2009, 21:39

@Emulgator
I saw from a couple of images you attached in this tread that you use an "unseen" bit rate viewer to check the *.m2v.
How can I get it?

@manolito
hi mate,
how do I set up this new version to make it works with D2S?
I mean, do I need just to replace it with the new one or what else?
I don't remember the way you explained me before.

Bye and thanks to both of you.
Stefano.

manolito

26th October 2009, 23:21

how do I set up this new version to make it works with D2S?
I mean, do I need just to replace it with the new one or what else?
Yes, and you should rename HCenc_024.exe to HCenc.exe.

Cheers
manolito

Revgen

27th October 2009, 00:47

Thanks Hank. I'll check it sometime this week.

stenews

27th October 2009, 09:17

@manolito,
many thanks for your kindly reply :)
which is the most great improvement of this new beta release?
I mean, could you suggest any extra command to add to my HC.INI?

Bye,
Stefano.

Emulgator

27th October 2009, 16:36

Konran's Bitrate Viewer is here:

http://www.winhoros.de/docs/bitrate-viewer/index.html#top

stenews

27th October 2009, 20:28

@Emulgator,
thanks a lot mate!

Bye,
Stefano. :thanks:

manolito

28th October 2009, 03:26

which is the most great improvement of this new beta release?
I mean, could you suggest any extra command to add to my HC.INI?
Hi Stefano,
all the latest HC improvements work very well using the defaults, so I do not have any extra commands in my HC.INI. The only commands different from the default values I use are
*LUMGAIN 2
*AQ 2
because they help a lot in avoiding artifacts in dark areas. And so far I have not noticed any adverse effects with these commands, so I use them all the time.

Cheers
manolito

Emulgator

28th October 2009, 11:54

manolito, I did not know too well: So I can have LUMGAIN and AQ together ?
I thought before to have read that if one specifies LUMGAIN > 0 then AQ would be disabled in HC.
But I would be happy to hear that I can specify both and both would be respected...
...I don't dare to check now for myself, x264 and extensive script frontends running for 20 more hours of testing...

stenews

28th October 2009, 19:20

@manolito,
sorry for my delayed reply, I have to say many thanks again for your great tips! :thanks:

hank315

28th October 2009, 20:59

manolito, I did not know too well: So I can have LUMGAIN and AQ together ?
I thought before to have read that if one specifies LUMGAIN > 0 then AQ would be disabled in HC.
But I would be happy to hear that I can specify both and both would be respected...
Yes, LUMGAIN and AQ can be used together, in the latest releases AQ = 2 is the default value.

Emulgator

29th October 2009, 22:05

Many thanks, Hank !

Revgen

31st October 2009, 17:11

This latest beta has been working fine so far. Quality looks great too.

manolito

1st November 2009, 00:55

This latest beta has been working fine so far. Quality looks great too.
Same here :)
Time to go stable?

Cheers
manolito

ron spencer

1st November 2009, 02:19

works for me too!!! Nice job. Would be nice to have an expert user document the new features though...in any case great!!!!

sunfish

5th November 2009, 07:01

Is there any reason why HCenc is only running at 70-75% CPU load? I believe running it at 100% at low priority is more optimal.

hank315

7th November 2009, 01:22

Is there any reason why HCenc is only running at 70-75% CPU load? I believe running it at 100% at low priority is more optimal.
It's because the way multi-threading works in HCenc, it's frame based. Because of the frame dependency the best it can do is encode 3 frames at the same time (P and 2 B-frames).
In the same time DGDecode or Avisynth decodes the input frames, this all results in 70-90% CPU load on a quad core.
It can be worse, if you use heavy filtering in Avisynth, CPU load can drop to 25-30% on a quad core, most filters are single core only, so the encoder has to wait for the input data.

mikenadia

27th November 2009, 20:51

hank315

29th November 2009, 22:00

For a given video, I have a bitrate of 3100kbps and avg quantizer of 5.5 for matrix Rockas5 and 6.5 for matrix Rockas4.
I was wondering if it were not possible to modify the matrix, after the 1st pass, to get closer to a desired avg quantizer of 6.0 for example.
Hc 3100 6.0 as a command (2-pass with a targeted avg quantizer).

We could also create a set of family of matrices that depends on only one parameter ( forcing the matrix to be symmetrical with same coefficient for i+j= Cte) and knowing the frequency spectrum ,find the parameter that will achieve the compression desired.

Another suggestion: a function to find CQ for a given bitrate. It is my undersanding that HC will be in a better position to make a better educated guess after a first pass than using Q* Size=Cte.

Creating a family of matrices to predict a desired Q should be possible, maybe something for the next release.
What do you mean with "forcing the matrix to be symmetrical with same coefficient for i+j= Cte" ?

cosmetics.
May be put the current fps to 0 when in "suspend" mode. That's a simple one, done...

BTW, here's (http://hank315.nl/files/HC024_beta_27-11-2009.zip) a new build, it fixes some bugs in the dead zone quantization and has 3 new matrices (thanks manono).
Still beta but almost final, working on some speed optimizing issues.

mikenadia

30th November 2009, 03:47

By looking the different Rockas matrices, I realized that most of them were having coefficients that were almost equal along the bottom-left to top-right diagonals:
From Rockas3.
*CUSTOMMATRIX
08 08 09 11 13 13 14 17
08 08 11 12 13 14 17 18
09 11 13 13 14 17 17 19
11 11 13 13 14 17 18 20
11 13 13 14 16 17 20 24
13 13 14 16 17 20 24 29
13 13 14 17 19 23 28 34
13 14 17 19 23 28 34 41

I assume that this matrix will have similar results than
*CUSTOMMATRIX
08 08 09 11 13 13 14 17
08 09 11 13 13 14 17 18
09 11 13 13 14 17 18 19
11 13 13 14 17 18 19 20
13 13 14 17 18 19 20 24
13 14 17 18 19 20 24 28
14 17 18 19 20 24 28 34
17 18 19 20 24 28 34 41

This matrix is defined by 15 coefficients (a11 to a18 and a28 to a88). Those 15 coefficients are in increasing order , could be defined with one parameter as a function of (a11=8,a88-a11=41-8=33) and all the coefficients could be adjusted with a factor "t" to get close to a targeted quantizer, the adjusted matrix, in this example, being a function of (a11=8,t*(41-8)).

Thanks for the new build,hank.

Edit 1: you could also have (I think it is overkill) have a second degree of freedom on each diagonal if you want the coefficients on each diagonal to have small variance between themselves.

Edit 2:Cosmetics in the GUI: it will be nice if Manono could provide us with a pct of Fox1 (or any other one) for his matrices (someone posted for example mpegstd=45 % and Manono2= 63 %) and if that "info" was added in the Matrix Tab. Or a bitrate range.
Also, to go to custom matrix, you have to unclick built-in matrix. The custom matrix appearing by default could be the latest built-in matrix instead of MPEG to make small changes or allow us to save to mtx file.

ron spencer

30th November 2009, 12:55

thanks....how does the min bitrate thing work? awesome work!!!

hank315

2nd December 2009, 22:27

how does the min bitrate thing work?
In a 2 pass encoding HCenc always starts the first pass with a Quantizer = 4.
Using this encoding data a bitrate curve is created, quantizers for I, P and B frames are estimated for each GOP for the second pass.
The min bitrate factor will tweak the lower boundary of the bitrate curve, a value > 1 will raise the lower boundary of the curve meaning lower quantizers will be used for the low bitrate parts of the movie.

Cosmetics in the GUI: it will be nice if Manono could provide us with a pct of Fox1 (or any other one) for his matrices (someone posted for example mpegstd=45 % and Manono2= 63 %) and if that "info" was added in the Matrix Tab. Or a bitrate range.
A compression percentage for all built-in matrices with respect to the MPEG matrix would be nice, it requires a lot of of tests to get an average value for each matrix.

Also, to go to custom matrix, you have to unclick built-in matrix. The custom matrix appearing by default could be the latest built-in matrix instead of MPEG to make small changes or allow us to save to mtx file.
Nice idea, will implement such a thing.

mikenadia

3rd December 2009, 03:19

From Hank315:The min bitrate factor will tweak the lower boundary of the bitrate curve, a value > 1 will raise the lower boundary of the curve meaning lower quantizers will be used for the low bitrate parts of the movie.

Is this adjustment a personal taste one (independent of source, bitrate...)? If it is source dependent, could it be based on the stdev of the expected quantizer distribution after 1st pass (value>1 if stdev is high).

Also, when you say first pass starts at Quantizer=4, it is Quantization at 4 Mbps or CQ=4 or just " starts at Q=4"?

Thanks in advance.

Edit: a link on MPEG2 FIFO adjustments (probably of no interest to HC).
http://forum.handbrake.fr/viewtopic.php?f=4&t=13396

hank315

4th December 2009, 21:41

Is this adjustment a personal taste one (independent of source, bitrate...)? If it is source dependent, could it be based on the stdev of the expected quantizer distribution after 1st pass (value>1 if stdev is high).
The default minimum bitrate is source dependent, it's mainly based on the maximum and average bitrate from the first pass but also on luminance.

Also, when you say first pass starts at Quantizer=4, it is Quantization at 4 Mbps or CQ=4 or just " starts at Q=4"?
It's CQ = 4, during the whole first pass.

a link on MPEG2 FIFO adjustments (probably of no interest to HC).
No, not really, HCenc encoding speed is independent of the size of the allocated buffers, only if you hit the limit (allocated memory is close to memory left) you will run into trouble.
In HCenc the allocated memory is almost linear with nMB * GOPlength (nMB = nr. of MacroBlocks per frame).

~Revolution~

6th December 2009, 07:41

hank315

16th December 2009, 23:56

The next table shows the compression ratio for the built-in matrices.

Hcenc 024 beta testclip1 @ Quant = 4
matrix size (bytes) factor

mpeg 41250657 1.0000
qlb 41213068 0.9991
notch 38494522 0.9332
bach1 36971188 0.8963
hc 40922436 0.9920
hclow 44994171 1.0908
jawor1cd 35544248 0.8617
hvsgood 39084618 0.9475
hvsbetter 40610182 0.9845
hvsbest 42612575 1.0330
avamat6 32298950 0.7830
avamat7 31357828 0.7602
fox1 97076189 2.3533
fox2 82854942 2.0086
fox3 56116450 1.3604
manono1 53362584 1.2936
manono2 54737764 1.3270
manono3 46335685 1.1233
mpegstd 50305606 1.2195
It's based on one clip only so it's a rough estimation.

AlanHK

23rd December 2009, 11:18

I'm encoding an animation with some fairly strong filters.

The first pass went at 1.6 fps, but the second pass is much slower, 0.6 fps.
The PC isn't doing anything otherwise intense to reduce the priority.

I've never seen such a large difference between first and second passes.

The AVS file:
AVISource("Flat.avi",false)
UnDot()
Dehalo_alpha(darkstr=0.5,brightstr=0.5,rx=3,ry=3)
Edgecleaner(strength=20)
Toon(strength=0.8)
Deblock(quant=35, aOffset=0, bOffset=0, mmx=true, isse=true)
LSFMod(strength=50)
Dup(threshold=6.0, chroma=false, show=false, copy=true, maxcopies=20, blend=true)
LanczosResize(704,576).AddBorders(8,0,8,0)

And the HC command:

HCenc_024 -i %%~fA -o s:\mpg\~nA.m2v -b 2251 -aspectratio 16:9 -pulldown -profile best -matrix qlb -frames all -noini -2pass -maxbitrate 8000 -log %%~dpAhenc.log -avsreload

hank315

23rd December 2009, 22:44

Don't know why this is, both seem *very* slow...
In this case using -lossless will certainly speed up the second pass.

AlanHK

24th December 2009, 05:48

Don't know why this is, both seem *very* slow...
In this case using -lossless will certainly speed up the second pass.

Thanks, but I don't have the space for a 70 minute lossless file.

But though the slowness is irritating, it's the disparity between passes that seemed strange.

Another oddity: it's now halfway through the 2nd pass, after about (or maybe exactly?) 24 hours. But the "Time used" has become 13 minutes, and the average speed is insanely high, though current speed is still 0.6. Looks like the timer has rolled over.

I really hope it gets through without blowing up, it would be very annoying for it to abort after 3 days....

Blue_MiSfit

30th December 2009, 01:05

just a note, I did some testing with 023 and compared it to ProCoder's "mastering quality", and HC blew it out of the water at all bitrates for SD :)

~MiSfit

tom942

3rd January 2010, 03:30

What is the best situation in which to use the MANONO1, MANONO2, and MANONO3 matrices? And how do they compare to the FOX matrices and other matrices? I always get a bit confused as to how all of these matrices rank but I do have an idea that it has to do with the value of the bitrate output of the encode.

In one of the several PM's that Manono sent me, he told me, more or less, this:

FHE (very similar to FOX1) matrix is used as reference. The resulting testclip is marked as 100%. Then the other matrices are marked with bigger or smaller percentages, taken into account, their level of compressibility (final size of the encoded clip).
Matrixes with low compression will have a bigger percentage than FHE.
Matrixes with high compression, will have a smaller percentage than FHE.

Manono1 matrix could be used as an enhaced version of MPEG matrix (83/16). It is sharper than MPEG (to my eyes) and compress fine.
As a guide, in DVD-RB I use it normally between 3000 to 4000 kpbs.
In the tests done by Manono, he got a 68%.

Manono2 matrix is an "old" matrix, similar to the previous one, but compress more and it is bit less sharp (to my eyes) than Manono1.
In this one, he got a 63%.

Manono3 matrix could be used as an enhanced version of MPEG_STD matrix (83/33). It is sharper than MPEG_STD (to my eyes) and compress fine. As a guide, in DVD-RB I use it normally between 2000 to 3000 kpbs.
In the tests that he done, he got with MPEG_STD a 45%, so as this one compress a bit less, it could be a 48-50%.

I hope it helps :).

Emulgator

3rd January 2010, 13:13

Hank, Happy New Year and my highest appreciations for HC 0.24 beta 27-11-2009!
This version finally kicks almost everything from the shelf.
Using no deadzones I can finally have dark and grainy parts encoded without the well-known block artifacts.
Grain stays grain as in CCE, no moving dark blocks as in Procoder (Express 2 I checked on)
or chequered-structure-changing-blocks as in Mainconcept.
I can force bitrate usefully into these dark parts as I like. No bitrate flipping as in CCE.
Picture is still sharp, even using MPEG Matrix.
Almost no visible artifacts encoding text onto gradients.
High motion is usefully encoded, no heavy block rows for 1 frame as sometimes briefly appearing in TE4XP.
I would say, I see HC on par now with CCE, better than Procoder, better than TE4XP, better than Mainconcept.
Very good work and the only MPEG-Encoder I have seen improving in terms of picture quality for the last 2 years.
And for now I have only tested SD resolutions...

Gromozeka

3rd January 2010, 18:54

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...

tin3tin

12th January 2010, 12:13

The HCenc024.exe Beta 27 - 11 crashes for me in the end of first pass(says 100%) with a 16 sec long avs script using the Soundout plugin too. It renders without errors in Hcenc 0.23.

This is the ini:
*dc_prec 10
*WAIT 0
*BITRATE 7000
*MAXBITRATE 7500
*AUTOGOP 15
*AVSMEMORY 1024
*PROFILE BEST
*PROGRESSIVE
*PRIORITY HIGH

manolito

17th January 2010, 01:53

First time that 0.24 beta got the sizing wrong in 1-pass VBR mode.

This is the deal:
Source: 90 min MPEG2 from a DVB-T capture.
Reason for reencoding: Logo removal
Encoder: Latest HC 0.24 beta Novemver 2009
Matrix used: Manono1
No filters except LogoAway, no resizing
Source characteristics: Last 4 min consists of vertically scrolling end credits, requires high bitrate
Average Bitrate: 6400
Max Bitrate: 8500
Base Q determined by HC: 2.20

The encode came out almost 4% oversized. Next thing I tried was disabling Deadzone Quantization, but the oversize was the same (speed was significantly faster, though).

Out of curiosity I did the same encode with an older 0.24 beta from May 2009, but no change, still almost 4% oversized.

And the last thing I tried was throwing this source to my old and obsolete 1-pass VBR plugin for DVD2SVCD (Hybrid Mode), and this encode came out perfectly (1% undersized).

Ok, this is purely academic. I used DVDShrink to shrink the size of the oversized encode and compared the result to the encode using my old plugin, and I was completely unable to notice any visual difference.

Cheers
manolito

manolito

20th January 2010, 01:12

I just ran into some nasty problems using the latest HC 0.24 beta in 1-pass VBR mode.

Source:
DVB-T capture, interlaced, average bitrate ~3500 kbps

Encoding settings:
HC 0.24 beta 1-pass VBR mode
Average bitrate 4950 kbps
Max bitrate 8500 kbps
GOP: Auto
Interlaced: Auto
IntraVLC: Auto
DeadZone: Auto
DC Precision: 10
Scene Change: On
Matrix: Manono2 (aka Sony medium)

Using my old and obsolete 1-pass VBR plugin (which is basically a CQ_MAXBITRATE mode with a couple of tweaks to ensure correct sizing) with the same HC settings these artifacts were all gone. This was also true when using HC 0.23.

Download the test clips here: http://www.mediafire.com/?jztmtj1kqum

Cheers
manolito

Emulgator

20th January 2010, 09:28

Oops. opv.mpg had a bitrate valley allocated at the scenecut, where a bitrate peak should have been,
especially with scene change on.

Hm. looks like poor bitrate allocation in 1-pass VBR?
In my tests I was only checking 2-pass VBR.
and this is what I use regularily together with Matrix MPEG.
(Other matrices did not help too much at my side.)

Emulgator

20th January 2010, 09:33

tin3tin
*PRIORITY HIGH
what if you leave priority at "idle"?
"High" may force its way through too much, while not offering any gain in speed,
not letting other processes in, which you may need for task to complete.

txporter

20th January 2010, 21:30

Can someone step me through the correct way to encode using 1-pass Quality VBR with HCenc? Is this similar to doing CRF encoding with x264?

I can get HCenc to encode using 1-pass, but it requires me to given it an average bitrate target. I guess I am expecting something more like x264 where I tell it I want some level of quality and it uses whatever bitrate is necessary to do this. Is that constant quant in HCenc?

Sorry for the beginner level questions. I have used HCenc for a while now with 2-pass target bitrate, but I wanted to test various avisynth functions out for compressibility and I can't see how to do that when I need to give a target bitrate.

edit: I am using HCenc 0.24 11-27-09 release

hank315

20th January 2010, 22:35

@Emulgator & manolito
Seems 1 pass VBR is seriously broken in the latest 024 beta.
In my development version I even get random crashes if all compiler options are on, running in debug mode is OK making it hard to track down the error :devil:

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Hcenc support only avs?
Or he support yuv from command line?
ffmpeg i "%1" -o - | Hcenc ...
Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

txporter

20th January 2010, 22:59

@txporter
The one pass VBR is a 2 pass only the first pass is done on 1 - 5% of the total frames.
An initial quantizer is estimated and the second pass starts with this quantizer, the Q can be varied a bit to achieve the desired average bitrate.
So the 1 pass VBR is a (approximately) constant quant encode while 2 pass is more like constant quality, both are targeted for a desired file size.
But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

Thanks, Hank! It does work in HCenc_023. I can use that to test different filters and then just apply that to 2-pass HCenc_024. I like targeted bitrate just fine, I just didn't know how to tell what effect my filters were having. Thanks a lot for your support and your program!

manolito

21st January 2010, 14:51

But 1 pass VBR in 024 beta is broken, AFAIK it's working OK in the 023 release.
Huh? HC 023 had a 1 pass VBR mode already? How stupid of me, I thought this feature was introduced in the first 024 beta :devil:

Just FYI I reencoded the clip which gave me those problems using the very first 024 beta (Nov. 2008), and the artifacts are identical. So this bitrate distribution problem in 1 pass VBR mode must have been there from the beginning.

The clip is a TV broadcast from last New Year's Eve, it is "Fleetwood Mac live in Boston 2004". The artifact comes up at the end of the last song, and this last song is a lot less compressible (requires higher bitrate) than most other parts of the clip. It looks like HC is desperately trying to reach the desired target size, so it does not even allow bitrate peaks at scene changes.

Anyway, I will give it another shot with yet another 024 beta version and report back...

Cheers
manolito

manolito

22nd January 2010, 23:08

In the meantime I ran the same encode using 3 different 024 beta versions, but the results were identical.

Conclusion: The 1-pass VBR mode strategy in HC 024 can fail under certain unfavorable conditions.

The source is really very demanding. But OTOH HC CQ_MAXBITRATE and HC 2-pass VBR do handle this source without problems, and even QuEnc 1-pass VBR can do it. I tried to cut out a 5min segment of the source with the problematic spot right in the middle, but this was no problem for HC 024 1-pass VBR mode.

In order to test different bitrate distribution strategies it is probably necessary to process the whole clip. The size is 1.2 GB, let me know if I should upload it....

Cheers
manolito

BTW I also tried to use *PROFILE BEST instead of FAST, but same result...

Gromozeka

23rd January 2010, 08:20

Only avs and DGindex input ATM.
Maybe piped YUV or YUY2 input in the future...

You will add her(its) in the future time?
Thank you, This very good function.
:thanks::thanks::thanks:

Emulgator

25th January 2010, 13:13

An interesting finding while further comparing HC vs CCE.
Source was 16mm Film (1.38:1), Duration 1:52:20, Transfer, PAL speedup to interlaced PAL-VHS, ugly and grainy.
I restored this PAL-VHS 25i to 25p using TGMC, selecteven, MC_spuds. No residual combing visible.

Now encoding using CCE, 2-Pass VBR @5,1Mbps, "Progressive frame"
and HC0.24 beta 27-11-2009, 2-PassVBR @5,1 MBps, "Progressive".

A: Comparing results on Notebook-TFT 1920x1200 using MPC+motion vector overlay, resized 200%, VitualDub resized200%, Konran's Bitrate viewer.
HC (tweaked: VBR bias 10, minBRF 1.20, no deadzones, matriy MPEG) shows more useful bitrate distribution than CCE,
Especially in the beginning, where CCE comes slowly up from poor bitrates, leaving some blocks in dark parts.
HC swings faster in to average bitrate with a light overshoot and starts from the very first frame with a smooth picture.
HC finds very useful motion vectors as well.

Later after some minutes both encodes look similar. No faults that would be visible without hard scrutiny.

B: Comparing results from DVD after authoring (DVDLabPro2.51) and burning to disc.
Panasonic BD-50 -> BD50-upscaler -> HDMI -> Loewe Connect 37 (TFT-TVFullHD37")
Here the TFT showing the DVD coming from HC sometimes shows some thin glitches,
height 1 single 1080HD line and width maybe 16 HD pixels
that were not part of the MPEG-2 stream before authoring.
These line glitches were only visible during 1xplay
and could not be seen when pausing and stepping frame by frame using RC.

(Later I saw another 23.976p pulldowned DVD exhibiting the same behaviour on this particular player/TFT combination.
The reason could be tracked down to the Panasonic BD-50's upscaler, when progressive frames should be displayed.
No problems on interlaced frames.
Panasonic DMR-EX95, while having an uglier upsizer with bad ringing, did not exhibit these glitches from the same DVD.
Phew, so not my restoration/encoding fault...)

Now comparing streams using DGIndex and ReStream.

HC: after specifying "Progressive", in GUI TFF/BFF Options are greyed out.
HC then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=no
Sequence Extension
"Progressive Sequence"=yes,
In GOPs: no tff flags.
Applies Scanning mode zig-zag.

Now CCE:
after specifying "Progressive Frame", in GUI TFF/BFF Options are not greyed out.
CCE then flags as follows:
PictureCodingExtension "Frametype Progressive",
"top field first"=yes
Sequence Extension
"Progressive Sequence"=no,
In GOPs: tff flags=yes.
Applies Scanning mode zig-zag.

Now I reflagged the HC stream to the same settings as CCE would output
using ReStream.
Authoring, burning, disc testing as before:

Now HC output looks smooth and ok on this particular BD-player/TFT-TV combination!

Then I forced interlaced encoding on progressive footage in HC
but then alternate scan order was forced by HC as well
and so I lost quality, picture became a bit blocky then.

It seems that these flags exhibit a bug in player/resizer/TFT-TV combination
that can be avoided if one reflags HC output as interlaced, even if 25p has been encoded.

To overcome this possible trap with 25p footage,
and if Hank approves this approach
I would suggest to extend GUI behaviour.

After "Assume input frames as progressive"
(info: This does not mean that HC would deinterlace !)
I'd still like to have PictureCodingExtension TFF,BFF tickable,
because no tff=1 == tff=0 == BFF;
I'd suggest to have SequenceExtension "Progressive Sequence" tickable/untickable;
(info: Unticked helps for DVD-compatibility)
and I'd suggest to have Block scan order manually settable using 3 radio buttons
"Auto" (info: Applies "zig-zag" on assumed progressive frames, "alternate" on assumed interlaced frames) or
"Override to zig-zag" (info: Useful on progressive frames, interlaced may suffer) or
"Override to alternate" (info: Useful on interlaced frames, progressive may suffer)
Fenced meaning info line in GUI that declares the use of these flags.

manolito

25th January 2010, 22:01

@Emulgator,

interesting findings, and for the most part they are consistent with my experiences...

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC. Any specific reasons? Does it generally kill grain in dark areas?

For the different TFF flagging of progressive streams in CCE and all the rest of MPEG2 encoders I had my problems a while ago (and there was a thread here which I cannot find right now), but here is the summary:

According to the MPEG2 standard the TFF flag has no meaning whatsoever for progressive content. A software- or hardware player should simply ignore it. But in reality at least some (maybe most) standalone DVD players do not ignore the TFF flag for progressive content.

I have an old and cheap Cyberhome DVD player (Mediatek chipset) connected to a CRT TV via SCART (using SVHS). The player always outputs fields (interlaced), and it does so observing the TFF flag, no matter if the stream is progressive or interlaced. For progressive content this does not make a difference, the TV picture will look the same.

The problem I had a while ago was that my encodes from TV captures had jumpy end credits. The film itself was progressive, but the credits were interlaced TFF. When I started with DVD conversions I always used CCE SP 2.50, and these encodes looked good because CCE always flags a progressive encode as TFF. But when I started using QuEnc and HC I got these jumpy end credits. Simple reason: QuEnc as well as HC do not allow to set the TFF flag for progressive encodes, so my end credits simply had the wrong field order.

The reason why CCE flags progressive encodes as TFF is probably historical. Older CCE versions would always produce TFF output. If the source was BFF, they would apply an offset of 1 scan line, so the output would be TFF. Newer versions can do it both ways, but the flagging has not changed.

Ok, the "right" solution for this problem would be to encode the film and the credits separately, but that was too much hassle for me. I asked DGZ (co-author of QuEnc) if he could modify QuEnc to allow TFF flagging of progressive encodes. He checked it, but libavcodec would not allow it. Same is true for HC: if you specify progressive, then the TFF flag cannot be set, and scan order will always be zigzag.

Well, reading your post it seems that newer upscaling standalone players also do evaluate the TFF flag even if the content is progressive. As a solution to this problem (except using CCE) I normally use HC's "Automatic" interlaced mode. If you specify neither *PROGRESSIVE nor *INTERLACED nor *DVSOURCE then HC will try to detect interlaced or progressive per frame automatically. In this "auto interlaced" mode TFF is the default, but you can change this. Scan order is also determined automatically in this mode. Works very well for me, give it a try...

Cheers
manolito

txporter

25th January 2010, 22:23

This is an interesting discussion. I started using HCenc a few months ago to re-encode DVD rips (mainly NTSC material) into single file MPEGs with hardcoded subs. I do this to serve videos up to my Tivos. At any rate, I started by encoding using the automatic detection method you described above. The videos produced from this played fine in MPC-HC. When I watched them in WMP10 though, they exhibited what I was calling strobing. For a short period of time the video was fine, and then it seemed to jump up and down a few scan lines at a time, and then would return to normal and repeat. At any rate, my tivo playback looked like the WMP playback with strobing video/smooth video/strobing video..etc. It was extremely annoying.

Well, I was able to avoid the strobing video by specifically encoding as either progressive or interlaced. Frankly, I was really only encoding NTSC material at that point, so I don't know if interlaced material showed the problem...but progressive did. I wonder if this was somehow also related to the field flagging?

Emulgator

26th January 2010, 19:57

You don't seem to like DeadZone Quantization in HC, even the latest automatic (adaptive) version in HC.
Any specific reasons? Does it generally kill grain in dark areas?
Hm, I simply fear to sacrifice something important where I could easily have given more bitrate.
For instance Procoder (2 Express) could be good, finds useful motion vectors even in the dark,
but blows it when giving bitrate there. A miser and therefore blocky.
Because only a few Y steps (16~32) are left from 16-235 to encode dark tones close to black,
I tend to value reproduction quality of dark areas higher than average.
Lots of Film lives from smooth darkness.
It may well happen that encoded output is watched on TFT with pushed levels, wrong settings etc.
Then anything blocky in dark parts becomes painfully visible and I hope to finetune my encodes to look better than average.
Even when pushed it should look smooth.
For testing I even go into a windowless and completely dark room,
doors closed, watching footage on a Laptop or portable TFT,
eyes offset from center, viewing from a top angle, just to find blocks in the dark.
If my encodes pass this, then I'm feeling safe.
As a sidenote: Sonic reps even suggested that compressionists using Cinevision
should tell editors to clip their shades below a value of 3 or 4 to pure black.
So a step from pure black to the next value of 3 or 4. ??
Introducing deadzones, therefore destroying shades. What a deal...
Do they fear to exhibit encoder flaws? I guess so.
I can be done better...
..but progressive did. I wonder if this was somehow also related to the field flagging?
I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

txporter

26th January 2010, 20:40

I guess so.
Luckily we can fix these flags to what is DVD-expected by using ReStream.

This is the first time that I have ever tried ReStream. For both cases, I am taking a 2min video clip that is 97.85% FILM according to DGIndex. I built a d2v file using Honor Pulldown flags and am running the video through TFM().TDecimate(). The input video to HCenc should be 23.976 fps progressive.

If I force HCenc to do progressive encoding, the video plays fine when I upload to my tivo. ReStream has tick marks on Frametype progressive and Progressive sequence. The field coding shows it as BFF with ----- as the tff-flag.

If I let HCenc use automatic interlace detection, the video experiences the strobing (not verified, but this is how I did it a few months ago when it did). ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.

In both cases, the scanning-mode is listed as zig-zig. Although I am not sure if that is just the type most often used or what.

I guess that I could play around with ReStream to see if I can get a strobing video to play back ok. Are the ReStream parameters what I should expect or should I be seeing something different?

um3k

26th January 2010, 21:51

Hank, HCEnc is amazing. Thank you for all your hard work.

The other day, I was wondering if it would be possible to add a tab, similar to the Preview/Zones tab, to select different sections of the video to be encoded as progressive or interlaced. This would be great for encoding files from mixed sources, and even better if 24p (soft pulldown) could somehow be incorporated, effectively allowing for variable frame rate MPEG2. I don't know if enough people would use it to make it worth adding, though.

Thanks again.

manolito

26th January 2010, 23:35

@Emulgator
All you are saying makes perfect sense, but it is all based on theory and assumptions. Would you be so kind to make some side by side tests using HC, once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way, but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Cheers
manolito

KCE

27th January 2010, 08:40

Hi. There's a problem with one of the parameters. Passing -filesize input at the command line results in a very low calculated average bitrate used. For example at the command line, I pass an avs file, an ini file (has bitrate setting) and then a single -filesize argument which has a reasonable number such as 4395283 kilobytes for 1 hr 36 min 23.976 movie, which HC should calculate to around ~6000 kbs but instead HC calculates ~250 kbs when it starts encoding.

Also could I get some more information on how you calculate the file length/average bitrate in the gui window? I'm trying to automate this for a large number of files, which is why I'm passing filesize input. The formula I use is to just subtract the audio file size from the total disc capacity [no menu/subs/etc. and includes author overhead] to get the resulting space left entirely for video. I then input that filesize into the HC GUI and get the average bitrate. Actually if I save the ini file and reload it, then the filesize shrinks by a few megabytes which I'm guessing is HC approximating to the nearest possible filesize it can accomidate. I have a feeling I'm missing something really simple in my calculations...

I also use this formula (http://neuron2.net/LVG/ratesandsizes.html) or videohelp calc to get the average bitrate but the average bitrate I calculate tends to be lower by ~50 than the one produced by HC when I manually input the filesize in the gui window. However since the filesize argument doesn't work, it ends up using my calculated bitrate in the ini file. Then I'm left with about 25 MB of extra space on the disc which could go towards the video (every little "bit" counts :p)

The second problem is lossless file option does not work with autogop of 12. I forgot some of the other options I tried at the moment...but it would just crash a second after I start encoding.

Emulgator

27th January 2010, 12:17

manolito:
Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

Yes, was in my mind too.

Emulgator

27th January 2010, 12:26

(using HC autodetect)
txporter:
ReStream now has tick marks on Frametype progressive and TFF (++++ as the tff-flag), but not Progressive sequence.
Are the ReStream parameters what I should expect or should I be seeing something different?
Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

midnightsun

27th January 2010, 16:28

txporter

27th January 2010, 17:21

(using HC autodetect)

Looks good to me. And using autodetect is what manolito suggested as well.
I just didn't dare to use autodetect...

Just as a follow-up, I went ahead and created two clips of 3:2 pulldown material. Both were sent through TFM.TDecimate. One was encoded using autodetect and the other being forced as progressive.

1) Auto clip - experienced strobing on Tivo
2) Prog clip - smooth playback
3) Auto + ReStream set to BFF - jerky forward/back motion indicative of setting the wrong field order
4) Auto + ReStream set to Progressive sequence - smooth playback
5) Auto + ReStream set to BFF and Progressive sequence - smooth playback

I guess it is just how the tivo handles MPEG2 video. It's easy enough to fix by just forcing progressive encoding when I need it.

KCE

28th January 2010, 02:49

KCE, if you really want to maximize video size and you say you fall X MB short, add [(X*8*1000)/(running time in seconds)]kbps to the pre-calculated bitrate.
Be aware that for 25MB and 1:30:00 worth of footage, the bitrate goes up by just 37kb/s.

Thanks. Actually I thought it was the same for each file but it's not though I can live with a *small amount of space left over. Anyhow, the values in HC don't seem to follow the standard formulas. Maybe hank can give some insight?

midnightsun

28th January 2010, 09:29

It has to be the other way around actually, i.e. the formulae are not 100% correct in calculating DVD overhead (another explanation could be that they're tuned to leave you a little bit of "safe area", 20-25MB you're finding out to be unused, so you don't end up with a slightly oversized DVD).
If that's of any help and my memory serves me right, I empirically noticed that DVD overhead is about 1.8% of the combined size of video+audio streams for a typical 1:30:00-2:00:00 DVD (give or take 0.1%). If you use 2.2% you should *always* be alright:

[(4,700,000*8*0.978)/(running time in seconds)]-[total bitrate of audio/subtitle streams] kb/s

(4,700,000 is the size of a DVD-R in kbytes as you know, DVD+R is slightly larger at 4,706,000)

For 1:40:00 of video+(192kb/s)audio you get 5937kb/s, while videohelp calculator yields 5903kb/s.

If you feel like it, try it out on a project of yours to see if you come closer to full capacity while not going oversize.

KCE

29th January 2010, 23:44

Actually I've seen a suggestion of 1.1% of Length(sec) which seems pretty good to me...don't have the link on me but it was somewhere on videohelp. I understand it's tricky calculating, predicting, and actually hitting a target filesize.

I just did 11 encodes of popular movies and two of them (Pirates of The Caribbean II/III) were oversized by ~10 MB (the final authored DVD). The others were undersized from 0.13 MB (Pirates I in fact) to 22 MB. All audio is AC-3 192. Compared to the other movies encoded, Pirates is a lot longer and thus more frames, and of course the average bitrate used was relatively lower. Here's something I noticed

MB (remaining)|Length(sec)
22.1 5340
16.6 6240
10.8 6720
8.5 7020
7.2 7200
4.3 7860
0.125 8580

Based on DVD capacity of 4596992 kilobytes (~DVD-5). Bitrate and MB are also correlated - high mb = high bitrate. Other than the obvious, I'm not sure what other conclusion to draw from this? There seems to be a cut-off point (in seconds?) at which the file becomes too big. At that same point is where my calculations aren't useful anymore - my target average bitrate is too high. The m2vs produced by HC enc are right under the targeted filesize so the problem lies with authoring overhead.

I calculated average bitrate inputted into HC with the videohelp formula:

(Size - (Audio x Length )) / Length = Video Bitrate

or simpler

(Size / Length) - Audio Bitrate = Video Bitrate

I didn't include overhead into the "Size", mainly because videohelp didn't either so I figured okay. Had I did, the Pirates movies would probably fit and the others would be a bit more undersized, but like I said I can live with that. At the same time the 22.1MB movie could have used an increase in average bitrate because of the free space. So I guess that's what I'm looking for - that bitrate offset - or a better formula. The thing is I need the file size that HC generates to know how much more to add/subtract.

Actually I'm writing a program to automate multiple files->DVD, which I would describe as a robust scripting-made-easy program, so I'm looking for more of a general formula to go by. If it works well, I'll release it here. I'm sure we could use another conversion program...:p

Okay I'm getting off-topic now. Back to HC...

mikenadia

1st February 2010, 17:07

Looks solved in Edit: For a specific DVD (Taxi2), HC024 27-11 beta (HC023 too) keeps computing but hangs (nb of frames encoded do not move, time left is blocked..) but on Windows Task manager, it seems the application is still working (workload moving from 40% to 99%). It happens to me only twice.
in a 100 000 frames files; Trim(1,60000) works fine (using HC with an avs file produced by DVD-RB) . With identical Trim, it seems (I checked for n<12) that it hangs for every n>=7 if I add to the DVD-RB avs file SelectRangeEvery(n,1) and does not hang for every n<=6. I thought it was related to GOP length but it does not look like (I tried a lot of settings in HC.ini to change the behaviour for n=6 or 7 but to no avail). Also, the encoding speed is ,from frame 1, much quicker when it will not hang. Even more :confused:
Also (7,1) hangs ; (7,2) does not and (8;2) does. So the issue seems to be (n1-n2)>=6 when using SelectRangeEvery(n1,n2). If I encode the DVD-RB indexed file with Trim only, reindex the output with DGindex 1.5.7 , everything is fine. I can use any SelectRangeEvery(n1,n2).

For other info, the file was preprocessed with VOBblanker ( I may have split some cells but not at this location) but I still do not understand why HC only hangs with some SelectRangeEvery (probably, SelectRangeEvery cannot return a valid frame and HC may loop) .
I was wondering if the version of DGdecode.dll could be the problem.
http://forum.doom9.org/showthread.php?p=1370451#post1370451

Thanks in advance.

Edit: It is probably the DGdecode.dll used by DVD-RB. Using DVD-Decrypter in IFO mode (no File Splitting) with the preprocessed file, HC is able to process the file with any SelectRangeEvery with the file indexed with DGindex 1.5.7. End of story.

Edit 2: The issue with SelectRangeEvery may for small values of n1-n2 be fixed with the 10 frame window of ChangeFPS as explained by IanB (way above my head).
http://forum.doom9.org/showthread.php?p=1370823#post1370823
It is possible that the MPEG2Source used by DVD-RB is not completely frame exact (and that my source file was not linear in one way) but HC indirectly was able to find that.
http://forum.doom9.org/showthread.php?p=1370788#post1370788
It may be an issue with 1-pass VBR because I suspect that HC expects a frame exact source.
Edit3: Surprise,surprise: Was able to get the "hang" on the unprocessed DVD (straight from the rip).The movie is 29.97 fps (for DGindex 1.5.7) .

Emulgator

14th February 2010, 18:51

I still owed an answer to manolito:
Would you be so kind to make some side by side tests using HC,
once turning Deadzone Quantization off and once have it turned on in automatic mode, all other settings being identical?

I am asking you because I did a couple of tests this way,
but on my equipment (CRT TV) I really could not detect even the slightest difference between the two.

Now that I got back to my old test source, I could make a encode comparison
between a straight version (2-Pass VBR 5000kbps, 800 frames), deadzones=off vs. deadzones=auto
and a GradfunDB regrained version (2-Pass VBR 5000kbps, the same 800 frames) deadzones=off vs. deadzones=auto.

I found the same as manolito did, in both cases, straight and GradfunDB:

No visible differences between deadzones off and auto under my TFT testing conditions,

So it looks as if qualitywise deadzones=auto is safe to use instead of deadzones=off.

Filesizes came out identical to the single byte, bitrate distribution identical as well,
so HC 0.24 beta 27-11-2009 must have picked the same quantization for auto as for deadzones (0,0).

File size identical... this smelt a bit suspicious to me.
Because some of the settings in 0.24 GUI are not taken over on encoding time,
(you have to save the .ini first, close HC and reopen to be safe),
I repeated the encodes, but results stayed the same.

hank315

17th February 2010, 00:47

New HCenc024beta (http://hank315.nl/files/HC024_beta_16-02-2010.zip)

- 1pass VBR bugs fixed
- asm optimizations, a bit more speed
- some cosmetics

Also there's a special version included which doesn't use CPU dispatching which runs 2-3 % faster on my Intel Q9450.
Any CPU which can do SSSE3 will run it, it might also speed up AMD systems, Intel compiler options: /O3 /Og /QaxT /QxT
(There are rumours CPU dispatching by Intel compilers doesn't do that well on AMD :) ).
I don't have an AMD system at home or at work so I can't test it.

hydra3333

17th February 2010, 01:01

Thanks ! What is CPU dispatching and will I miss out on anything by using it (I also have a q9450).

edit: oh. http://www.agner.org/optimize/blog/read.php?i=25

hydra3333

17th February 2010, 11:40

Hmm, thanks for trying.
*INFILE D:\DVD\test.avs
*OUTFILE D:\DVD\test.mpv
*LOGFILE D:\DVD\test-HC.log
*PROGRESSIVE
*AVSMEMORY 256
*SMP
*1PASS
*LOSSLESS
*DBPATH G:\TEMP
*LLPATH G:\TEMP
*PROFILE BEST
*ASPECT 16:9
*LASTIFRAME
*AUTOGOP 15
*CLOSEDGOPS
*PULLDOWN
*DC_PREC 10
*MATRIX FOX1
*AQ 0
*BIAS 75
*LUMGAIN 1
*BITRATE 9200
*MAXBITRATE 9400
*CQ_MAXBITRATE 2
"C:\software\HC\HC024beta\HCenc_024_QaxT.exe" -ini "D:\DVD\test-HC.ini"
Yielded a windows error window "MPEG2 encoder has encountered a problem and needs to close ..."

(edit: i'd put in the *1PASS as a new option because i'd just spotted it, so it's my fault) It seemed to spend a minute or so "Sampling", before crashing, which I've never seen before and could potentially be very long for a glacial script ( even on a q9450 :) ). Commenting out the *LOSSLESS made no difference to crashing. Now I have to undo updating about 10 scripts ...

edit: same script and .ini (with and without lossless) worked just fine in v023 with and without any "Sampling" message thing whatever that was.

edit2: took out the *1pass and the "Sampling" message thing went away and it appears to be working. Looks like I don't understand what *1PASS does as compared to *CQ_MAXBITRATE which it appeared to over-ride. Any hints on the difference ?

PS nice work, with HC in videoredo :)

Emulgator

17th February 2010, 14:30

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.

This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4

What do you think ?

Gromozeka

17th February 2010, 18:19

hank315
Thanks, may be you add yuv input in next release :)

hank315

17th February 2010, 18:50

An enhancement wish for 8mm silent movie encodes and the like:

Would it be possible to add 3:3 pulldown
for progressive encodes to the pulldown settings ?

I'd like to experiment with the unusual, but possible.
8mm film, usually shot around 12..18 fps, frame scanned.
Assumefps (fps=16.66666666) (fps=50,3)

Encoded as progressive frames, to be presented in PAL mode on PAL devices
by applying 3:3 pulldown on this 16.67p footage.

To have playback as PAL DVD with 3:3 pulldown,
flag pattern should be tff=1,0 alternating, rff always =1.

tff=1,rff=1 for frame 0 and all consequent even numbered frames,
tff=0,rff=1 for frame 1 and all consequent odd numbered frames.

This would serve Film 16.67p on PAL 25i
and would as well serve for Film 19.98p on NTSC 29.97i
(besides maybe adding 2:3, 3:2:2:3, 2:3:3:2 to the already available 3:2 for Film 23.976p on NTSC 29.97i,
and Euro-pulldown 2:2:2:2:2:2:2:2:2:2:2:3 for Film 24p on PAL25i)
This is easy to implement but DGPulldown can do all this AFAIK.

The other silent movie pulldowns would take more repeats than rff can provide:
* 16 fps (actually 15.985) to NTSC 30 fps (actually 29.97): pulldown should be 3:4:4:4
* 16 fps to PAL 25: pulldown should be 3:3:3:3:3:3:3:4
* 18 fps (actually 17.982) to NTSC 30: pulldown should be 3:3:4
This can be done with progressive pulldown, repeating frames instead of mixing fields.
Progressive sequence = 1, TFF = 0
RFF = 0 play frame 1 times
RFF = 1 play frame 2 times

16 --> 25: 1:2:1:2:1:2:1:2:2:2:1:2:1:2:1:2
16 --> 30: 1:2:2:2:2:2:2:2:1:2:2:2:2:2:2:2
18 --> 30: 1:2:2:2:1:2:1:2:2:2:1:2:1:2:2:2:1:2

Don't know if HW/SW players actually support this.

Emulgator

17th February 2010, 19:38

Ah, thank you for pointing me to that!

manolito

17th February 2010, 20:43

A big THANKS for the new beta!

Already tested it with the problematic source from this post: http://forum.doom9.org/showthread.php?p=1365549#post1365549
and the artifacts are gone.

So whatever changes you applied to 1-pass VBR mode, they sure did the trick.

Thanks again
manolito

mikenadia

18th February 2010, 02:33

1-pass VRB with 30 000 frames (sample size is not an issue).
1 pass CQ=9 (usual for the Extras of a DVD) with ip_ratio=1 and ib_ratio=1.2 leads to a 1600 kbps bitrate.
Did one pass VBR with 1600kbps . Got 1601 kbps.
Did the same thing with 1530 kbps (the max bitrate for which I have the warning "Very high quantizer"). Got 1340 kbps (the initial Q jumps by more than 25 % between 1530 kbps (warning) and 1540kbps (no warning).
Also, it is very sensitive to "sampling".
10 031 frames (1899 kbps for a targeted 2000 kbps : no High quantizer warning: initial Q:10.22).If I add 1 frame ,10 032 frames. right on target, 1993 kbps :initial Q:9.73).It looks it is assuming that the initial Q of the sample is the right one and does not adjust to it if the encoding shows differently ( if a coin always goes to "tail", after N trials, you have to believe the coin is biaised and bet accordingly even if your initial assumption was that the coin was not biased).
And the issue is bitrate dependent. If I increase bitrate to 4000 kbps, both clips (10 031 and 10032 frames) give me undersized enodes (both around 3525 kbps).On the 30 000 frames, no problem at 2000 kbps and 4000 kbps.

Edit: I do not know how sampling is done but rb-opt is doing it with
SelectRange Every(299,15) for 5% sampling (15 could be replaced by Max GOP length). This might enable them to estimate also ip_ratio and ib_ratio and may be to have a more accurate initial Q (when encoding with 1-pass VBR, I can often see 25% variation in I-frame quantizer).

Edit 2: a pause and resume during sampling phase leads to negative and then above normal average fps.

txporter

19th February 2010, 18:42

Question about re-encoding 1080i MPEG2 captures: I am seeing low CPU utilization (~30-40%) when re-encoding 1080i captures to remove the pulldown and resize to 1280x720. Is this due to decode performance? I don't think I see this with 720p material, but I haven't looked really hard there because I was getting ~2X the encode speed (~11 fps for 1080i and 22-23 fps for 720p). This is on a Q6600, 4gb ram, Vista64. My avs files are pretty simple. I am encoding at constant quant with the latest 0.24beta.

1080i
loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test.d2v",cpu=3)
TFM().TDecimate()
Undot()
Bicubicresize(1280, 720)
TextSub("test.srt")

720p
loadplugin("Path\To\DGDecode.dll")
LoadPlugin("Path\To\Undot.dll")
loadplugin("Path\To\TIVTC.dll")
loadplugin("Path\To\VSFilter.dll")

MPEG2Source("test2.d2v",cpu=3)
SelectEven().TDecimate()
Undot()
TextSub("test2.srt")

I have tried the 1080i with setmtmode 1 and 2. I see maybe 5-10% improvement in encode speed, but I am still hovering between 30-40% cpu utilization. What am I doing wrong? Where to look to increase the utilization and hopefully framerate?

Boulder

19th February 2010, 19:25

Add Distributor() as the last item in your script (if using SetMTMode())

txporter

19th February 2010, 20:22

Add Distributor() as the last item in your script (if using SetMTMode())

Ok, will try that. I totally missed that before! Do I need to call the MT plugin for that, or is that an internal filter for the MT avisynth.dll?

update: Ok, I confirmed on CoreDuo T2400 laptop that I was doing absolutely nothing with my setmtmode call without distributor(). ;) (Thanks, Boulder) CPU utilization on dual core was already fairly close to max'd, so might see more improvement on quad core at home. Also, setmtmode(1) crashes with my script.

Dual core numbers:
1080i-->1280x720 no setmt - 5.1 fps (~70% cpu util.)
1080i-->1280x720 setmt2 - 5.4 fps (~70-75% cpu util.)
1080i-->no resize no setmt - 4.5 fps (~90% cpu util.)
1080i-->no resize setmt2 - 4.8 fps (~90-95% cpu util.)

Also, am I right that I should resize after IVTC and other filters? Or should I resize before all of that?

Boulder

19th February 2010, 21:19

It's Avisynth's internal functions so no need for mt.dll.

hank315

19th February 2010, 22:20

@mikenadia
To sample a (15 frame) GOP, 3 frames are encoded: one I, one P and one B-frame, the total GOP size is then calculated:
GOPsize=Isize+4*Psize+10*Bsize
The amount of sampling depends on the movie size but it is small, for a 10000 frame clip, 100 frames are actually encoded.
Sampling is done for Qvalues 2.0, 5.0 and 10.0, after that a second order polynomial is fitted on this data to determine the final Qvalue which will (theoretically) hit the desired bitrate.
With this Qvalue a compression curve is created (stage 2), this curve is smoothed and scaled.

So it's important the Qvalue lies between 2.0 and 10.0 otherwise the value will be extrapolated from the fitted data.

Edit 2: a pause and resume during sampling phase leads to negative and then above normal average fps.
Cool, people always try things I didnt test :)
Think I will simply gray out the pause button during sampling.

manolito

20th February 2010, 14:32

I did three real world encodes (movies from 90 min to 130 min length) with the current 0.24 beta in 1-pass VBR mode, and sizing was right on the money every time. Great quality, too :)

As I already said, my problematic source with very high bitrate requirement towards the end of the clip now comes out perfectly. Would you be willing to explain in a little more detail what exactly you did to fix 1-pass VBR mode?

Cheers
manolito

Brazil2

21st February 2010, 13:19

I see a lot of people are using the *CLOSEDGOPS option.
Is it required for making a DVD ?

Amefurashi

21st February 2010, 13:20

Sorry for the noob question, but I was not able to retrieve a clear answer from Google. Why does HCgui_024 is not able to find DGDECODE.DLL even if I put it in HCenc's folder? BTW, when I open the gui, the DLL file "vanishes" from the folder.

:confused:

nevragain

21st February 2010, 16:04

I see a lot of people are using the *CLOSEDGOPS option.
Is it required for making a DVD ?

Closed GOP is required for muliangle DVD if I recall correctly. Open gop is fine for regular DVDs. I'm reencoding a dvd right now with open GOPs.

hank315

21st February 2010, 17:33

@Brazil2
No, that's not required.
Indeed closed gops is required for multi-angle DVDs, also it makes editing easier, you can make a cut at each I-frame.
You will have a small quality degradation using closed gops because the compression is a bit less (less B-frames in a GOP).

@Amefurashi
Because there are different DGIndex/DGDecode versions, HCGui always starts deleting the DGDecode.dll in the HC directory, then reads the D2V file and copies the right version from the DGDecode directory.
After the encode is done the DGDecode.dll is deleted.

Brazil2

21st February 2010, 19:09

Closed GOP is required for muliangle DVD if I recall correctly. Open gop is fine for regular DVDs.
No, that's not required.
Indeed closed gops is required for multi-angle DVDs, also it makes editing easier, you can make a cut at each I-frame.
You will have a small quality degradation using closed gops because the compression is a bit less (less B-frames in a GOP).
OK thanks for the info, and phew I'm happy I don't have to reencode all of the DVD's I've made with HCenc (which is great, thanks for this great encoder) :)

txporter

27th February 2010, 20:59

Been doing some testing with setmtmodes and MT calls with HCenc 0.24 beta 2-16-10. I am having odd irregularities with 1-pass VBR.

If I run this script (on 1080i 6min sample clip), it takes quite a long time to go through sampling (~14-15mins) and then it hangs at the end (never starts 2nd pass, and actually I cannot kill the encode with the exit button either...need to kill in Task manager).
setmtmode(5)
MPEG2Source("video.d2v",cpu=3)
setmtmode(2)
TFM()
TDecimate()
Undot().FluxsmoothT(5)
setmtmode(5)
MT("LSFMOD()",threads=4)
Bicubicresize(1280, 720)
TextSub("video.srt")
Distributor()

If I run that same script through 1-pass constant quant, it seems to process fine @ ~9.5fps.

If I run this script through 1-pass VBR, it completes sampling in ~3.5mins and goes on through 2nd pass fine. Encoding rate is ~8.4fps.
MPEG2Source("video.d2v",cpu=3)
MT("
TFM()
TDecimate()
Undot().FluxsmoothT(5)
LSFMOD()
",threads=4)
Bicubicresize(1280, 720)
TextSub("video.srt")

I would like to use the setmtmode version since it is faster, but it doesn't play nice with sampling. Is this expected?

HC.ini
*DBPATH I:\ToConvert
*LLPATH J:\temp
*BITRATE 4000
*MAXBITRATE 8000
*PROFILE best
*AUTOGOP 12
*AVSMEMORY 512
*INTRAVLC 2
*MATRIX hclow
*LUMGAIN 2

encoding parameters (CLI):
%hcenc% -i %~nA.avs -o %~nA.m2v -1pass -b 5000 -maxbitrate 10000 -progressive -log %~nA.log

UPDATE: The initial script runs fine in lossless mode as well. I guess however sampling is happening, it doesn't like the fact that setmtmode sends consecutive frames to different threads.

As far as lossless mode goes, it seems to use up all the resources on my computer. Everything is very sluggish when it is encoding (I do not notice this when doing 2-pass or 1-pass vbr or CQ). I am showing 90+% cpu utilitization and 95% of the physical memory used (even though HCenc_024.exe only shows using 640mb of 4GB). The encode is 52% complete for the 1st pass and the HC01.lls file is still 0kb in my lossless temp directory (GUI shows lossless file size of 10GB). Is this what everyone else sees as well when using lossless?

rcubed

28th February 2010, 10:50

As far as lossless mode goes, it seems to use up all the resources on my computer. Everything is very sluggish when it is encoding (I do not notice this when doing 2-pass or 1-pass vbr or CQ). I am showing 90+% cpu utilitization and 95% of the physical memory used (even though HCenc_024.exe only shows using 640mb of 4GB). The encode is 52% complete for the 1st pass and the HC01.lls file is still 0kb in my lossless temp directory (GUI shows lossless file size of 10GB). Is this what everyone else sees as well when using lossless?

txporter,
If you have sufficient physical drives in your system things will run faster if you place the source on one drive, the target on another, and the lossless file on another. The idea is to minimize the seeks for the reads and writes involved during the encoding process. If it is all on one drive a big cause of slow encoding is due to one single drive having to do a lot of seeks. If you have two physical drives putting the lossless on one drive and the source and target on another would probably work better. If you only have one drive it'll take longer.

Hope this helps

rcubed

txporter

28th February 2010, 17:15

txporter,
If you have sufficient physical drives in your system things will run faster if you place the source on one drive, the target on another, and the lossless file on another. The idea is to minimize the seeks for the reads and writes involved during the encoding process. If it is all on one drive a big cause of slow encoding is due to one single drive having to do a lot of seeks. If you have two physical drives putting the lossless on one drive and the source and target on another would probably work better. If you only have one drive it'll take longer.

Hope this helps

rcubed

Ok, thanks for the pointers on that. I had source and target on same drive and lossless file on another. Doesn't the lossless file ever get written out? I never saw the .lls file go above 0kb. I may just stick with 1-pass VBR with the MT calls rather than trying to deal with setmtmode. The total encode time still ends up being less than dealing with the lossless file and the computer resources are not so taxed.

Is there a way to use 1 avs for the sampling phase and a different one for the 2nd pass?

rcubed

28th February 2010, 22:19

txporter

28th February 2010, 22:48

txporter,
If you are using a My Computer window to monitor the Lossless file size you may have to do a View/Refresh on the window to get the file size to reflect the actual size. I don't remember is the filesize is only available once the sourcing program has closed (closed might not be correct term) the file or not. I would think the file size would be visible after the 1st pass of a 2 pass encode using refresh.

The lossless file is of most use if you have an AVIsynth script that is very time consuming and uses a lot of slow filters. If you have a simple .avs file to just open the file, the lossless may actually make the process run slower. I had some files that had to use the filter Mrestore. Lossless in that case made the 2nd pass significantly faster.

Perhaps Hank can shed more light.

rcubed

Yes, what you said about the file size/My computer is exactly what I thought. I refreshed it multiple times (before and after 1st pass) and never saw the file size change.

Yeah, I understand the reasoning behind using Lossless. My scripts are not too complicated (Src, TIVTC, denoise, sharpen, resize, hardsub), but I do see a 2-3X framerate improvement using it. I would rather use 1-pass VBR since it is faster still and I am not extremely concerned about final file size (not for DVD). Anyhow, thanks for your thoughts.

txporter

1st March 2010, 18:02

Hmm, apparently the setmtmode/1-pass VBR issues are not the same for my 2 core laptop as my 4-core desktop. The sampling phase can complete on the laptop. Perhaps it is due to the # of threads? Sampling with/without setmtmode(2) seems to take the same amount of time (rather than ~10X as seen on desktop machine).

Laptop is also Window XP SP3 32-bit rather than Vista64. Avisynth and plugins are the same though.

rcubed

2nd March 2010, 07:07

Anyhow, thanks for your thoughts.

txporter,
You are more than welcome.

rcubed

Revgen

3rd March 2010, 20:15

I'll check out the new version as soon as I get a chance.

AlanHK

7th March 2010, 23:58

In HC 0.24 the progress in percent shown on the toolbar often gets way out of step with the actual percent, as shown in the app window. Eg, here 20% and 40%.

Or is it now supposed to indicate the percentage of the total job -- 40% of 1st pass of 2 = 20% of job?

txporter

8th March 2010, 17:40

I had noted this also. Yes, it appears to be percentage of total on the menu bar and percentage of current in the actual window itself.

hank315

8th March 2010, 22:32

I had noted this also. Yes, it appears to be percentage of total on the menu bar and percentage of current in the actual window itself.
Yes, this was on request by mikenadia, well there's some logic in it.

soneca

8th March 2010, 23:02

I'm sorry for my English ...

The 0.23 version has always worked very well, as all older, but this latest version does not work with DVD Rebuilder, the HCenc begins encode and suddenly crashes. HCenc not respond.

I use 32-bit Windows 7 + i7 920 + 3GB ram

Any idea what might be happening?

Emulgator

31st March 2010, 21:25

A little wish to hank315, easy to fulfill:
Can we have UserData string editable or at least empty ?
At the moment I am running HC streams through Restream to get rid of UserData,
I have to do a Closed Caption remux while authoring and fear DVDLabPro2 may block CC muxing as long as Private stream contains UserData.
This behaviour of DLP has been confirmed by others.

manolito

31st March 2010, 22:25

Just adding *NOUSERDATA to HC.ini will do the trick...

Cheers
manolito

Emulgator

31st March 2010, 23:24

Many thanks, manolito !

Edit: Unfortunately this does not work here with 0.24 beta 16-02-2010.

*NOUSERDATA is not represented in ini window.
If I edit manually in .ini using Editor, HC removes this line from HC.ini. Hm.

Mediainfo still shows |HCEnc 0.24.0.0 - (c) 2004/2010|

Emulgator

31st March 2010, 23:45

Another finding with 0.24 beta 16-02-2010:
If I bring progressive footage and rely on HC to find the proper progressive/interlaced setting and scan order
HC throws sometimes progressive, sometimes interlaced frames and scanorder also flips with this if I can believe the framelog.

Only if I force progressive I get 100% progressive frames encoded now.
This had worked before nicely in 0.24 beta 2009 10 20 and 2009 11 27.
Can anybody repeat this ?

Edit: Now after scrutiny the bug seems to dissolve,
now it works as it should, building 100% progressive frames.
I had it for the last 4..5 encodes sometimes coming up with around 5..10% interlaced frames...?

2nd Edit: No. Never mind.
My "autodetect" setting was not respected, I had not saved to HC.ini,
so HC took the shortcut and used the last setting which was forced progressive.

The flipping is still there, here is just one example.
720x576 PAL progressive, 700frames, motion menu with fly-in animation at the beginning.
The pattern comes out different everytime, here we have mostly interlaced:

frame type length enc.type % scantype intraVLC
------------------------------------------------------
0 I 39727 P 0 zigzag 14
1 P 32052 I 0 alt 14
2 B 21046 I 0 alt 14
3 P 33572 I 0 alt 14
4 B 22586 I 0 alt 14
5 B 21109 P 0 zigzag 14
6 P 35575 P 0 zigzag 14
7 B 23509 P 0 zigzag 14
8 B 23816 I 0 alt 14
9 P 43956 I 0 alt 14
10 B 25767 P 0 zigzag 14
11 B 26307 I 0 alt 14
12 P 50873 I 0 alt 14
13 B 28835 I 0 alt 14
14 B 29688 I 1 alt 14
15 I 65591 I 0 alt 15
16 P 50899 I 0 alt 15
17 B 16144 I 0 alt 15
18 P 57696 I 1 alt 15
19 B 20473 I 0 alt 15
20 B 18363 I 0 alt 15
21 P 59330 I 0 alt 15
22 B 22151 I 1 alt 15
23 B 21120 I 0 alt 15
24 P 64970 I 1 alt 15
25 B 22966 I 1 alt 15
26 B 21626 I 0 alt 15
27 P 61241 I 1 alt 15
28 B 22638 I 1 alt 15
29 B 21590 I 1 alt 15
30 I 91832 I 1 alt 15
continuing like that until end:
695 B 18781 I 1 alt 15
696 P 39473 I 1 alt 15
697 B 18252 I 1 alt 15
698 B 18357 I 1 alt 15
699 I 41558 I 1 alt 15

I remember this behaviour coming up with HC 0.23,
HC 0.22 was clean in PROG/INT decision, the earlier 0.24s were clean too.

manolito

1st April 2010, 12:26

This is from Hank himself at Doom10:
To disable the user data in HCenc, use the latest beta (16-02-2010) and add the next command in the ini file: *NOUSERDATA
I believe this will only work when calling HC directly, not from the GUI.

For the progressive / interlaced autodetect mode I noticed this flipping from the beginning. But does it really hurt if some progressive frames are encoded in interlaced mode? I was never able to point a finger on a reproduceable quality issue resulting from this behaviour. The other way around would be a real problem, so I believe that Hank wanted to play it safe and switch to interlaced if in doubt.

Cheers
manolito

txporter

1st April 2010, 17:40

hank315

3rd April 2010, 20:22

For the progressive / interlaced autodetect mode I noticed this flipping from the beginning. But does it really hurt if some progressive frames are encoded in interlaced mode? I was never able to point a finger on a reproduceable quality issue resulting from this behaviour. The other way around would be a real problem, so I believe that Hank wanted to play it safe and switch to interlaced if in doubt.
Yes, it's biased to interlaced because as you said it won't hurt that much to encode progressive as interlaced... it only takes a few extra bits.

A little wish to hank315, easy to fulfill:
Can we have UserData string editable or at least empty ?
At the moment I am running HC streams through Restream to get rid of UserData,
I have to do a Closed Caption remux while authoring and fear DVDLabPro2 may block CC muxing as long as Private stream contains UserData.
This behaviour of DLP has been confirmed by others.
Editable isn't an option because it can be screwed up pretty easy (start code emulation), I will just dump the user data.

I have also noted this flipping for all versions of 0.24beta that I have used as well as 0.23. The only reason that I noted it was because it was causing playback of HC encoded files on my Tivo to move up and down a line or two between some frames when I used autodetect (I didn't check, but it was likely due to scan type switching). It created a strobe-like effect. I simply force progressive or interlaced based on input and no longer see this effect.
IMHO the switching between progressive/interlaced frames shouldn't cause playback issues :confused:

mp3dom

4th April 2010, 00:26

Emulgator

4th April 2010, 09:07

I can confirm that this flipping is visible on certain chipsets, so dependent on player/TFT combinations.

Panasonic BD-50 -> Loewe 37" is one of those.
Occasionally appearing thin (bunch of single HD pixel wide horizontal lines where interlaced motion parts should appear)
Picture upsized by player to 1920x1080, so pointing to player chipset bug.

I believe this (*NOUSERDATA)will only work when calling HC directly, not from the GUI.
I see, I was using GUI.

manolito

4th April 2010, 12:57

Switching between progressive/interlaced frames shouldn't cause any playback issues, but switching between field/frame structure should (if I'm not wrong, the specs says that both field and frame structure need to be supported, but doesn't say if the structure could be changed in the stream during playback, so some (old) chipset shows the strobe-effect (on some it shows some kind of macroblocks). The same problem appear with ProCoder that does the same automatic field/frame structure.
But AFAIK HC is not even capable to encode with field structure. To make sure I just checked a DVD I made with HC using DGIndex, and while it does flip between progressive and interlaced all the time, the structure indicator is frozen at "FRAME".

Cheers
manolito

Boulder

4th April 2010, 14:17

But AFAIK HC is not even capable to encode with field structure. To make sure I just checked a DVD I made with HC using DGIndex, and while it does flip between progressive and interlaced all the time, the structure indicator is frozen at "FRAME".

Cheers
manolito
I think the issue is due to the DVD player or TV deinterlacing the frames where interlaced encoding has been used.

txporter

5th April 2010, 17:54

IMHO the switching between progressive/interlaced frames shouldn't cause playback issues :confused:

Yeah, I didn't expect to have a problem with it (and my PC didn't care), but playback on the Tivo was affect. /shrug It works fine if I just specify prog/int, no biggie.

I posted it some time ago in this thread, but wanted to bring it up again in case you missed it. Is it possible to get HCenc to work with setmtmode when using 1-pass VBR? Or maybe use two different avs files for the sampling phase versus the 2nd pass? MT works fine with 1-pass sampling, but setmtmode takes much longer and many times will cause HCenc to crash/hang.

Boulder

5th April 2010, 18:05

You need to have Distributor() as the last item in your Avisynth script to make SetMTMode work with HC.

txporter

5th April 2010, 18:11

You need to have Distributor() as the last item in your Avisynth script to make SetMTMode work with HC.

Yes, that I understand. It works fine with lossless or 2-pass mode, it just doesn't play well with the 1-pass mode sampling.

hank315

7th April 2010, 20:51

Boulder

7th April 2010, 20:55

Yes, that I understand. It works fine with lossless or 2-pass mode, it just doesn't play well with the 1-pass mode sampling.Perhaps the sampling part of the script (I assume HC uses Avisynth to create the sample clip) should be moved after Distributor(). However, this is something that must be done inside HC.

Chumbo

8th April 2010, 00:48

soneca

8th April 2010, 22:00

The same happens here with Windows 7 32bit.

Emulgator

8th April 2010, 22:26

BTW, here HC 0.24 beta 16-02-2010 in 32-bit and 64-bit version
worked without problems on a Win7U64 with i7-920XM CPU.

txporter

8th April 2010, 22:43

New HCenc024beta (http://hank315.nl/files/HC024_beta_04-04-2010.zip)

As usual some bug fixes, some changes to the 1pass VBR mode and user data is removed.
The TFF/BFF flag can now be set on progressive encodes, also Progressive Sequence can be set.

I tested the new beta with using setmtmode and 1-pass VBR. It seems to work now! One question though, is Distributor() no longer needed? If I use Distributor(), I get an immediate crash of HCenc. If I do not, it runs fine. I tested with and without the setmtmodes in the avs file and it does appear to result in a higher encode speed even without Distributor() [4.7fps with setmtmodes and 3.3fps without].

Chumbo

9th April 2010, 02:29

BTW, here HC 0.24 beta 16-02-2010 in 32-bit and 64-bit version
worked without problems on a Win7U64 with i7-920XM CPU.
HC works just fine. It's just that it "crashes" for whatever reason when the encoder finishes and closes which means I can't run multiple batch commands one after the other. The same with the GUI.

One of my computers is the same configuration as you mention but this problem is consistent on all my Win7U64 on the i7 920, 950 Quad-Core and my Core Duo P8800 notebook.

hydra3333

10th April 2010, 00:47

Hi Hank,
http://forum.doom9.org/showthread.php?p=1390290#post1390290 and
http://forum.doom9.org/showthread.php?p=1390298#post1390298
mentions what could be a bug with HCenc when using avisynth MT0.7 and Neuron2's DGMultiSource. HCenc crashes fatally, mostly.
Seems suggestive of an initialisation type bug.
Cheers

Emulgator

10th April 2010, 11:52

Hank,new 0.24 beta :As usual some bug fixes, some changes to the 1pass VBR mode and user data is removed.
The TFF/BFF flag can now be set on progressive encodes, also Progressive Sequence can be set.
And many thanks, Hank !
A second donation from me is coming soon.

manolito

10th April 2010, 19:17

New HCenc024beta (http://hank315.nl/files/HC024_beta_04-04-2010.zip)

As usual some bug fixes, some changes to the 1pass VBR mode and user data is removed.
The TFF/BFF flag can now be set on progressive encodes, also Progressive Sequence can be set.

Thanks very much for this new version...
Some findings / questions:

I just finished an encode in 1pass VBR mode (it was this problematic source where I got major blocking towards the end of the clip with earlier 0.24 versions), and it came out perfectly. One thing I noticed was that the latest version takes almost twice as long for the sampling phase compared the the previous version. Higher percentage of frames used for sampling and creating the compression curve?

And a question concerning the flagging for progressive streams:

Previous versions of HC (as well as QuEnc) would flag a progressive encode with "Frametype Progressive" and would also set the "Progressive Sequence" Flag. The TFF flag was not (and could not be) set.

CCE SP on the other hand would flag a progressive encode as "Frametype Progressive", but the "Progressive Sequence" flag is not set. But the TFF flag is always set for a progressive encode.

The latest HC beta now gives the user the freedom to use all possible combinations for flagging a progressive encode. My question now is if there is any advantage to set the "Progressive Sequence" flag in addition to the "Frametype Progressive" flag. Do decoders care at all? I never had any issues with DVDs created by CCE where the "Progressive Sequence" flag is never set.

Cheers
manolito

Emulgator

11th April 2010, 09:54

I found "Progressive Sequence not set" to be an advantage for certain DVD player/TFT combos.
CCE streams would play fine in such combos, whereas HC streams, although encoded properly, would give a brief flicker sometimes.
Reflagging HC to "Progressive Sequence not set" using ReStream helped in such cases.
I used it ever since on HC and was fine.
Others may not encounter any problems at all regardless of setting, but I preferred to do as CCE does from then on.

sneaker_ger

11th April 2010, 13:12

How do I have to set the new HC to act in the same way as CCE? "progressive + TFF" or is there anything else? It kinda irritates me having to set field order on progressive streams.

manolito

11th April 2010, 13:39

To act like CCE SP you must specify *PROGRESSIVE and *TFF for a progressive encode.

But you do not have to set field order for a progressive encode. AFAIK the standard says that the TFF flag has no meaning whatsoever for a progressive stream, decoders should ignore it.

The reason why I like to have the TFF flag set even for a progressive encode is that I convert a lot of TV captures where the movie itself is progressive, but the end credits are often interlaced TFF. Since I am too lazy to convert the credits separately it comes in very handy if the encoder sets the TFF flag. If it doesn't (like previous versions of HC or QuEnc) then these end credits will be displayed with the wrong field order which looks terrible.

Cheers
manolito

sneaker_ger

11th April 2010, 13:52

But you do not have to set field order for a progressive encode. AFAIK the standard says that the TFF flag has no meaning whatsoever for a progressive stream, decoders should ignore it.

Well, the new HC requires you to choose a field order. Standard is BFF.

The reason why I like to have the TFF flag set even for a progressive encode is that I convert a lot of TV captures where the movie itself is progressive, but the end credits are often interlaced TFF. Since I am too lazy to convert the credits separately it comes in very handy if the encoder sets the TFF flag. If it doesn't (like previous versions of HC or QuEnc) then these end credits will be displayed with the wrong field order which looks terrible.

If your setup separates/deinterlaces frames flagged progressive something is wrong to begin with, isn't it? :confused:

manolito

11th April 2010, 15:38

Well, the new HC requires you to choose a field order. Standard is BFF.Wrong. The new HC does not require you to choose a field order for progressive encodes. There is nothing like a BFF flag, only a TFF flag exists which can be set or not set. For progressive encodes this flag is not set by default. So just do not specify *TFF in HC.ini, and everything will be like it used to be in previous versions.

If your setup separates/deinterlaces frames flagged progressive something is wrong to begin with, isn't it?
Wrong again. I watch my movies through a standalone DVD player connected to a CRT TV. By definition this TV can only display fields, not whole frames. So the DVD player does indeed separate fields all the time, regardless if the content is progressive or interlaced. Even if you have an LCD TV which does display whole frames, as long as your standalone DVD player is connected to this TV through an analog connection, the player will separate the fields and the TV will have to rebuild the frames (and deinterlace if necessary).

Cheers
manolito

sneaker_ger

11th April 2010, 16:16

Wrong. The new HC does not require you to choose a field order for progressive encodes. There is nothing like a BFF flag, only a TFF flag exists which can be set or not set. For progressive encodes this flag is not set by default. So just do not specify *TFF in HC.ini, and everything will be like it used to be in previous versions.

Ah, I see. I only looked at the GUI options. And it seems to not set "*BFF" in the ini if you select BFF.

Wrong again. I watch my movies through a standalone DVD player connected to a CRT TV. By definition this TV can only display fields, not whole frames. So the DVD player does indeed separate fields all the time, regardless if the content is progressive or interlaced. Even if you have an LCD TV which does display whole frames, as long as your standalone DVD player is connected to this TV through an analog connection, the player will separate the fields and the TV will have to rebuild the frames (and deinterlace if necessary).

Ana..what? ;) Didn't think of that - you're right again.

hank315

11th April 2010, 21:47

I just finished an encode in 1pass VBR mode (it was this problematic source where I got major blocking towards the end of the clip with earlier 0.24 versions), and it came out perfectly. One thing I noticed was that the latest version takes almost twice as long for the sampling phase compared the the previous version. Higher percentage of frames used for sampling and creating the compression curve?
Yes, higher sample rate:

- old code -
perc=0.5
if (nframes < 200001) perc=0.6
if (nframes < 50001) perc=0.7
if (nframes < 20001) perc=0.8
if (nframes < 12001) perc=1.0
if (nframes < 6001) perc=1.5
if (nframes < 3001) perc=2.0

- new code -
perc=0.8
if (nframes < 200001) perc=1.0
if (nframes < 50001) perc=1.2
if (nframes < 20001) perc=1.5
if (nframes < 12001) perc=2.0
if (nframes < 6001) perc=3.0
if (nframes < 3001) perc=4.0

I'm thinking of setting the TFF flag by default, also on progressive sources.
If the playback device decides to separate the fields it's better to have TFF set.
Only for Digital Video source it will be BFF by default.

ATM I'm migrating from XP 32bit to WIN7 64bit so I can run tests with it soon.

Emulgator

12th April 2010, 14:25

I'm thinking of setting the TFF flag by default, also on progressive sources.
If the playback device decides to separate the fields it's better to have TFF set.
Only for Digital Video source it will be BFF by default.

I support that.

blanco

26th April 2010, 10:44

JoeH

26th April 2010, 16:41

When encoding using HCEnc on a Core i7 overclocked to 3.8GHz I usually only get about 20% processor usage, with speed around "realtime". Is this normal?

When encoding DVDs with TMPGEnc Authoring Works I tend to get around 80% CPU usage, with 4x realtime speed.

If this is normal for HCEnc, I would like to add a feature request to improve Core i7 support. If not, any ideas to improve encoding times and up that processor usage? Thanks!

um3k

26th April 2010, 19:20

Are you doing any processing in AviSynth? If so, that could be slowing down the process, as regular AviSynth is not multithreaded. There are mods to make it so that you might want to look in to.

soneca

26th April 2010, 23:19

Here operates at full
i7@2.66 with DVDRebuilderPro + HCenc 0.23

http://i131.photobucket.com/albums/p305/soneca1/Forum/ram.png

hank315

27th April 2010, 22:06

-Filesize input at the command line option does not work properly.
I loaded a 60000 frames videofile and set desired filesize at 960000. HCenc calculated an average bitrate of 270 kbs!

If i encode 5001 frames of the same video and set desired filesize at 90000 kilobytes, HCenc does calculate and encode at the right bitrate (+2000 kbs).

Is it a bug or did I do something wrong?
I can replicate it, it's certainly a bug, filesize > 262143 will screw up.
Will be fixed in the next release, thanks for the report.

JoeH

28th April 2010, 17:00

Are you doing any processing in AviSynth? If so, that could be slowing down the process, as regular AviSynth is not multithreaded. There are mods to make it so that you might want to look in to.

I was using it with MultiAVCHD, which does in fact use AviSynth.

Here operates at full
i7@2.66 with DVDRebuilderPro + HCenc 0.23

OK - that's good to know it's not a limitation of HCEnc. I'll try to solve it by other means then.

Koutsoubos

2nd June 2010, 06:38

I've been observing this in a short test:

Source: 25fps PAL true progressive.
Aim: to make a 25fps PAL DVD preserving the progressive frames

1st Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=off)
This is what is advised as CCE-like output. Plays fine on interlaced PAL displays. On two progressive-scan PAL players (different brands), with progressive PAL displays (different brands), very obvious combing artefacts. Perhaps players are wrongly assuming from TFF flag that frames are interlaced?

2nd Attempt (Progressive Frame=On; TFF=on; Progressive Sequence=on)
Same problem. It seems that the TFF throws the tested progressive scan models off.

3rd Attempt (Progressive Frame=On; TFF=off; Progressive Sequence=on)
Now displays are just "normal". Note that this is the old HCEnc and QuEnc behaviour with "progressive" encodes. It seems that Procoder also behaves this way: http://forum.doom9.org/showthread.php?p=1061653#post1061653

But ... it is said that Progressive_Sequence=On is out of spec for DVD??? http://forum.videohelp.com/threads/234894-How-to-get-progressive-ouput-using-cce-sp?p=1378458&viewfull=1#post1378458 [Note: I cannot find such a remark in the current version of the DVD FAQ]

In another discussion about menus, users are advised to set Prog_frame and Prog_seq both to ON even though these are "theoretical spec violations"?? http://www.dvdafteredit.com/node/1433

Oh, what is happening and how does one make a proper 25fps progressive DVD?

Koutsoubos

3rd June 2010, 05:17

Actually, is it actually known as a fact that progressive_sequence=ON is out of spec in DVDs? (Of course those who really know are under NDA)

See here, where a Progressive_sequence problem is flagged for the reason: "progressive_sequence shall be 1 when the vertical_size equals 240": www.ip.philips.com/download_attachment/4119/4119.pdf

Guest

3rd June 2010, 05:41

The 1st attempt should work. Maybe you have a field shift in your progressive PAL as inputted to HCEnc.

Koutsoubos

4th June 2010, 14:25

Regarding 1st attempt: I think there is no field shift because the source is absolutely progressive 25fps PAL. Nothing is interlaced.

Guest

4th June 2010, 14:39

Post a link to an unprocessed source sample.

Koutsoubos

4th June 2010, 16:47

The source is big. But anyway, it is absolutely 25fps progressive AVI. No mistake there.

The inquiry should be about the flags now, as to what is permissible under DVD

Guest

4th June 2010, 17:39

You can direct stream copy a small sample.

Also, give a sample of the encoded M2V.

Anyway, no samples, no help from me.

hank315

4th June 2010, 23:53

@Koutsoubos
This table can be found on the web, the "Pulldown Table of Truth"

nr prog prog pic TFF RFF
seq frame struct

1 0 0 Field 0 0 First coded field displayed first (TB or BT)
2 0 0 Field 0 1 Illegal combination
3 0 0 Field 1 0 Illegal combination
4 0 0 Field 1 1 Illegal combination
5 0 0 Frame 0 0 Bottom first, 2 fields displayed (BT)
6 0 0 Frame 0 1 Illegal combination
7 0 0 Frame 1 0 Top first, 2 fields displayed (TB)
8 0 0 Frame 1 1 Illegal combination
9 0 1 Field 0 0 Illegal combination
10 0 1 Field 0 1 Illegal combination
11 0 1 Field 1 0 Illegal combination
12 0 1 Field 1 1 Illegal combination
13 0 1 Frame 0 0 Bottom first, 2 fields displayed (BT)
14 0 1 Frame 0 1 Bottom first, 3 fields displayed (BTB)
15 0 1 Frame 1 0 Top first, 2 fields displayed (TB)
16 0 1 Frame 1 1 Top first, 3 fields displayed (TBT)
17 1 0 Field 0 0 Illegal combination
18 1 0 Field 0 1 Illegal combination
19 1 0 Field 1 0 Illegal combination
20 1 0 Field 1 1 Illegal combination
21 1 0 Frame 0 0 Illegal combination
22 1 0 Frame 0 1 Illegal combination
23 1 0 Frame 1 0 Illegal combination
24 1 0 Frame 1 1 Illegal combination
25 1 1 Field 0 0 Illegal combination
26 1 1 Field 0 1 Illegal combination
27 1 1 Field 1 0 Illegal combination
28 1 1 Field 1 1 Illegal combination
29 1 1 Frame 0 0 1 prog frame displayed
30 1 1 Frame 0 1 2 progressive frames displayed (illegal in MP@ML)
31 1 1 Frame 1 0 Illegal combination
32 1 1 Frame 1 1 3 progressive frames displayed (illegal in MP@ML)

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

qyot27

5th June 2010, 13:38

Using the 04-04-2010 beta, is the TFF flag being set by default on progressive encodes? ReStream seems to think so, at any rate. If it is, how can I turn it off? My .ini does not include the *TFF option - I actually used the *PROG_SEQ option but the resulting encode didn't have that flag set, again according to ReStream. This is why I'm thinking it's flagging for TFF and then ignoring the PROG_SEQ parameter.

I ask, because I was encoding 25fps material and then using DGPulldown for 25->29.97. On playback in MPC, the files had line jumping issues that went away if I opened the .m2v in ReStream, set the Progressive Sequence header, and then remuxed.

hank315

6th June 2010, 00:05

No, the 04-04-2010 beta doesn't set TFF by default.
But if the stream has been processed by DGPulldown, some frames will have the TFF set, it's just the way pulldown works.
I don't know how Restream handles pulldowned streams.

In HCenc you can always disable TFF by using the *BFF command, this will clear the TFF flag.

Koutsoubos

8th June 2010, 04:23

This would mean your 1st attempt should be OK (nr 15).
Your 2nd attempt is illegal (nr 31).
The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

Thanks for replying. I don't know much about the "Pulldown Table of Truth". This is the first time I have heard of it.

The 1st attempt is HCenc's current behaviour (granted that the TFF has to be set manually). This gives Progressive_Frame=ON; TFF=ON; Progressive_Sequence=OFF

This "legal" attempt is what's giving artifacts on my progressive setups.

The 3rd attempt was HCenc's default behaviour prior to the 04-04-10 version. Previous versions would give Progressive_Frame=ON; TFF=OFF; Progressive_Sequence=ON.

The 3rd attempt, albeit "illegal", is displaying fine on my progressive and interlaced setups !?

Prior to 040410, the 3rd attempt would also be the output if I clicked on the "ensure DVD compliance" option.

[Bear in mind that my source is absolutely kosher 25fps PAL progressive. I don't know about NTSC or pulldowns, which may involve other conventions or rule-sets.]

1) The $64000 question is, what set of flags should I adopt? A lot will have to depend on whether Progressive_sequence=ON is allowed in DVD MPEG2 spec. We have only very obscure and indirect references. Frankly I have my doubts and would really appreciate something more direct and authoritative.

2) This comment from DVD FAQ #3.4:
In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?

qyot27

8th June 2010, 12:38

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.

2) This comment from DVD FAQ #3.4:
In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays. In other words, the player doesn't "know" what the encoded rate is, it simply follows the MPEG-2 encoder's instructions to produce the predetermined display rate of 25 fps or 29.97 fps.

The $128,000 question. Can I feed the 25fps source into HCEnce as 24fps, so that HCEnc adds the 2:2: pulldown flags for PAL encoding? What would the flags be, or maybe I should ask what flags I should activate then?
2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.

Koutsoubos

9th June 2010, 03:21

Yeah, my previous inquiry as to the TFF flags and PROG_SEQ option were wrong. I re-did some of the tests and found out that what I was seeing was related to the fact that pulldown sets TFF flags normally - in specific, it's that DGPulldown removes the PROG_SEQ header when performing pulldown. I didn't think of that possibility before. So I'm thinking that it's just my software MPEG-2 decoder that has a problem with that sort of pulldown, although manually respecifying the Progressive sequence flag doesn't cause a problem on my DVD player.

2:2 pulldown isn't used on 24fps footage. It's used on 25fps footage. As it said in the FAQ, "or 2-2 pulldown (with resulting 4% speedup)" - the speedup has to be performed prior to the pulldown, meaning you just speed up 24fps to 25fps and then the pulldown happens. And if my interpretation of Wikipedia's Telecine article is correct, 2:2 pulldown contains no flagging because it's simply a regular conversion of a progressive stored frame into two interlaced fields. The exact same thing happens for progressive 29.97fps footage in NTSC regions.

Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?

qyot27

9th June 2010, 19:33

Your setup is different. It seems you're doing a PAL --> NTSC conversion. Mine is PAL all the way, even to DVD. So I'm wondering about the flags and all, what is permissible, because it seems Prog_Seq=ON solves the problems I am observing. It could well be that what applies to your workflow does not apply to mine.
Well, that part was just an update on my previous post, I wasn't directing it in my quote.

2:2 pulldown etc. This is "black magic" to me. The passage from DVD-FAQ sounded as if Repeat_field flags are added to individual frames or something. Are you saying that no flags are added, and that the decoder should be able to repeat frames on its own?
No frames would be repeated at all, no flags would be set (again, from how I understand it; I don't claim to be any kind of expert on pulldown flagging). The player would manually interlace the progressive frames on the fly, turning each frame into two fields and giving them to the display interlaced. The point being, if it was 24 (or rather, 23.976) fps, then it would undergo 3:2 pulldown, which involves the spreading of fields delayed/repeated over time so as to make it compatible with 29.97 fps (or 25 fps, in the case of Euro pulldown which is 2:2:2:2:2:2:2:2:2:2:2:3). In order to make 23.976fps Film content compatible with 25fps for PAL, the traditional method involves speeding up the video by 4% and adjusting the audio pitch so that things aren't out of sync. Then the pulldown is applied just like it is to content produced natively for 25fps. You can see what 2:2 pulldown actually does in diagram form on the Wikipedia page for Telecine (http://en.wikipedia.org/wiki/Telecine#2:2_pulldown). The difference is that - again, my understanding - 2:2 pulldown isn't actually flagged, it's assumed by the player. 3:2, Euro, 25->29.97 pulldown, these are all flagged because the input and output framerates don't match. In 2:2, the input/output does match, meaning flags aren't necessary.

Interlacing all progressive frames on playback is probably the automatic behavior of the player, so it might be able to be adjusted either A) in the DVD player's settings by forcibly setting for progressive scan if it hasn't been already or B) the progressive_sequence header being ON tells the player not to perform the interlacing, resulting in the player giving the display normal progressive frames, not interlaced ones. Even if it's not spec-compliant, it would be counted as a necessary evil.

Koutsoubos

11th June 2010, 13:59

Well, despite some potentially misleading uses of the term "2:2 pulldown" from web sources, I am now convinced that it is a nullity as stated so expertly by neuron2 here: http://forum.doom9.org/showthread.php?p=1187268#post1187268

Now, I'm hoping that someone can shed more light on the progressive_sequence flag? For a definite fact, is it or is it not a violation of spec?

If it is a violation of spec, what is a good way for me to encode and/or flag a progressive 25fps PAL source into a DVD-compliant MPEG2 stream?

Guest

11th June 2010, 14:42

You can encode progressive with progressive_sequence false. I don't understand your problem. However, I only read the last post of the thread. :)

mp3dom

12th June 2010, 00:33

2:2 pulldown on PAL land is simply the term that mean: on a 25p film (24->25 with 4% speedup) display 2 fields for every frame a.k.a. display a progressive frame in an interlaced way without field shift. Basically if you're encoding your video as interlaced, you've already made a '2:2 pulldown'. The term pulldown is erroneous here, compared to what 'pulldown' means in NTSC land.
Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.

Koutsoubos

12th June 2010, 08:02

The 3rd attempt should be OK (nr 29)

But according to the author of "DVD Demystified", MPEG2 video for DVD is limited to non-progressive sequences.

Anyway, regarding the progressive_sequence flag... if you encode your video with progressive_sequence flag on, top field off, progressive_frame on, picture structure frame (table #29) you're making a in-spec stream for PAL. If you're encoding a 24p in that way and THEN apply the pulldown (for 24p->50i with field repetition) your stream will be like table #15/#16 which is, again, in spec.

Alright, here's the point-blank question. In respect of DVD specs (not just generic MPEG2), if I encode my 25fps progressive PAL source into a stream with:

1. TFF flag off
2. Progressive frame ON
3. Progressive sequence ON

Is this within the DVD spec?
Does the "Pulldown Table of Truth" #29 mean conclusively that this is allowed within DVD specifications? I mean, this is PAL and not NTSC (note the 3:2 in the title "3:2 Pulldown Table of Truth")

mp3dom

12th June 2010, 10:09

Koutsoubos

14th June 2010, 05:23

Yes, it's within the dvd specs as long as the flags/encode are made in the proper way. Note that also picture structure must be fixed to 'frame' and not change over time. I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process). :)

Thanks for that clarification. Yes, all the frames are encoded as frames. No fields.

So, DVD FAQ #3.4: http://www.dvddemystified.com/dvdfaq.html#3.4
where it says: "MPEG-2 progressive_sequence is not allowed"

Can we debunk this FOR CERTAIN?

Also:

"In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays"

Can we lay this to rest, at least for PAL?

mp3dom

14th June 2010, 15:22

Can we debunk this FOR CERTAIN?
To say 'for certain' you must have the dvd-video specification from the DVDForum. What I can say is that I've made a lot of these dvd and no one complain about strange incompatibility with standalone players. Also, EclipseSuite (which is very reliable) validate the verification.

"In the case of 24 fps source, the encoder embeds MPEG-2 repeat_first_field flags into the video stream to make the decoder either perform 2-3 pulldown for 60Hz NTSC displays (actually 59.94Hz) or 2-2 pulldown (with resulting 4% speedup) for 50Hz PAL/SECAM displays"
Can we lay this to rest, at least for PAL?

If your source is 24p and you want to convert to PAL there are only 3 ways:
- Encode as 25p with 4% speedup
- Encode as 50i with 4% speedup (this is the 2:2 pulldown)
- Encode as 24p with 2^12:3 euro-pulldown

As for the third option, there's no progressive-scan players able to reconstruct the original 24p and display at 25p since the cadence is quite uncommon. You'll see it always as interlaced with 2 stutters every second. This is different with software dvd decoders that will output true 24p. I don't know if this happends even for dvd players with HDMI output. If the player can output 24p via HDMI it could be that the hdtv will display the original 24p... but I've never tried it.

Koutsoubos

15th June 2010, 09:03

Point One is quite settled in my opinion, and to draw it back to the topic of this thread, the Prog_frame=ON TFF=off Prog_sequence=ON was the default behaviour in HCEnc when doing progressive encodes.

I think it'd be good to have that default behaviour back? But anyway the choices are now clear.

As to the second point of 2:2 pulldown again! Apparently this is a nullity. Are there any real world applications of 50i, where 2:2 pulldown happens, when encoding from progressive sources (film or not, it has been sped up to 25fps frame-wise)?

Is this another possibility for HCEnc? 50i?

qyot27

15th June 2010, 16:30

Are there any real world applications of 50i, where 2:2 pulldown happens, when encoding from progressive sources (film or not, it has been sped up to 25fps frame-wise)?

Is this another possibility for HCEnc? 50i?
50i = 25fps interlaced. The number differs only because 50i measures the total fieldrate per second, rather than relating it in terms of how many complete frames per second - interlaced or not - are being displayed.

So yes, there are plenty - anything that broadcasts or displays interlaced material uses it.

Koutsoubos

15th June 2010, 17:07

I was responding to mp3coder's observation that with a progressive source we could choose to "Encode as 50i with 4% speedup (this is the 2:2 pulldown)".

It may seem like splitting hairs, but what I was driving at is whether that would entail interlacing the progressive source and causing the MPEG stream to be hard-encoded at 50i (i.e. picture structure = field)

If, when fed a truly progressive source, HCEnc or any other EN-coder splits the frame into two fields, it would be hard 50i encoding and then we'd probably have a case where "2:2 pulldown" is an apt description.

Of course, interlacing a progressive source is at the EN-coder level is programmmatically simple. But should we do it? Is it done this way? Perhaps. I don't know.

Right now it seems that HCenc (or any other encoder) does not do that to a truly progressive source. Instead the frames are encoded (25fps nominally) and it is the job of the DE-coder to interlace the output auto-magically. There is no "pulldown", in the strict sense of the term, by the EN-coder.

And even if such an option were available, I still think it would be cleaner to encode as progressive stream.

Any thoughts?

Koutsoubos

15th June 2010, 17:36

I'm in the PAL land and I've made about 100+ dvd titles in that way and all passes the EclipseSuite verifier (the verifier used by replication facilities just before the glassmastering process). :)

You know, I was pretty convinced. But just a few minutes ago I came across this statement:
"Although Eclipse Image Analysis is crucial, it does not check the VIDEO_TS folder for DVD Spec compliance (IFO files, VOB data structure, navigation commands, etc)."
http://www.dvdverification.com/public/119.cfm

I honestly wish that this basic information became public instead of staying subject to some hefty licence fee and NDA.

mp3dom

15th June 2010, 18:17

It may seem like splitting hairs, but what I was driving at is whether that would entail interlacing the progressive source and causing the MPEG stream to be hard-encoded at 50i (i.e. picture structure = field)

Picture structure field is not the only choice for encoding interlaced. Very often, for interlaced encoding the picture structure is still 'frame'. Generally, the field structure is optimized for high-motion interlaced scene, whereas the frame structure work best for progressive source or moderate-motion interlaced scene. Frame structure is also more compatible. You can also mix the structure on a per-frame basis (so use frame structure for normal motion and switch to field when the encoder detects high-motion). The latter method is quite incompatible since it's a situation not covered by the specs (the specs says that both field and frame structure must be supported by mpeg decoder but doesn't says anything regarding the fact that the structure could be switched over time on the same segment. So some player support it, and other doesn't).
Returning in topic, if you encode your progressive source as interlaced (top field, frame or field structure, alternate scanning order etc etc), you're making a '2:2 pulldown encode'.

If, when fed a truly progressive source, HCEnc or any other EN-coder splits the frame into two fields, it would be hard 50i encoding and then we'd probably have a case where "2:2 pulldown" is an apt description.

Exactly, but this depends by your settings and your needs.

Of course, interlacing a progressive source is at the EN-coder level is programmmatically simple. But should we do it? Is it done this way? Perhaps. I don't know.

A lot of dvd were encoded in that way (with interlaced flags). This also prevent some dvd players to output in progressive-scan (if they rely completely to the flags embedded in the mpeg2 stream)

But just a few minutes ago I came across this statement:
I don't remember which part of the suite were used, but I've received a report from the replicator (made with the EclipseSuite) with a warning regarding the dts audio: "Since dts is not mandatory, some dvd players could be not able to decode it correctly". So I think that at least one tool of the suite have indeed read the IFO/VOB.

foxyshadis

15th June 2010, 19:33

That might be a lawyer-worded way of saying that while they'll check for their idea of conformance, they do not guarantee that their idea is a perfect complete test of the spec. Otherwise, lawsuits and yada yada.

manolito

15th June 2010, 19:56

First of all a small bug report for HC 0.24 beta 04-04-2010:
In 1-pass VBR mode the HC log file always reports an average quantizer which is too small by a factor of 10 (0.400 instead of 4.00). Haven't tested it in 2-pass VBR. Peanuts...:)

@Koutsoubos
Interesting discussion, but I still think you should follow neuron's advice and post a sample. If your source is purely progressive, but your display chain still shows combing artifacts, a phase shift in your source is pretty much the only explanation.

Just assume for a moment that I (and neuron) are right and your source is phase shifted by 1 field:

Play it through a standard DVD player (no progressive scan) on an interlaced display (CRT), no combing would be visible as long as the field order is right.

Play it on a progressive scan player fed to a progressive display device, and say that either the player or the display has turned on their built-in deinterlacer: No combing artifacts, but certainly a loss of quality. And according to your first post not setting the TFF flag must have triggered the deinterlacer which is not good...

Play it in a progressive chain where neither the player nor the display have turned on their deinterlacer (which would be correct) then you would see combing artifacts.

So please provide a sample...

Cheers
manolito

Koutsoubos

16th June 2010, 02:05

Hey, it's not that I do not want to heed the advice of the likes of neuron2 or you. I see that both of you have a very rich bank of experience behind you.

Just that the source is very big (even a 4 second Lagarith-compressed snippet is 80 MB) and copyrighted. I am not authorized to release the material.

And it cannot be a field shift. For starters, there are no fields. The source is progressive.

In my "case 1 encoding" (HCEnc's new behaviour), in two progressive test setups (progressive player and display), when I actually force the DVD standalones to output progressive, things are fine. When I leave them in "standard" interlaced mode, perceptible combing appears, shall we say they do not seem smart enough to produce a kosher 50fps interlaced image from the 25p encoding. Commercial interlaced DVDs play fine.

In a plain-vanilla non-progressive scan DVD player and interlaced CRT, the decoder-interlaced image appears just fine.

So... that led to the long discussion about flags and all and whether "case 3" is the way to go ...

I would like to pose a question to all. Given a nominally 25 fps (i.e. forget about speed-up or not, it's now 25fps) progressive source bound for PAL, would you encode it as 25p or go ahead and encode it as 50i? Why or why not? How would you do it?

Koutsoubos

16th June 2010, 02:12

I don't remember which part of the suite were used, but I've received a report from the replicator (made with the EclipseSuite) with a warning regarding the dts audio: "Since dts is not mandatory, some dvd players could be not able to decode it correctly". So I think that at least one tool of the suite have indeed read the IFO/VOB.

This from an individual known as Trai, expressed in the Creative Cow forums (http://forums.creativecow.net/thread/155/869917):

The EclipseSuite of tools is crucial, no doubt; I've been licensed with them since September 2002, and use them everyday. But these tools make sure that what's presented to you as a replicator, i.e., the DVD Image, and that it can be mastered. Only a very few checks of the contents of the build folder (DVD-Video file structure in the VIDEO-TS folder) are included that pertain to mastering the Image; things like proper placement and flagging of layer break cell (which I and my partner working on DVDAfterEdit got Eclipse to implement during 2003/2004), making sure the DSI (data search information) has the correct number of sectors listed as are actually on the disc (more to it than that), and just a very few others...

And yes, it's true I'm afraid, Eclipse, as good as it is, doesn't check for DVD for spec compliance. The MEI DVD-Video Verifier does that. Eclipse (worth it's weight for the crucial tasks it performs) verifies if the Image presented is configured properly for your mastering equipment, and that the Image follows certain file system specifications - but again (and again) does not check the DVD for DVD-Video spec compliance.

mp3dom

16th June 2010, 09:17

Given a nominally 25 fps (i.e. forget about speed-up or not, it's now 25fps) progressive source bound for PAL, would you encode it as 25p or go ahead and encode it as 50i? Why or why not? How would you do it?

Personally I would encode it as 25p. Just to be sure, have you tried to encode your footage with same/similar settings on other encoders? Does it changes something?

This from an individual known as Trai, expressed in the Creative Cow forums

From that thread, the answer is not that clear. Other users says that the ImageMapper tool is able to check dvd compliancy. Maybe it depends by the version too. Anyway, if you're so worry, stay on the safe side by using conservative settings.
Regarding your problem about 'combing' artifacts on progressive image, it could be that some flags are switching over time. Even the scanning order (from Alternate to Zig-Zag and viceversa) could led to some strange artefacts on some standalone players.

manolito

16th June 2010, 17:57

Maybe to avoid all possible issues with different hardware player setups you could do what virtually all commercial authoring houses do:

Flag progressive PAL DVDs as interlaced TFF

Out of curiosity I just pulled 20 commercial progressive PAL DVDs from my collection and checked their flagging with ReStream.

Result:
None of them had progressive flags whatsoever, all were flagged as interlaced TFF. Nine of them showed zig-zag scan order, eleven had alternate scan order.

This leads to a feature request for the next HC version:
I would like to have an additional setting called *NO_PROGRESSIVE_FLAGS (or similar)
This way I could correctly encode progressive content with zig-zag scan order, but there would still be no progressive flag in the stream. This is currently not possible with HC (CCE SP can do this...) In AutoInterlaced mode this would also avoid the constantly alternating progressive flags which has been reported to cause trouble in some hardware configurations.

Cheers
manolito

Koutsoubos

17th June 2010, 07:36

Just to be sure, have you tried to encode your footage with same/similar settings on other encoders?
I have only tried HCEnc.

Other users says that the ImageMapper tool is able to check dvd compliancy. Maybe it depends by the version too.

ImageMapper seems to be a software used to mount DDP images as if they were DVDs: www.eclipsedata.com/PDFs/ImageMapper.pdf

Regarding your problem about 'combing' artifacts on progressive image, it could be that some flags are switching over time. Even the scanning order (from Alternate to Zig-Zag and viceversa) could led to some strange artefacts on some standalone players.

I don't know about switching flags. Would HCEnc do that sort of thing? What could I use to check this? Restream?

Koutsoubos

17th June 2010, 07:37

Out of curiosity I just pulled 20 commercial progressive PAL DVDs from my collection and checked their flagging with ReStream.

Out of curiosity, how would you know that the DVDs are "progressive PAL"? I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced. Did you inspect the stream and the picture structure?

Result:
None of them had progressive flags whatsoever, all were flagged as interlaced TFF. Nine of them showed zig-zag scan order, eleven had alternate scan order.

Do I correctly understand that the streams are flagged as: Progressive_frame=OFF; TFF=ON; Progressive_sequence=OFF ?

This is currently not possible with HC (CCE SP can do this...)

How is this achieved in CCE SP?

mp3dom

17th June 2010, 09:17

I have only tried HCEnc.
It could be a good thing to test other encoder if you can.

I don't know about switching flags. Would HCEnc do that sort of thing? What could I use to check this? Restream?
Maybe, other encoders can do that too if you tell to. With ReStream you can check only the first header, not the whole file. If the scanning order switch over time, you'll see only the starting scanning order.

I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced.
I've seen quite of them but in general they don't have the progressive_sequence set to ON (because in general the encoder used for dvd is CCE which doesn't support the progressive_sequence). They set progressive frame = ON and Top Field First set to ON, with constant ZigZag scanning order. This is probably the most compatible settings since it allows anyway to output progressive in a progressive-scan player.

How is this achieved in CCE SP?

Set ZigZag scanning order, leave unchecked 'progressive frame', set offset line to 0 and check Top Field First

manolito

17th June 2010, 12:48

Out of curiosity, how would you know that the DVDs are "progressive PAL"? I have never seen a PAL DVD labelled as "progressive" or otherwise, in fact I have a few that seem to be encoded interlaced. Did you inspect the stream and the picture structure?
To determine if a movie is progressive or interlaced you have to step through single frames (in a scene with horizontal motion) and look for combing. DGIndex or VirtualDub-MPEG2 are perfect for this, do not use a player which automatically deinterlaces. If there is no combing then the film is progressive no matter if it was encoded interlaced.
The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.

Do I correctly understand that the streams are flagged as: Progressive_frame=OFF; TFF=ON; Progressive_sequence=OFF ?
Yes!

How is this achieved in CCE SP?
mp3dom already answered this.

They set progressive frame = ON and Top Field First set to ON, with constant ZigZag scanning order.
Unfortunately not. None of the 20 DVDs I tested had progressive frame = ON, and not even half of them had been encoded with ZigZag scanning order.

Cheers
manolito

Koutsoubos

18th June 2010, 03:52

Set ZigZag scanning order, leave unchecked 'progressive frame', set offset line to 0 and check Top Field First
In CCE, assuming a 25fps progressive source, is the "Rate Conv" or "Rate Conversion" option selected in the GUI?

The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.
How would YOU encode a 25fps progressive source PAL DVD, Manolito? What settings would you use (assuming spec compliance I guess).

I really want to thank you all, guys. You were a revelation.

I'm off to do more tests. I hope this discussion has been fruitful for the HCEnc developers too. It's a great program.

Lyris

18th June 2010, 06:48

The probable reason why most commercial progressive PAL DVDs are encoded (or just flagged) as interlaced is that the authoring folks are too lazy or just too ignorant to flip a couple of switches on their hardware encoders.

I also thought that the reason for encoding interlaced was simply to avoid any cases where small Interlaced segments were hiding in progressive content. (Or what you said, worded less strongly :) )

The Cinema Craft line of encoders (at the very least the high-end versions) are designed for DVD compressionists, and the manual tells you to set PAL Progressive as Progressive if the source is such. So I would be surprised if Cinema Craft a) even allowed you to produce non-compliant output when its "DVD" mode is enabled and b) would recommend you to do so in their user manual!

FWIW, this is the UK PAL version of "Howl's Moving Castle" which is one of the only PAL discs I've seen that's flagged Progressive. I believe the same version is on sale in Australia:

http://img96.imageshack.us/img96/2411/howls.jpg

I've only done one PAL DVD (sold in the UK) and encoded it Progressive where applicable. I would hate to encode it Interlaced due to the decline in compression efficiency/quality. Has anyone ever seen any issues with doing this?

Emulgator

18th June 2010, 15:15

Some posts back:
http://forum.doom9.org/showthread.php?p=1367663#post1367663

I encode all my PAL-Speedup25p-from-film-24p DVDs as 25p with HC since then
and am happy with the outcome on different setups.

To flag Progressive Sequence as: no
seems to be the vital part to be DVD compliant.

So, DVD FAQ #3.4: http://www.dvddemystified.com/dvdfaq.html#3.4
where it says: "MPEG-2 progressive_sequence is not allowed"

Yes, I would follow this.
Given the fact that PAL and NTSC are field-based
and the main video connection 1995 was the yellow Cinch connector
carrying the composite video signal in interlaced sequence
there was no need to allow an encoding parameter to be set to an untransmittable value.

Otherwise, DVD players should simply skip this value if set to Progressive sequence.
But it made a visible difference on my setup...
All pulldowns, 3:2 (23.976p Film on 29.97i NTSC),
Euro (2:2:2:2:2:2:2:2:2:2:2:2:3 24p Film on 25i PAL) come out a bit borked on this particular combination.

To encode and flag the progressive frame as progressive frame came out beautiful,
player will have to perform 2:2 pulldown to output as fields via Composite anyway.
A progressive-capable player may probably output such encodes as progressive if this is allowed via HDMI.
I can't tell exactly, but reflagging tests showed an advantage in certain player/TFT-combinations.

To encode and flag progressive material as interlaced (no matter if tff=1 or 0)
came out less optimal encodingwise.
On playback player will follow tff/rff flags anyway.

hank315

18th June 2010, 15:55

Encoding settings suggestion:
- progressive sequence: off
- progressive frame: off
- TFF: set
- alternate scan: off
- MacroBlock dct_type: always frame

This way the encode is pure progressive and the player will see it as TFF interlaced (2:2 pulldown).

But AFAIK there's no encoder that can do this.

manolito

18th June 2010, 19:04

But AFAIK there's no encoder that can do this.
Could you implement it into a test version of HC? :devil:

Cheers
manolito

Koutsoubos

19th June 2010, 09:02

The Cinema Craft line of encoders (at the very least the high-end versions) are designed for DVD compressionists, and the manual tells you to set PAL Progressive as Progressive if the source is such.
Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)

What program did you use to check out "Howl's Flying Castle"? What are the flags? If Prog_Frame=ON; TFF=off; Prog_Seq=ON that would be good news!

Any artifacting on playback?
(Nevermind this question. I came across this: "What's more annoying still, is that only ONE of the PAL discs I tried - the UK release of "Howl's Moving Castle" - was free of combing artefacts on movement (particularly noticeable on camera pans)." http://www.lyris-lite.net/dv490v_review3.html)

Euro (2:2:2:2:2:2:2:2:2:2:2:2:3 24p Film on 25i PAL) come out a bit borked on this particular combination.
I think there's one key difference! In the scenario I am postulating the source is PURELY 25FPS PROGRESSIVE (either sped up or not, film source or not, it's 25fps nominally) and so there is no need for the "Euro pulldown".

Given that situation, would that change your opinion? Would your setups show any artifacting? Or "borking"? ;)

In a nutshell, I am seeing artifacting wih progressive setups (without forcing progressive output from progressive scan DVD) when TFF=on; prog_seq=off, and no artifacting when TFF=off; prog_seq=ON.

Encoding settings suggestion:
- progressive sequence: off
- progressive frame: off
- TFF: set
- alternate scan: off
- MacroBlock dct_type: always frame

But AFAIK there's no encoder that can do this.
Umm ... any difference between this suggestion and the CCE settings proposed by mp3dom (http://85.230.118.163/showthread.php?p=1409231#post1409231) If even CCE cannot do this, then ... we have a scoop with HCEnc?

mp3dom

19th June 2010, 11:21

CCE can do this. In SP/SP2 you can't decide the macroblock dct type (which is fixed to "always frame") and progressive sequence (fixed to "off") but all the other settings could be changed. So if you uncheck 'Progressive Frame' (progressive frame off), check "Top Field First" (TFF set), Offset to 0 (keep the TFF) and select ZigZag Scanning Order (Alternate scan off) you end with the same settings suggested by hank315.

Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)
He's speaking (I think) of CCE-SP3, or CC-X, not SP2 or SP. SP3 and X are the highest level/quality of CCE and allows more quality and also a lot of more other options too. The SP3 manual says: "If your PAL source is 2:2 pulldowned (conversion from ﬁlm to PAL video), Progressive will be appropriate." However from the manual it isn't clear if that option will set the progressive sequence to ON or (as SP/SP2 already does) set only the progressive frame.

Lyris

19th June 2010, 20:53

Where is exactly stated in the manual? I don't remember reading anything like that. (Could be a case of senility.)

What program did you use to check out "Howl's Flying Castle"? What are the flags? If Prog_Frame=ON; TFF=off; Prog_Seq=ON that would be good news!

Any artifacting on playback?
(Nevermind this question. I came across this: "What's more annoying still, is that only ONE of the PAL discs I tried - the UK release of "Howl's Moving Castle" - was free of combing artefacts on movement (particularly noticeable on camera pans)." http://www.lyris-lite.net/dv490v_review3.html)
Yes, that's my site - an old review I might add :D
I used DGIndex. What other program should I check with?

Regarding CCE manual - this is from SP3. It recommends setting Progressive instead of Interlaced in the "Picture" menu, which codes Frames instead of Fields.

If your PAL source is 2:2 pulldowned (conversion from film to
PAL video), Progressive will be appropriate.

And from SP2's manual:
6.3.11 Progressive frame
Select when the footage is progressive. This setting works on MPEG-
2 output. If you apply Inverse 3:2 pulldown, this setting does not
work.

If your source is 2:2 pulldowned (film PAL), it may bring a
better result.

If you want me to make some test files with SP2 and/or SP3 for analysis, let me know.

hank315

20th June 2010, 00:41

CCE can do this. In SP/SP2 you can't decide the macroblock dct type (which is fixed to "always frame") and progressive sequence (fixed to "off") but all the other settings could be changed. So if you uncheck 'Progressive Frame' (progressive frame off), check "Top Field First" (TFF set), Offset to 0 (keep the TFF) and select ZigZag Scanning Order (Alternate scan off) you end with the same settings suggested by hank315.
The MB dct_type has a large impact on the encoding efficiency, on a progressive source it should be set to "always frame", on an interlaced source the encoder has to decide which method to use (frame/field) based on the interlaced motion of the MB.
Do you really mean in CCE dct_type is "always frame" even when progressive frame is unchecked ?

Some very nice (free) MPEG apps which can analyse a stream onto the MacroBlock level can be found here: http://tsviatko.jongov.com/

Koutsoubos

21st June 2010, 10:52

FWIW, this is the UK PAL version of "Howl's Moving Castle" which is one of the only PAL discs I've seen that's flagged Progressive.

What are the flags of the MPEG2 stream? Is it:
1. Progressive_frame=ON
2. TFF=OFF
3. Progressive_sequence=ON ?

If that's good enough for Optimum Releasing, that's good enough for me!

What were the flags for the UK DVD that you encoded?

Boulder

21st June 2010, 11:01

Is it possible to have AutoGOP for shorter GOPs than 12 frames? I have to use a maximum of 10 frames per GOP for my Super-8 material which I'll pulldown from 18 to 25fps. Without AutoGOP, most scene changes are missed.

Lyris

21st June 2010, 15:27

What are the flags of the MPEG2 stream? Is it:
1. Progressive_frame=ON
2. TFF=OFF
3. Progressive_sequence=ON ?

If that's good enough for Optimum Releasing, that's good enough for me!

What were the flags for the UK DVD that you encoded?

The UK DVD I encoded was done with CCE SP2. I checked "Progressive" in the picture window for the progressive parts. One of the music videos switched between I and P so I manually told it to encode the Progressive parts as Progressive frames for better compression.

Howl's Moving Castle shows the following in Restream:

Frametype Progressive: CHECKED
Top field first: *not* checked
Progressive Sequence: CHECKED

Interestingly, the "User data" section shows this for "Char string":
"| HC 0.16A beta - (c)2004/2005 |"

Really? It was encoded with HCenc? In case you're wondering, yes, this is a real store-bought copy, not a "backup".

The disc I did shows the following:

Frametype Progressive: CHECKED
Top field first: CHECKED (I left this unchanged from the CCE default)
Progressive Sequence: NOT checked, because CCE does not write Sequence Display Extension. Too bad, if it did, even cheap Progressive PAL DVD players would be able to play back without jaggies.