HCenc 025

[qyot27]

Hello all,

Has anyone been having trouble with FFMpegsourceMT and HcEnc in 1-pass mode? I'm using Windows 7 Ultimate and get a crash every time in 1-pass mode. Works fine in 2-pass. Any thoughts?

Thanks,

Southstorm
I've been experiencing a crash during the sampling pass with the C-interface branch of FFMS2 ('MT' is questionable now since much of that actually got committed to FFmpeg officially), but I don't entirely know whether it's an HCenc issue or an FFMS2 issue. It doesn't seem to happen on my computer (which uses XP, but I also don't do episode-length stuff on here due to the fact it would have to take hours for a single half-hour video), but it does on another Vista-based setup I have access to. I don't remember it happening with previous revisions of FFMS2, but said previous revisions had problems with the framerate conversion routines that caused a crash at the end of encoding.

The solution I found to avert this problem is:
Mount the FFMS2-using script as a folder with Pismo File Mount and the AVFS plugin.
Write secondary AVISource() script referring to the .avi in the mount folder.
Give the AVISource() script to HCenc.
Result: no crash.

[Revgen]

FEATURE REQUEST -

Support for "non-MPEG2 compatible" framerates and the ability to turn them into "MPEG2 compatible" framerates using RFF flags.

I want to encode a 19.98fps silent movie into a 29.97fps DVD-compatible movie using "soft telecine". Currently the only way I can do this is to use an outdated and buggy version of Mencoder that has Neuron2's DGPulldown feature included in the build. The video works fine with DVDLab Pro and DVD Styler and plays fine in both my standalones, but I'd rather do it with HCEnc instead since the only way to keep Mencoder from crashing is to encode the video into 4 chunks separately and combine with DGMPGDec. Way too messy.

[manono]

I want to encode a 19.98fps silent movie into a 29.97fps DVD-compatible movie using "soft telecine".
I do this sort of thing quite a bit, although using CCE. After making the script for 19.98fps, add an 'AssumeFPS(23.976)' at the bottom and encode for 23.976. Adjust your bitrates upwards. For example, if the average bitrate is 4000, set it for 4800. 23.976/19.98=1.2. 4000x1.2=4800. Do the same for the max and min bitrates. When all done run the 23.976fps MPV through DGPulldown set for 19.98->29.97 and everything comes out OK.

[Revgen]

I do this sort of thing quite a bit, although using CCE. After making the script for 19.98fps, add an 'AssumeFPS(23.976)' at the bottom and encode for 23.976. Adjust your bitrates upwards. For example, if the average bitrate is 4000, set it for 4800. 23.976/19.98=1.2. 4000x1.2=4800. Do the same for the max and min bitrates. When all done run the 23.976fps MPV through DGPulldown set for 19.98->29.97 and everything comes out OK.

Hmm...I didn't know that DGPulldown worked that way. I thought I had to feed it video that was actually 19.98fps. Thanks for the tip.

However, it would be nice if HCEnc could do the pulldown instead so I don't have to take extra steps.

[Trixter]

I'm trying to evaluate HCenc with other encoders (running a 3SSIM comparison) and I can't get HCenc to encode my source material in VBR mode (both 2-pass and 1-pass). It works fine in CQ mode, but when trying to encode my longform project (3.5 hours interlaced NTSC material, 376642 frames), it always hangs around 75% of the encode, with the log file stating that it's trying to reduce the bitrate into negative numbers.

I can try multiple parameters and post a log, but before doing so I wanted to know if there was a known limitation with really long projects. I tried both 0.25 and 0.26 beta.

[Fishman0919]

the HC026_beta_16-06-2011 release seems to have the HCenc_026.exe from HC026_beta_05-03-2011

[manolito]

the HC026_beta_16-06-2011 release seems to have the HCenc_026.exe from HC026_beta_05-03-2011
I don't think so. Rightclicking the executable and selecting 'properties' shows that the file has been created on 16-06-2011. Looks more like Hank just forgot to change the window title...


Cheers
manolito

[hank315]

I thought I updated the title bar, apparently not...
Now it is :)

[techreactor]

It seems that the ver0.25 is not able to control the bitrate with the Maxbirate command.

I encoded with *MAXBITRATE 9000, the DVD was skipping and freezing at a certain timeframe which left me perplexed,(I re-encoded 2-3 times removing parameters like Luma and AQ, just to be sure, but still it froze at the same frames) after checking on various things, checked only the video stream for bitrate in VLC player and found the bitrates jumping to 11000-12000 on certain frames during that specific playtime.

Will move back to 0.21, just wanted to report here.

[Trixter]

Why all the way back to 0.21? Why not 0.22 or 0.23? 0.23 added adpative quant, surely that would improve quality substantially.

hank, if I can provide a clip and .ini file that always produces the bug, where should I upload them to for analysis?

[techreactor]

Why all the way back to 0.21? Why not 0.22 or 0.23? 0.23 added adpative quant, surely that would improve quality substantially.

hank, if I can provide a clip and .ini file that always produces the bug, where should I upload them to for analysis?

I havent used .22 or .23 extensively whereas .21 I have, without any problems.

[Richard1485]

It would be nice to know which previous version of HCenc to use until a fix is released.

[Taurus]

@techreactor & Trixter:
how did you get this overshoot in maxbitrate?
I've just tested a few encodes I made in the past with Bitrate Viewer.
All maxbitrate are under choosen limit.
Example:
maxbitrate:9300 ->9286, a slight undershoot in all encodes.
average bitrate:6500 ->6500 in all encodes.
And no problems with hardware and software DVD players.
So what is so specific with your sources or settings?:confused:
Edit: maybe it's the decoder: directshowsource?

[hank315]

The max bitrate is based on a GOP level, so for a DVD this means about every 0.5 sec.
The average bitrate per GOP is checked, if it overshoots the max bitrate set, the GOP will be recycled with higher quantizers until it satisfies max bitrate.
For now I have no clue what's causing this because nothing really changed in the bitrate control.
Can you post the ini file and a part of the encoded m2v file where it overshoots, mediafire.com will be fine.

[techreactor]

@techreactor & Trixter:
how did you get this overshoot in maxbitrate?
I've just tested a few encodes I made in the past with Bitrate Viewer.
All maxbitrate are under choosen limit.
Example:
maxbitrate:9300 ->9286, a slight undershoot in all encodes.
average bitrate:6500 ->6500 in all encodes.
And no problems with hardware and software DVD players.
So what is so specific with your sources or settings?:confused:
Edit: maybe it's the decoder: directshowsource?

Nothing specific in the sources, in the same hardware/software environment if i switch to 0.23, HCenc works like a charm. Anyways once Trixter uploads, Hank can check himself and hopefully will find the problem.

[techreactor]

The max bitrate is based on a GOP level, so for a DVD this means about every 0.5 sec.
The average bitrate per GOP is checked, if it overshoots the max bitrate set, the GOP will be recycled with higher quantizers until it satisfies max bitrate.
For now I have no clue what's causing this because nothing really changed in the bitrate control.
Can you post the ini file and a part of the encoded m2v file where it overshoots, mediafire.com will be fine.

@Hank,

I have been testing this further in the limited time I get. Few observations using the same source and environment.

- I was using A2Dvideosource() & spline64resize() to downsize but DVD used to hang and bitrate went over the defined in HC ini.
- I moved to Lanczos4resize and the bitrate control is well within the defined rate in ini, DVD now skips instead of hanging.
- Now testing with Lanczos (3 taps by default).

- Will test later with a low bitrate setting (6k) instead of 8k, just to see if HCenc works better at lower bitrates.

But I hope its not the same problem you rectified in 2005 in 015.

Problem:

However, at the middle of this scene, the DVD starts to stutter and then hangs, like pausing. Fast forward and the DVD continues to play.

I tried this on my PC and it played OK (my DVD player is Pioneer DV-563A). I tried burning with 2x and still had the problem.


Your Solution:

In fact it is a bug in HC which was introduced in HC015.
All software decoders and most HW-players will play it without any problem but some HW-players (some Philips, Koss and Pioneer types) don't like it.
It is solved in the next HC016 release

Coincidently both my players are Philips. :confused:

[hank315]

In fact it is a bug in HC which was introduced in HC015.
All software decoders and most HW-players will play it without any problem but some HW-players (some Philips, Koss and Pioneer types) don't like it.
It is solved in the next HC016 release
I remember that one, motion vectors were too long, some HW players didn't like that.
But that's certainly solved.


hank, if I can provide a clip and .ini file that always produces the bug, where should I upload them to for analysis?
Could you please upload the clip and ini file, preferably at mediafire.com.

[Trixter]

Could you please upload the clip and ini file, preferably at mediafire.com.

I have successfully reproduced the "infinite underflow adjustment" bug and am uploading to mediafire. It will take a few hours to complete the upload though (the clip has to be a bit large to reproduce the bug 100%) so I'll email again with links when it's finished and ready.

[Trixter]

Okay, everything necessary to reproduce the bug should be here:

http://www.mediafire.com/?1l9u8faf6okl3

Let me know if you have trouble accessing it.

[hank315]

OK, got the files, thanks, will investigate.

[hank315]

Try this one:

*INFILE h:\work\hcenc bug\_chromosphere_test_480i.avs
*OUTFILE H:\work\hcenc bug\test_output.m2v
*BITRATE 4600
*MAXBITRATE 9200
*FRAMES 0 3565
*PROFILE best
*GOP 15 2
*INTERLACED
*TFF
*INTRAVLC 2
*MATRIX mpeg


and:

AVISource("_chromosphere_test_480i.avi")
ConvertToYV12()


Seems the *CHROMADOWNSAMPLE command screws up, don't know why ATM.

[Trixter]

I'm afraid that .ini just makes things worse. Now it just hangs earlier in the 2nd pass, with a different error message:

http://img190.imageshack.us/img190/9941/hcenchang.png (http://imageshack.us/photo/my-images/190/hcenchang.png/)

Were you able to reproduce this on your setup? What does my file package do on your test/development machine?

[TheSkiller]

@Trixter You're still feeding HCenc with YUY2 (4:2:2), add ConvertToYV12(interlaced=true) at the end of your AVS script. This way HCenc can skip the chroma downsampling (as the chroma gets downsampled by AviSynth).

I'm going to download your file package and try to encode the thing.

Edit: OK, downloaded it. Changed ConvertToYUY2() to ConvertToYV12(true) and used your HCenc settings. Error at 86% (total) "underflow, fixing VBV", freezes.
Then I removed *CHROMADOWNSAMPLE from the ini, still the same error.
Then I used Hank's settings (GOP 15-2), which gives me the same error that you get, "VBV at critical level".

[Trixter]

Yeah, I was pretty sure *CHROMADOWNSAMPLE wasn't the problem; the issue is with bitrate (you can see the VBV buffer empty right when the problem happens). Besides, the sample clip has no chroma info anyway ;-) (joking, I'm sure that has nothing to do with it, it's a high-contrast clip with a lot of motion which is obviously difficult to encode)

Hank, any thoughts, progress?

[hank315]

@trixter

If I use the next HC.ini file:

*INFILE G:\hcenc bug\_chromosphere_test_480i.avs
*OUTFILE G:\hcenc bug\out.m2v
*LOGFILE G:\hcenc bug\out.log
*BITRATE 4600
*MAXBITRATE 9200
*FRAMES 0 3565
*PROFILE best
*GOP 15 2
*INTERLACED
*TFF
*INTRAVLC 2
*MATRIX mpeg


and AVS script:

AVISource("_chromosphere_test_480i.avi")
ConvertToYV12()


The encode finished without a problem.

This is a source which is pretty tough for MPEG2, quantizers are very high.
Just look at the next links:
HCenc encode (http://hank315.nl/files/HCenc026.png)
CCE encode (http://hank315.nl/files/CCE_basic.png)

If the *CHROMADOWNSAMPLE command is added, HCenc freezes at 69% in pass 2.
That's weird because the down sample routines aren't used, the source is already YV12.
Still investigating, I thought this would be an easy one but it's not :devil:

[Trixter]

I tried these settings exactly and it worked. Since that chunk of footage is actually part of a much longer video, I then used those settings with the full video and it hung :-(

I'm not sure what else to tell you. I certainly can't upload the 141G source file for you to test...

[Trixter]

BTW in your screenshots you show a program called MPEG Analyzer Pro. Where can I download that program? I searched google for a while but couldn't find anything that looked like that program.

[hank315]

Still investigating but I have to admit my time for it is very scarce ATM :(

MPEG analyser Pro is a pretty old program which I still use to analyse MPEG2 streams.
Development has stopped AFAIK, developer is Jean-Luc Pons, I don't know if it's available for download.

[Trixter]

It's not available for download, could you make it available somewhere?

[Sharc]

http://www.labdv.com/software/encoding-en.html

You must register for downloading

[TheSkiller]

MPEG Analyser looks neat. But is it just me or is registering on that site not free? "Download service is restricted to VIP user. // Register for 1 year $$$39.90" :mad:

[Sharc]

Looks like, yes. 3 months is USD 19.90.
It's just the only link I found for downloading. I don't have it myself.
If you should find 9 more useful downloads there it's 2 USD each :D

[techreactor]

Still investigating but I have to admit my time for it is very scarce ATM :(


Any luck with the new version ??, I see 0.26beta only.

[techreactor]

@hank, Also are there any plans to have a 64bit version of HC enc ?

With the new release of FFMpeg which has fixed many bugs and is available in 64bit version. FFMpeg has come hairline close to HCenc in terms of Bare eye Quality and encoding speeds have also improved drastically.

Hope to get the 64bit of HCenc someday. :)

[smok3]

@techreactor: care to share some complete ffmpeg command lines?

[AlanHK]

I encode in batch files and like to put all the parameters on the command line, and use "-noini" to make it independent of any ini file.

I'm just trying the lossless option, which you can set with "-lossless", but there does not appear to be an option to set the path of that or of HC's database. It then defaults to HC's program folder, and I don't have enough room to put a lossless file there.

Can you add -llpath and -dbpath to the batch options?

[zambelli]

Has anyone run into issues with 4:2:2 sources? When I use any YUY2 Avisynth source (regardless of whether I pick 4:2:0 or 4:2:2 output), HCEnc just crashes on me immediately. I've tried both 0.25 and 0.26beta versions.

Here is my config:
*INFILE d:\videos\sources\foo.avs
*OUTFILE D:\Videos\Sources\foo.m2v
*BITRATE 50000
*MAXBITRATE 50000
*FRAMES 0 10439
*PROFILE best
*AUTOGOP 15
*DC_PREC 10
*INTERLACED
*TFF
*CLOSEDGOPS
*LASTIFRAME
*INTRAVLC 2
*MATRIX mpeg
*LUMGAIN 1
*COLOUR 1
*PRIORITY low
*CHROMAFORMAT 422

Note that I'm actually trying to use the 4:2:2 mode to produce 50 Mbps 4:2:2 mezzanines, so ConvertToYV12() in Avisynth is not an acceptable workaround.

[Carpo]

Any news on an update version of this great app or is the beta the last we will see of development Hank?

[Fishman0919]

@techreactor: care to share some complete ffmpeg command lines?

Here's one I've used with ffmpeg and a sim ver with Mencoder... works well with encoding above 4500k

ffmpeg.exe -i "3.avs" -framerate 23.976 -aspect 16:9 -dc 9 -g 12 -qmin 1 -qmax 31 -maxrate 7500k -q:v 2 -b_strategy 1 -bf 2 -pre_dia_size 3 -dia_size 3 -me_range 24 -b_qfactor 1.0 -i_qfactor 1.0 -b_qoffset 0 -mbd rd -mbcmp 2 -cmp 2 -subcmp 2 -preme 2 -precmp 2 -subq 2 -skipcmp 2 -qcomp 0.7 -qblur 0 -scplx_mask 0.2 -p_mask 0.5 -last_pred 4 -bidir_refine 4 -deblock 0:0 -pass 1 -passlogfile ffmpeg -an -f mpeg2video pass1.m2v

ffmpeg.exe -i "3.avs" -framerate 23.976 -aspect 16:9 -dc 9 -g 12 -lmin 0.01 -mblmin 1 -qmin 1 -qmax 31 -qsquish 1 -maxrate 7500k -b:v 5000k -bf 2 -pre_dia_size 3 -dia_size 3 -me_range 24 -b_qfactor 1.0 -i_qfactor 1.0 -b_qoffset 0 -trellis 2 -flags:v +mv0+cbp -mbd rd -mbcmp 2 -cmp 2 -subcmp 2 -preme 2 -precmp 2 -subq 2 -skipcmp 2 -qcomp 0.7 -qblur 0 -scplx_mask 0.2 -p_mask 0.5 -last_pred 4 -bidir_refine 4 -deblock 0:0 -pass 2 -passlogfile ffmpeg -an -f mpeg2video pass2.m2v

[zambelli]

Also, has anyone done quality comparisons of HCEnc vs x262 encoder?

[Fishman0919]

Also, has anyone done quality comparisons of HCEnc vs x262 encoder?

Yes... HC

x262 is not there yet.

[OtonVM]

I'm trying to encode video with HC but the resulting DVD doesn't work. I wonder if I'm using some incompatible settings but if I am I'm not getting a warning.
In any case, I encode the avs with these settings (HCenc 0.26 btw):
*INFILE d:\videos\to dvd\hc test\test.avs
*OUTFILE d:\videos\to dvd\hc test\test.m2v
*LOGFILE d:\videos\to dvd\hc test\test.log
*BITRATE 5500
*MAXBITRATE 9200
*FRAMES 0 4999
*PROFILE best
*AUTOGOP 12
*DC_PREC 10
*PROGRESSIVE
*PROG_SEQ
*TFF
*BIAS 10
*AVSMEMORY 500
*PULLDOWN
*INTRAVLC 2
*CUSTOMMATRIX
8 8 8 10 12 14 16 18
8 10 12 14 16 18 20 20
8 12 14 16 18 20 22 22
10 14 16 18 20 22 24 24
12 16 18 20 22 24 26 26
14 18 20 22 24 26 28 28
16 20 22 24 26 28 30 32
18 20 22 24 26 28 32 64

16 16 16 18 20 22 24 26
16 18 20 22 24 26 28 28
16 20 22 24 26 28 30 30
18 22 24 26 28 30 32 32
20 24 26 28 30 32 34 34
22 26 28 30 32 34 36 36
24 28 30 32 34 36 48 56
26 28 30 32 34 36 56 128
*LUMGAIN 1
*PRIORITY high
*CHROMAFORMAT 422

Then I import the m2v as asset in adobe encore which doesn't complain. But when I mux the file into a DVD structure and I try to play that in various players I get different results: MPC-HC with it's own decoder produces only a grey checkerboard (I guess macroblocks?), using ffdshow works fine, on the mac (see screenshot) the apple dvd player produces only a pink checkerboard (that can't be picked up by the screenshot), mplayer only a grey picture and VLC works fine. They all show subtitles.
And the main problem: my hardware player doesn't work or rather, like these software players it only shows subtitles.
http://i50.tinypic.com/30hv96e.jpg

Any ideas?

[AlanHK]

*MAXBITRATE 9200


Max bitrate is usually 8000.

Technical Info for DVD-Video (http://www.videohelp.com/dvd#tech)
Total bitrate including video, audio and subs can be max 10.08 Mbit/s (10080 Kbit/s)

So with subs, and audio, I think you've gone over.

[poisondeathray]

*CHROMAFORMAT 422[/CODE]


Any ideas?


422 for DVD ? Or was that a typo/ copy/paste error ?

[OtonVM]

422 for DVD ? Or was that a typo/ copy/paste error ?

Ha! That would be a typo wouldn't it? :)

No, I have no idea why I set that option, probably while I was testing different settings... Anyway, that was it, obviously...

Thank you!

[colinhunt]

Can anyone help me find (and remove) the bottleneck in my set-up?

I'm using HCenc to create DVD-compliant MPEG-2 streams from ProRes HD .mov files. Using high image quality settings my encoding rig does 16fps in both passes -- but CPU usage never goes above 10%.

Here's an example of a Avisynth script I load into HCenc:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\QTSource.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
QTInput("X:\HDProRes.mov")
Spline36Resize(704,576)
AddBorders(8,0,8,0)
ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)

I've also tested with ffms2:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
FFVideoSource("X:\HDProRes.mov", threads=1)
Spline36Resize(704,576)
AddBorders(8,0,8,0)
ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)

but fps and CPU usage remain the same.

Is there anything I can do to squeeze a faster encoding speed out of the system?

[Groucho2004]

Can anyone help me find (and remove) the bottleneck in my set-up?

I'm using HCenc to create DVD-compliant MPEG-2 streams from ProRes HD .mov files. Using high image quality settings my encoding rig does 16fps in both passes -- but CPU usage never goes above 10%.

Here's an example of a Avisynth script I load into HCenc:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\QTSource.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
QTInput("X:\HDProRes.mov")
Spline36Resize(704,576)
AddBorders(8,0,8,0)
ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)

I've also tested with ffms2:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
FFVideoSource("X:\HDProRes.mov", threads=1)
Spline36Resize(704,576)
AddBorders(8,0,8,0)
ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)

but fps and CPU usage remain the same.

Is there anything I can do to squeeze a faster encoding speed out of the system?

Things to check:
- How fast is the script itself? (test with AVSMeter (http://forum.doom9.org/showthread.php?t=165528))
- Have you enabled multithreading in HC?

[AlanHK]

Can anyone help me find (and remove) the bottleneck in my set-up?

Try using the lossless intermediate file.

It's probably lower res than your source, so the 2nd pass encode to MPEG should be faster. Anyway, it won't be slower.

It needs 10 or 20 GB per hour of video for the temp lossless file, see the settings if you need to specify where it goes.

[colinhunt]

Things to check:
- How fast is the script itself? (test with AVSMeter (http://forum.doom9.org/showthread.php?t=165528))
- Have you enabled multithreading in HC?
Thank you very much for taking the time to reply!

Gotta love this forum; one gets to learn new things every day :)

AVSMeter says my scripts are slooooooow. ffms2:

Frames processed: 436 (0 - 435)
FPS (min | max | average): 10.54 | 23.76 | 17.19
CPU usage (average): 5%
Thread count: 18
Physical Memory usage (peak): 201 MB
Virtual Memory usage (peak): 202 MB
Time (elapsed): 00:00:25.358

QTSource:

Frames processed: 362 (0 - 361)
FPS (min | max | average): 9.67 | 27.65 | 17.36
CPU usage (average): 7%
Thread count: 11
Physical Memory usage (peak): 245 MB
Virtual Memory usage (peak): 249 MB
Time (elapsed): 00:00:20.851

Of course, those are only a small sample from the start of the file.

As for HCenc multithreading... I have some reading to do.

[Groucho2004]

Please post the complete script you used for this test:

ffms2:

Frames processed: 436 (0 - 435)
FPS (min | max | average): 10.54 | 23.76 | 17.19
CPU usage (average): 5%
Thread count: 18
Physical Memory usage (peak): 201 MB
Virtual Memory usage (peak): 202 MB
Time (elapsed): 00:00:25.358

Also, which version of Avisynth are you using?