HCenc 025

[colinhunt]

Please post the complete script you used for this test:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
FFVideoSource("X:\HDProRes.mov", threads=1)
Spline36Resize(704,576)
AddBorders(8,0,8,0)
ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)

Also, which version of Avisynth are you using?
It's 2.58, the non-MT one.

[Groucho2004]

Weird. I don't understand why 18 threads are utilized even though you specified only 1 thread for FFVideoSource.

Anyway, your bottleneck is clearly the decoding. Have you tried with "threads=4" or similar for FFVideoSource?

[colinhunt]

Anyway, your bottleneck is clearly the decoding. Have you tried with "threads=4" or similar for FFVideoSource?
I'm kinda sick of FFVideoSource because it bloody well re-indexes the 138GB source file *every* time I try to run an encode -- even with a .ffindex file visibly present in the same directory.

As for HCenc multithreading, I tried adding *SMP into HC.ini and reloading the .ini... but looking at "Current HC.ini" in "Settings 1" tab I don't see it there. Oh, hang on, could this be because I'm running the latest beta version, dated 16-06-2011?

[colinhunt]

I ran AVSMeter again, this time on a FFVideoSource script which had "threads=16":

Frames processed: 373 (0 - 372)
FPS (min | max | average): 20.73 | 23.74 | 21.26
CPU usage (average): 6%
Thread count: 18
Physical Memory usage (peak): 191 MB
Virtual Memory usage (peak): 193 MB
Time (elapsed): 00:00:17.547

And yet it pulls 18 threads out of somewhere on its own. Huh?

But notice how min FPS doubled compared to "threads=1" script, although max FPS remained the same.

[Groucho2004]

ColorMatrix(mode="Rec.709->Rec.601",Clamp=false)
Your version of ColorMatrix is horribly outdated. The change from bool to int for "Clamp" took place 6 or 7 years ago. That might also explain the strange threading behaviour.

[colinhunt]

Your version of ColorMatrix is horribly outdated. The change from bool to int for "Clamp" took place 6 or 7 years ago. That might also explain the strange threading behaviour.
Hmm, very possible. I downloaded it from http://avisynth.org/warpenterprises/ and the file is titled colormatrix_20070828.zip. I'll search for a newer one.

OK, found and installed v2.5, changed the script to "clamp=0" and ran it through AVSMeter: no change in FPS and # of threads AVSMeter reports.

[Groucho2004]

Hmm, very possible. I downloaded it from http://avisynth.org/warpenterprises/ and the file is titled colormatrix_20070828.zip. I'll search for a newer one.

Here. (http://bengal.missouri.edu/~kes25c/)

Just for testing, remove the ColorMatrix line and run AVSMeter again.

[colinhunt]

Just for testing, remove the ColorMatrix line and run AVSMeter again.
Ran two tests:

1) raised threads to 32: AVSMeter reports 34 threads. FPS and CPU usage remain the same.
2) threads at 32, commented out ColorMatrix line, ran AVSMeter: thread count 33, no change in FPS or CPU usage.

One more: Threads=32, commented out ColorMatrix, Spline36Resize and Padding:

Frames processed: 922 (0 - 921)
FPS (min | max | average): 18.17 | 54.40 | 29.34
CPU usage (average): 4%
Thread count: 33
Physical Memory usage (peak): 272 MB
Virtual Memory usage (peak): 274 MB
Time (elapsed): 00:00:31.429

[colinhunt]

Script:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\QTSource.dll")
QTInput("X:\HDProRes.mov")


AVSMeter:

Frames processed: 1036 (0 - 1035)
FPS (min | max | average): 11.07 | 93.67 | 30.59
CPU usage (average): 13%
Thread count: 10
Physical Memory usage (peak): 432 MB
Virtual Memory usage (peak): 435 MB
Time (elapsed): 00:00:33.864

[Groucho2004]

ProRes HD .mov files
Any chance you can cut a sample from that file?

[colinhunt]

Any chance you can cut a sample from that file?
I'm not allowed, sorry :/ ... but I can create one with exactly same specs this weekend, if that helps any.

[colinhunt]

Here's a link to a file with the same codec, bitrate and resolution: http://www.sendspace.com/file/rvmjv8

It's a 1080p24 ProRes HQ .mov file, 7.5 seconds and 151 MB.

Happy testing! ;)

[Groucho2004]

Here's a link to a file with the same codec, bitrate and resolution: http://www.sendspace.com/file/rvmjv8

It's a 1080p24 ProRes HQ .mov file, 7.5 seconds and 151 MB.

Happy testing! ;)
I get about 40 FPS with my i5 2500K @ 4GHz, 4 threads, ffms2.
It appears that Quicktime decoding is simply sloooow.

[colinhunt]

I get about 40 FPS with my i5 2500K @ 4GHz, 4 threads, ffms2.
It appears that Quicktime decoding is simply sloooow.
Well, ProRes HQ 220, but yes, so it would seem. So who do I approach with a donation to write a much faster decoder?

[Quarkboy]

Hey everyone, I was wondering if someone could help here although maybe only Hank would know...

I'm trying to compile my own custom matrix.dll for experimental purposes on windows, but it looks like I need the Intel fortran compiler...
I tried using gfortran and made the .dll, but the encoder gives an error saying it can't find the EXTMATRIX function.

Looking into the DLL I realized that gfortran uses different conventions when it comes to function name exports and the exported function was being lowercased to extmatrix...

Does anyone know how to successfully compile a matrix.dll on windows? Maybe I could use C++ or some other language?

[manono]

What's a Matrix.dll? A quantisation matrix isn't a DLL. The ones in HCEnc are all CQMs. Or am I misunderstanding you? Care to show us the matrix you created? There are already quite a few available in HCEnc.

If you're looking to create custom matrices, maybe LigH's Custom Quantization Matrix Editor will prove useful:

http://forum.doom9.org/showthread.php?p=469040

Or maybe the Rockas Rebuilder Matrix Editor will do:

http://www.jdobbs.com/Pages/Downloads.htm

[Quarkboy]

What's a Matrix.dll? A quantisation matrix isn't a DLL. The ones in HCEnc are all CQMs. Or am I misunderstanding you? Care to show us the matrix you created? There are already quite a few available in HCEnc.

If you're looking to create custom matrices, maybe LigH's Custom Quantization Matrix Editor will prove useful:

http://forum.doom9.org/showthread.php?p=469040

Or maybe the Rockas Rebuilder Matrix Editor will do:

http://www.jdobbs.com/Pages/Downloads.htm
Read the last pages of hcenc's manual.
There is an option to manipulate the matrices procedurally based on the motion and luminosity per GOP. To do this you need to compile a custom dll. Sample fortran code is provided for this.

[manono]

Oh, OK then. Sorry.

[techreactor]

Still investigating but I have to admit my time for it is very scarce ATM :(

MPEG analyser Pro is a pretty old program which I still use to analyse MPEG2 streams.
Development has stopped AFAIK, developer is Jean-Luc Pons, I don't know if it's available for download.

@hank, any updates for HCenc in the planning.

Are you planning to integrate Intel Quick Sync Video (http://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html) in HC Enc by using the Intel's Media SDK for its 4th gen processor, Intel claims that by using this, the video creation and conversion is 17x faster since its uses the dedicated media processing which is built right into their processor.

Handbrake started using it (in Beta) and some commercial converters are already on it.

[hank315]

I've recently started again doing some work on HCenc after not seeing the code for about 2 years.
Until now some small improvements are done.

Things I'm also thinking about:
- 64 bit version
It will improve speed, not because its 64 bit but more 128 registers available to work with.
This will take a while, all hand written assembler code has to updated and optimized.
- 4:4:4 color space
- Update bitrate and VBV control
For some high bitrate encodes this could be better...
- Trellis quantization
The articles I've read about it clearly show a full trellis will be slow as hell, if I ever finish it it will probably be some kind of pseudo-trellis.

And now find some time for it :)

Are you planning to integrate Intel Quick Sync Video in HC Enc by using the Intel's Media SDK for its 4th gen processor, Intel claims that by using this, the video creation and conversion is 17x faster since its uses the dedicated media processing which is built right into their processor.

This isn't easy to use/implement, also don't know if this will really improve performance.
Some dedicated asm code using the latest SIMD SSE/AVX instructions will probably be a better choice for HCenc.

[manolito]

Hi Hank,

good to see you back after such a long time...:)
I already thought you had abandoned HCenc for good.

HCenc 0.26 Beta is in almost daily use here - I only use it for DVD output, no problems whatsoever (I even hexedited out the 'Beta' from the title bar).


Welcome back!
manolito

[Sparktank]

Wonderful to see you are well (enough to post). :)

I had wondered about updated instruction sets for AVX.
My current machine I use HCenc on (mostly) has AVX instrcution set but SSE4 (4.2) is used instead.

64bit would make for an interesting update to play with.

Question regarding 64x HCenc: If the input is an AVS script, would the AVS version have to be x64 as well?

[techreactor]

This isn't easy to use/implement, also don't know if this will really improve performance.
Some dedicated asm code using the latest SIMD SSE/AVX instructions will probably be a better choice for HCenc.

Good to know that some changes are in the offing.

Currently the following converter are using it as per Intel: (http://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-consumer.html)

Arcsoft MediaConverter
Arcsoft MediaImpression
Badaboom Media Converter
Corel Digital Studio
CyberLink MediaEspresso
CyberLink PowerDirector
MainConcept
Movavi Video Converter
Roxio Creator

along with freeware's like Mediacode HQ and Handbrake, I personally have not used any of the above commercial softwares but reading the feedback posted by some endusers definitely mention the gains in terms of speed and time. Dvdfab has posted this comparison sheet (http://www.dvdfab.com/intel-quick-sync.htm)on their website.

You are the best judge of what will work better for HCenc.

[Sparktank]

Might I ask but a humble request for any future considerations?

Request:
If the source is an AVS script, would it be possible to copy the script to the designated *LOGFILE?

It would come in handy when doing multiple samples with different AVS scripting; such as, brightness tweaking, etc.

[Fishman0919]

I've recently started again doing some work on HCenc after not seeing the code for about 2 years.
Until now some small improvements are done.

Things I'm also thinking about:
- 64 bit version
It will improve speed, not because its 64 bit but more 128 registers available to work with.
This will take a while, all hand written assembler code has to updated and optimized.
- 4:4:4 color space
- Update bitrate and VBV control
For some high bitrate encodes this could be better...
- Trellis quantization
The articles I've read about it clearly show a full trellis will be slow as hell, if I ever finish it it will probably be some kind of pseudo-trellis.

And now find some time for it :)



This isn't easy to use/implement, also don't know if this will really improve performance.
Some dedicated asm code using the latest SIMD SSE/AVX instructions will probably be a better choice for HCenc.

Oh yeah... nice to see around again.

[ChiDragon]

All of a sudden I'm getting a crazy Visual Fortran popup window when I try to encode more than 961 frames with HCenc 0.26 beta.

[EDIT: Err. I figured out the issue. The drive I'm writing to ran out of space. Perhaps this could be handled with a more elegant message.]

"Visual Fortran run-time Error

forrtl: severe (27): too many records in I/O statement, unit -5, file
Internal Formatted Write
Image PC Routine Line Source
HCenc.exe 00593B5A Unknown Unknown Unknown

Stack trace terminated abnormally."

Also, HCgui 0.26 crashes upon attempting to open a script whose return value is not a video clip (undefined), rather than rendering an error message.

Example:
BlankClip(30,720,480,"YV12",30000,1001)
X = 0

[Sparktank]

Is there any simple math to calculate the *LOSSLESS temp file?

I'd like to better guess a better place for it instead of going to largest place (usually USB 2.0 external with at least 500GB free space).
I use anamorphic resolution (NTSC) 704x480.

[hank315]

The size of the lossless file can't be calculated exactly, the size is variable, it depends on the compressibility of the source.

In your case (YV12 color space):
each plane:
704 x 480 = 337920 bytes (Y plane)
2 x 352 x 240 = 168960 bytes (UV planes)
total per plane: 506880 bytes
per minute:
23.976 x 60 x 506880 = 730 MB

Each frame is compressed, average compression: 40 - 50%, so per minute the file size will be approx. 330 MB.

[Sparktank]

Thank you, this should be plenty enough to work with.