Huffyuv for the HD Era: An SMP/Multithreaded Port?

[3ngel]

I see, good then, so i'm waiting for a RGB32 version to test :)

[IS2000]

Chances are it is your hard disk array that is the problem. I have captured 24-bit RGB at 720p/60 with Huffyuv using a two disk RAID-0 array. I'm also sure I've done YUY2 1440x1080 at 30fps with no problems whatsoever. I have captured YUY2 720p/60 using Huffyuv with no problems whatsoever either.

I have found that capturing with Huffyuv with a preview window drains a lot of performance. Turn off the preview window and see how you get on.

Thanks for the response.:thanks:

I have RAID-10 setup with 4 drives (7200 rpm). The HuffYUV compressed data (1080i,720p) when played back via VirtualDUB, with both in/out preview disabled, hardly reaches 12fps@1080i, 45 fps@720p (59.94).

I now have a system with Intel Core2Duo E6600 (2.4Ghz, 4MB L2 Cache), Hardware RAID-10 4 Drives on PCIe, 4GB DDR2-800. Here are the numbers with this setup;

Application: VirtualDUB
Compressed FIle Format: HuffYUV predict left
Preview: OFF

1080i(29.97) => 18 fps sustained
CPU Util => 50% of both cores

720p(59.94) => shall try today

I'm lost as to why my 'fps' numbers are that low when compared to some of benchmarks in this thread.

Could you share more about your benchmark setup?

[easy2Bcheesy]

Not sure what to say except that I have achieved 720p/60 capture into YUY2 Predict Left (Fastest) on a P4 3.0GHz - and that is without the multithreaded version at all!

Before I opted to embrace CineForm compression I used Huffyuv all the time.

What capture speeds are you getting with no compression at all? Are you sure you are feeding the codec YUY2 input?

[squid_80]

Question: Does anyone actually use huffyuv to compress RGBA? Not RGB32 (8 bits per channel + 8 bits padding) but true 32-bit RGBA (8 bits per R, G, B and alpha). The reason I ask is because of how the original huffyuv is written; to make it accept RGB32 you have to turn on the "Enable RGBA" option, and it includes the padding byte in the compressed bitstream (poorly compressed, since there's no tailored huffman table for that channel). I'm thinking it would make more sense to always drop the alpha channel and save about 30% in output size.

Directshow differentiates between the 2 types with MEDIASUBTYPE_ARGB32 and MEDIASUBTYPE_RGB32. If I weren't so cynical I could use the input type to decide what to do with the padding byte... But IMO that's just asking for trouble.

[Shinigami-Sama]

big question

is that software raid?
if it is I'll bet thats your bottle neck....

[3ngel]

Personally i don't use RGBA. But RGB32 is important for alignment reason (i read on few places that GPU algorithms can take advantages of 16* multiplier alignment) and of course for quality reason 'cause for maximal quality GPU has to be fed with already decoded RGB values.
So if you want to drop RGBA support i don't have problem, but of course i suggest not to drop RGB32 support.

[easy2Bcheesy]

What sort of hardware do you have that gives you a RGB32 HD capture? Just about every commercial HD card centres on YUY2 - though 8-bit and 10-bit precision is often offered (eg Blackmagic/AJA etc)

[smok3]

Question: Does anyone actually use huffyuv to compress RGBA? Not RGB32 (8 bits per channel + 8 bits padding) but true 32-bit RGBA (8 bits per R, G, B and alpha). The reason I ask is because of how the original huffyuv is written; to make it accept RGB32 you have to turn on the "Enable RGBA" option, and it includes the padding byte in the compressed bitstream (poorly compressed, since there's no tailored huffman table for that channel). I'm thinking it would make more sense to always drop the alpha channel and save about 30% in output size.

Directshow differentiates between the 2 types with MEDIASUBTYPE_ARGB32 and MEDIASUBTYPE_RGB32. If I weren't so cynical I could use the input type to decide what to do with the padding byte... But IMO that's just asking for trouble.
i do use huff for rgb + alpha daily.

[sagacity]

What sort of hardware do you have that gives you a RGB32 HD capture? Just about every commercial HD card centres on YUY2 - though 8-bit and 10-bit precision is often offered (eg Blackmagic/AJA etc)

When recording SD/HD 4:4:4 material it is typically provided as 10-bit RGB.

[3ngel]

@squid_80
Any news for the RGB version of the codec?

[zambelli]

Just about every commercial HD card centres on YUY2 - though 8-bit and 10-bit precision is often offered (eg Blackmagic/AJA etc)
It would indeed be interesting to have a 10-bit lossless codec as an alternative to the commercial solutions such as Cineform.

[zambelli]

Any progress on HuffYUV development? It'd be really great to have a version that surpasses both 2.1.1 and Ffdshow versions in terms of feature set and performance. Right now you often have to pick one or the other and they're not necessarily mutually compatible.

[squid_80]

I dream of putting out huffyuv v3.0, with the following features:

YV12 encoding/decoding, compatible with the existing ffdshow implementation
Multithreading for all encoding modes (also for decoding, but that's less of a priority)
Proper backwards compatability to v2.x i.e. can decode anything produced with v2.x and encodes made with 3.0 that use advanced features are rejected by earlier versions
Full directshow and vfw codec in one dll
Proper ARGB and RGB32 support
A selection of preset huffman tables tailored to specific material as well as support for custom tables generated by an avisynth plugin
SSE speedups for the prediction functions

I've done little bits and pieces of all of these things, just don't have the time to put them all together into one tidy package.

[3ngel]

Full directshow and vfw codec in one dll
Proper ARGB and RGB32 support
Mmmm... :D

[lilhobo]

hey guys, the future is a dream, but how do u use huffyv for the present ?? :D

i have the 2.11 with cce patch but realplayer and wmp dont play it??? is this just the vfw codec, and i need ffdshow to do directshow?

[sebastian___]

I read all of the posts here but I'm afraid I have to ask again :
So is there a newer build of Huffyuv (or similarly faster codec) which takes advantage of multi core CPU's ? And I'm only talking about full lossless 24 bit RGB.
- Because I (and many others) have a quad core Intel which can use this, I think.
Thanks.

[Blue_MiSfit]

Lagarith is multithreaded...

~MiSfit

[Adub]

Well, Squid said he had pieces of the overall puzzle, he just hadn't put them together yet. Maybe he will soon?

[Blue_MiSfit]

That would be great, as HuffYUV is stupid fast to begin with... :)

Lossless 1080p24 HD-SDI capture to RGB24 HuffYUV would be very nice.

[squid_80]

I'm working on adding a new compression method (similar to ffdshow's adaptive tables). Can't promise anything at this stage since it's just an idea that I haven't had time to implement.

[Adub]

I think I speak for all of us when I say, "Cool! Keep us updated!".

[squid_80]

http://members.optusnet.com.au/squid_80/huffyuvadapt.zip

Look in the encoder config and you'll see an adaptive huffman tables option. Turning it on should give slightly better compression but the files will be incompatible with older huffyuv builds (and ffdshow).
After lots of testing I realize now this isn't the best way to improve compression. The problem is huffman compression works best when the input symbols have probabilities that are powers of 2 - quite often the probabilities of the 256 input symbols will be skewed so much that one symbol has a probability of >70%, especially with digital sources. Huffman is inefficient with any probability more then 50% which is evident if you try and zip or rar the huffyuv compressed avi file - frequently it can be compressed by a further 50%.

[Boulder]

Did the dreamt v3.0 ever come to life?

[squid_80]

Not quite. I think last time I tweaked the version number I was up to 2.4.6 - featuring YV12 support (not fully complete, but good enough), adaptive tables, median mode for RGB (practically redundant) and a complete DirectShow encoder interface. Multi-threading was abandoned since I just couldn't seem to make it work. Never got around to making it compatible with ffdshow either.

[johnsonlam]

Not quite. I think last time I tweaked the version number I was up to 2.4.6 - featuring YV12 support (not fully complete, but good enough), adaptive tables, median mode for RGB (practically redundant) and a complete DirectShow encoder interface. Multi-threading was abandoned since I just couldn't seem to make it work. Never got around to making it compatible with ffdshow either.

Thanks for the improvement.

What I concern is HuffYUV is the minimal delay, if the compression is too high it may affect the "on-the-fly capture with acceptable compression ratio" purpose.

There're some codec got really good compression ratio, but lossless with low latency (such as Largarith) seems all spin off from HuffYUV.

Sadly, largarith have few bugs unresolved.

[squid_80]

If you find bugs in lagarith, report them to the author. He normally fixes them pretty quick.

[johnsonlam]

If you find bugs in lagarith, report them to the author. He normally fixes them pretty quick.

He's nice, I've contact him years ago.

The problem is I can't reproduce the bug, it happen only under some special circumstances which I don't know.

When Largarith fail I have to fall back using HuffYUV.

[mariush]

I'm sorry for bringing this old thread back.

Have you guys checked the multithreaded Huffyuv from this page ?

http://home.h01.itscom.net/para/intro/huffyuv-mt/

I've checked the thread and couldn't find any reference to it. Any oppinions?

I'm currently doing lots of encodings where I'm splitting 30-50 minutes videos in 5-8 minutes intervals (which have to be stored lossless for various reasons) so I'm hoping to get better decompression speed (it feels like xvid encoding speed is limited by the huffyuv decoding) so overall i'd get much faster encodings.

I'm just worried there may be some bugs in it that would make me waste a lot of time.

[smok3]

lagarith does work well for me on my new work machine.

http://lags.leetcode.net/codec.html

(no, i did not try huff mt yet)