New version of MLC lossless codec.

[MiroLx]

Hello.

I have finished version 0.2 of lossless MLC codec. Link is here http://www.linek.sk/mlc/

Changes from previous version:
- now three color spaces are supported losslessly (YUY2,YV12,RGB24)
- better compression

more in readme file.

It is free, has many options... Comments are welcome.

P.S.
This is a new codec and I haven't oportunity to test it in all operating systems and configurations.
If you will have any problems with installation or basic functionality please let me know. Namely in Windows Vista I haven't test it at all.

Update: Version 0.7 26.10.2009

[Dark Shikari]

How does it compare in terms of:

a) compression ratio
b) decoding speed
c) encoding speed

to Lagarith, HuffYUV, and lossless x264?

[mitsubishi]

I'm currently toying around with the lossless codecs, tested this one, compression isn't bad and is validly lossless, but really quite slow.

Initial test from a simple 5 min d2v source, all on fastest settings:



Codec Enc.Time Size Dec. time
MLC 186 1.22GB 304
FFD huffy 43 2.12GB 92
Alpy 40 1.65GB 250

[MiroLx]

Thank's Sirber. It is corrected now.

Dark Shikari:

From codecs you mentioned I only tested it against Huffy. MLC is slower with higher compression.

mitsubishi:

Please can you tell me your configaratin. Is it MP? With single processor MLC is roughly 2 times slower than Huffy. But I didn't test version you mentioned.

[Sirber]

There's also a delete button ;)

removing my post above, gonna remove this one later.

[mitsubishi]

Yeah MiroLx, I have a c2d E6300. I am testing by running an avisynth script containing nothing more than a d2v source through VDub I plan on doing a full test later, with a harder source and encoding off the decoded result, both of which should negate the advantage of multithreaded versions.

Like I say this is only a rough test, but I've added lagarith and huffyyuv

Codec Enc.Time Size Dec. time
MLC 186 1.22GB 304
FFDs huffy 43 2.12GB 92
Alpy 40 1.65GB 250
Lagarith 134 1.34GB 193
Huffyyuv 45 2.56GB 104

[Dark Shikari]

What is the algorithmic reason for slower decode time than encode time, in this case?

[MiroLx]

Sirber:
Thanks again.

mitsubishi:
This codec wasn't really tested in MP enviroment so your test results would be very interesting.

Dark Shikari:

The reason is in range coder (modification of arithmetic coder), which is faster encoding than decoding. It is similar with codec's which use Huffman'c code (Huffy...)

[Dark Shikari]

The reason is in range coder (modification of arithmetic coder), which is faster encoding than decoding. It is similar with codec's which use Huffman'c code (Huffy...)
Ah k, so it basically works similarly to Lagarith.

I've been thinking for a while about an LZMA-based lossless codec, i.e. 7zip's algorithm; it would give considerably fast decoding than encoding while getting better compression ratio than any of the ones listed here.

The only way to beat it I would think would be to use a codec that doesn't only use I-frames, or to use something even slower like a PAQAR-based algorithm.

[MiroLx]

Dark Shikari:
I had to do some web browsing, because I am not very familiar with compression methods you post, so only some remarks:
- they both use range coder at the end
- they are both more text oriented compression methods, and compressing text and video are really 2 different things
-non key frames algorithm is nice idea (MLC also use it) but very time consuming.

In any case if you decide to implement video codec I wish you good luck.

[MfA]

I've been thinking for a while about an LZMA-based lossless codec, i.e. 7zip's algorithm; it would give considerably fast decoding than encoding while getting better compression ratio than any of the ones listed here.
I would not be so sure about that. Generic history based coder implicitly learn a model for the probabilities of a stream of symbols, rather than having an in build explicit model which you try to fit to the data. The problem is the curse of dimensionality, with a large context around a pixel you get an exponential growth in the possible combinations. The history will only ever cover a very small amount of that and it can not efficiently generalize (some "smart" coders can do partial matching to determine probabilities, but that is not nearly enough).

Even for text the state of the art has moved to including language models (see the Hutter prize winners).

[Dark Shikari]

I would not be so sure about that. Generic history based coder implicitly learn a model for the probabilities of a stream of symbols, rather than having an in build explicit model which you try to fit to the data. The problem is the curse of dimensionality, with a large context around a pixel you get an exponential growth in the possible combinations. The history will only ever cover a very small amount of that and it can not efficiently generalize (some "smart" coders can do partial matching to determine probabilities, but that is not nearly enough).

Even for text the state of the art has moved to including language models (see the Hutter prize winners).
The language models aren't very helpful for the PAQ encoders; they help a bit but PAQ8 is quite good overall and can pack basically anything, not just text. Language modelling and dictionary replacement helps a bit but not a huge amount.

I've done tests of LZMA-compressed uncompressed AVIs; it seems to perform a good bit better than Lagarith or HuffYUV. Adding delta-frame compression for cartoons and such would probably help further.

[MiroLx]

I've done tests of LZMA-compressed uncompressed AVIs; it seems to perform a good bit better than Lagarith or HuffYUV. Adding delta-frame compression for cartoons and such would probably help further.

I have also done some tests with 7-zip but with different results. Problem is that AVI is a mess of video data, audio data, indexes.. and universal compression program has no chance to decide which is which. Even if you separate only RAW video data such program can only find line similarities, but not row or interframe similarities if he doesn't explicitely know organization of picture data - colorspace and dimensions of the frame.

[mitsubishi]

Dark Shikari, I'd certainly be interested in a faster decoding lossless codec. As you saw in another post I am trying to find which has the fastest (1x encode + 2x decode) since this gives the best advantage in prerendering for a two-pass encode.

[MfA]

We are talking about encoding speeds an order of magnitude slower than decoding speed and the encoding speed of fast lossless coders. Even though you decode twice if you are using some fast filters it's going to slow you way down.

[Dark Shikari]

Dark Shikari, I'd certainly be interested in a faster decoding lossless codec. As you saw in another post I am trying to find which has the fastest (1x encode + 2x decode) since this gives the best advantage in prerendering for a two-pass encode.
LZMA is not fast in any sense of the word.

HuffYUV is fast.

LZMA is 1-1.5Mb/s encode, 6-8Mb/s decode.

That's only a few frames per second decoding on most video.

[akupenguin]

The reason is in range coder (modification of arithmetic coder), which is faster encoding than decoding. It is similar with codec's which use Huffman'c code (Huffy...)
I don't know if this applies to MLC, but entropy coding isn't the only reason huffyuv decoding is slower than encoding. There's also the pixel prediction algorithm: when encoding, all pixels are independent so they can be predicted in parallel with SIMD, whereas when decoding each pixel depends on the previous so they have to run in series with latency-bound scalar math.
I have some ideas about how to fix this, but they can't be implemented without changing the bitstream. Maybe some future version of ffvhuff.
And the only thing which might be faster than that is keeping byte alignment, a la LZO. (Not that I'm suggesting to actually use LZO, just look at how it's optimized compared to other LZ codecs.)

I've been thinking for a while about an LZMA-based lossless codec
To expand on the reasons why this isn't a good idea:
A generic context model will require much more context than a domain-specific model to provide the same prediction efficiency. Even if you do spend that extra memory and cpu-time it won't be as good, because increasing the number of contexts without incresing the prediction efficiency will dilute each context, thus it will take longer for the generic model to adapt to the current probabilities. This is especially problematic in an intra-only codec, because it has to reset the probabilities at the beginning of each frame.

I've done tests of LZMA-compressed uncompressed AVIs; it seems to perform a good bit better than Lagarith or HuffYUV.
That's because the LZ sliding window is a half-assed version of inter prediction. But LZ fails compared to real inter prediction because (a) it's line based instead of block based, so each motion vector has to be specified many times, (b) it has no subpixel interpolation, and (c) LZ can only match or not, it has no concept of residual.

or to use something even slower like a PAQAR-based algorithm
FFV1 beats PAQ8L for still image compression (and PAQ8L does have an image-specific model). All of PAQ's model mixing and other stuff that costs tons of memory and cpu are useless here: only the image model gets used.

Even for text the state of the art has moved to including language models (see the Hutter prize winners).
Language models are a filter, they don't replace the LZMA/PAQ/other entropy coding step. The most well-known equivalent for images is Paeth.

1x encode + 2x decode
The fastest algorithm for anything is not to compute it at all:
You can get away with 1x encode + 1x decode, simply by forking the the decompressed video stream and sending one copy to the first pass of the lossy coder and the other copy to the lossless coder.
avs2yuv (http://akuvian.org/src/avisynth/avs2yuv/) supports this, or you can do it in general with tee (http://unixhelp.ed.ac.uk/CGI/man-cgi?tee) + named pipes (http://unixhelp.ed.ac.uk/CGI/man-cgi?mkfifo).

[MfA]

I have some ideas about how to fix this, but they can't be implemented without changing the bitstream. Maybe some future version of ffvhuff.
With some sideband information (starting value/location of image rows) you could parallelize predict left when a matching encoder was used without changing the bistream, doesn't make too much sense to spend the effort though.

[akupenguin]

I don't need any sideband info, I only need to eliminate huffyuv's weird convention whereby the rightmost pixel of one row is considered to be the left neighbor of the next row.
It's not worthwhile for left prediction, but median is 30% of the decode time, whereas the encoder side version with SIMD is 12 times faster.

[squid_80]

The fastest algorithm for anything is not to compute it at all:
You can get away with 1x encode + 1x decode, simply by forking the the decompressed video stream and sending one copy to the first pass of the lossy coder and the other copy to the lossless coder.
avs2yuv (http://akuvian.org/src/avisynth/avs2yuv/) supports this, or you can do it in general with tee (http://unixhelp.ed.ac.uk/CGI/man-cgi?tee) + named pipes (http://unixhelp.ed.ac.uk/CGI/man-cgi?mkfifo).There's an avisynth filter called twriteavi that does a similar thing.
I doesn't need any sideband info, I only need to eliminate huffyuv's weird convention whereby the rightmost pixel of one row is considered to be the left neighbor of the next row.I can see how that could be faster, but can't see how it helps with SIMD.

[akupenguin]

Each pixel depends on its left, top, and topleft neighbors. I can't load consecutive pixels in an mmreg because that violates the left dependency, but I can load a diagonal stripe into an mmreg.

Pixels in their logical arrangement in a frame, with one register's worth highlighted:
http://img227.imageshack.us/img227/6629/stripe0fa7.png
How I'd store the temporary values in memory:
http://img151.imageshack.us/img151/9839/stripe1xg5.png
Cost: one extra transpose operation (the other transpose is free, since the bitstream parser has to store pixels one by one into an array anyway.)

[MiroLx]

I don't know if this applies to MLC, but entropy coding isn't the only reason huffyuv decoding is slower than encoding. There's also the pixel prediction algorithm: when encoding, all pixels are independent so they can be predicted in parallel with SIMD, whereas when decoding each pixel depends on the previous so they have to run in series with latency-bound scalar math.
I have some ideas about how to fix this, but they can't be implemented without changing the bitstream. Maybe some future version of ffvhuff.

Thanks for your explanation akupenguin. I don't use SIMD in
MLC (maybe futere version). If I understand it right you are able to use bytes for calculations, but numbers in MLC are bigger (32bit) in most cases so the advantage of SIMD is falling down. I am also waiting for Barcelona (desktop version) but AMD seems to have some problems recently...

[akupenguin]

32bit numbers still get 4x parallelism from SSE*, just not as much as the 16x that bytes get.
BTW, how do you get 32bit numbers out of video with 8bit colordepth? I can't think of any process that would add so much dynamic range without impeding lossless compression.

[MiroLx]

32bit numbers still get 4x parallelism from SSE*, just not as much as the 16x that bytes get.
BTW, how do you get 32bit numbers out of video with 8bit colordepth? I can't think of any process that would add so much dynamic range without impeding lossless compression.

No nothing like that. It's index to context model, which has to be big. So at the beginning there are 8bit color numbers and at the end 32bit index numbers. But you are right 4x parallelism is still better than none.

[Dark Shikari]

That's because the LZ sliding window is a half-assed version of inter prediction. But LZ fails compared to real inter prediction because (a) it's line based instead of block based, so each motion vector has to be specified many times, and (b) it has no subpixel interpolation
You're right about that: in fact, except in the case of actual constant image data between frames, LZMA fails to provide any benefit for inter compression at all. Therefore any LZMA codec would be intra-only like HuffYUV or Lagarith, as inter-frame compression with LZMA would basically be useless.

FFV1 would probably end up being better anyways.

And FFV1 outperforms PAQ8?? I've never tried it but PAQ8's image compression has always impressed me, it nearly doubles the compression ratio of PNG.

[MfA]

PAQ8 uses linear predictive coding in a roundabout way, so I wouldn't be surprised if it did well ... still if you want to go for unusably slow MRP has it beat.

[akupenguin]

I think the main problem with PAQ8's image compression is that the linear prediction is only for determining contexts, it's not used as a filter. Thus, if PAQ8 predicts a pixel value to be 128 and it turns out to be 127, then every single bit is predicted wrong (or if it knows the prediction is inaccurate, then every bit is coded with near 50% probability). Whereas in FFV1 that would be a residual of -1, which still compresses well.
There's also context dilution: If FFV1 sees a neighborhood of (128,128,128) and the current pixel is 128, then that's also evidence for a neighborhood of (127,127,127) implying that pixel is 127. PAQ8 treats those as separate events, and doesn't infer the probability of one from observations of the other.

[Dark Shikari]

PAQ8 uses linear predictive coding in a roundabout way, so I wouldn't be surprised if it did well ... still if you want to go for unusably slow MRP has it beat.
Yeah, even its image compression (with the special BMP context model, no context mixing) is 100KB/s on a fast machine... too slow to be useful for video.

But I think its still faster than the MSU lossless codec... ;)

[MfA]

For laughs I'll try a combination of MED prediction and QuickLZ, it should be fast enough for realtime ... dunno if the performance will be worth crap though.

[MfA]

Well crap was just about right, even slightly worse than huffyuv on some really clean CGI anime I tried it with (best case for this kind of entropy coding) ... still it might be of interest to someone (cut down and cleaned up cedocida, fitting in a different type of compression requires only minimal changes now).

http://karton.student.utwente.nl/sc.0.2.zip

[MiroLx]

Well crap was just about right, even slightly worse than huffyuv on some really clean CGI anime I tried it with (best case for this kind of entropy coding) ... still it might be of interest to someone (cut down and cleaned up cedocida, fitting in a different type of compression requires only minimal changes now).

http://karton.student.utwente.nl/sc.0.2.zip

And what was the speed compared to huffyuv?

[MfA]

Almost the same ... but both are probably mostly disk limited on my crappy system so that doesn't mean much. Didn't spend too much time with it ... it's not meant as competition, it's just a toy ;)

[Dark Shikari]

Almost the same ... but both are probably mostly disk limited on my crappy system so that doesn't mean much. Didn't spend too much time with it ... it's not meant as competition, it's just a toy ;)Test it encoding to /dev/null then :p

[bill_baroud]

Well crap was just about right, even slightly worse than huffyuv on some really clean CGI anime I tried it with (best case for this kind of entropy coding) ... still it might be of interest to someone (cut down and cleaned up cedocida, fitting in a different type of compression requires only minimal changes now).

http://karton.student.utwente.nl/sc.0.2.zip

Great !
I was just looking around for the fastest lossless codec possible and quickLZ (along with lzo) looked like good candidates...

[Dark Shikari]

Just a thought: would it be beneficial to apply a transform to video going into a line-based compressor (like LZ, etc) to turn, say, 8x8 blocks into 64-long lines, so that the compressor acted more efficiently rather than scanning across lines of the video?

[MfA]

I did consider it for a moment, but I am not sure it would be a sure win. If you are just encoding a straight edge between 2 colored planes then you can't generally pick a block and overlap it with the present one to get a good match. When you just encode lines though then in the line above you can always pick a section which you can overlap with the edge on the present line.

Still, it's not a lot of trouble to try it ...

[MiroLx]

Just a thought: would it be beneficial to apply a transform to video going into a line-based compressor (like LZ, etc) to turn, say, 8x8 blocks into 64-long lines, so that the compressor acted more efficiently rather than scanning across lines of the video?

I am afraid this would be problematic, because when finding block match you have to slide one block in pixel steps and your model only allows block steps.

[mitsubishi]

Thanks akupenguin and squid on the heads up on the one decode methods, I'll look into those more thoroughly later.

MfA, I tested your codec, not bad.


Codec Enc.Time Size Dec. time
MLC 186 1.22GB 304
FFDs huffy 43 2.12GB 92
Alpy 40 1.65GB 250
Lagarith 134 1.34GB 193
Huffyyuv 45 2.56GB 104
sc.0.2 103 2.56GB 120

Through "read.avs"
AVIsource("w:\1.avi")which I use for the decode speed test and lossless verification it is fine, but is all blue tinted when played back in MPC directly.

[squid_80]

MfA, I tested your codec, not bad.Are these tests using YUY2? Everytime I compare original huffyuv with ffdshow using YUY2 (no adaptive tables), ffdshow ends up about 40% bigger. Using adaptive for both the size difference is < 0.1% (as you'd expect). Am I just choosing bad material?

[mitsubishi]

Everything is in the sources YV12, except for Huffyyuv which I had to convert back on reading it.

source.avs
DGDecode_mpeg2source("D:\test1.d2v")

read.avs
AVIsource("w:\1.avi")

verify.avs
A=import("source.avs")
b=import("read.avs") #.converttoyv12() # needed for huffyyuv

return ssim(a,b,"results.csv","averageSSIM.txt")

I've only tested on one source so far, I intend to check a HD source later. This source is clip from Doctor Who (DVB)

[Dark Shikari]

I am afraid this would be problematic, because when finding block match you have to slide one block in pixel steps and your model only allows block steps.
Yeah but it would be better than linear encoding wouldn't it? Block-matching would make a lossless codec much too slow I'd think.

[MiroLx]

Yeah but it would be better than linear encoding wouldn't it? Block-matching would make a lossless codec much too slow I'd think.

I don't know. Certainly it could be better. Some experiment would be usefull..

[akupenguin]

would it be beneficial to apply a transform to video going into a line-based compressor (like LZ, etc) to turn, say, 8x8 blocks into 64-long lines, so that the compressor acted more efficiently rather than scanning across lines of the video?
Take a look at the Hilbert curve. You don't have to pick a block size; Hilbert maximizes the spatial locality at any scale.

It may help in codecs where the entropy coder exploits the correlation between consecutive pixels (e.g. zero run). But it shouldn't matter for explicit-context-based codecs like ffv1, and it should interfere with LZ compression. When pixels are coded in raster order, a line in one place will match the same line in another place. If you munge the pixel order, the same image content may end up with a different byte representation depending on where it is.

[MfA]

but is all blue tinted when played back in MPC directly.
Oops, got the yv12 to rgb conversion wrong (only used when you are playing it back with a media player, since it doesn't accept anything else except for yv12 and when you are testing it you are requesting yv12 output directly). Reversing the U/V pointers in the transform code makes it work right.

[bill_baroud]

Codec Enc.Time Size Dec. time
Alpy 40 1.65GB 250

Hi, sorry to barge in but i skipped this codec from your results, the speed and the size looks very good compared to the others, but google or doom9 search only point to this thread when looking for "Alpy". Where can I find this codec ?

Thanks.

[Boulder]

Hi, sorry to barge in but i skipped this codec from your results, the speed and the size looks very good compared to the others, but google or doom9 search only point to this thread when looking for "Alpy". Where can I find this codec ?

Thanks.http://www.alparysoft.com/products.php?item=35

[MiroLx]

Hello.

There is new version (0.3) of codec.

Changes:

- faster and better compression (in most cases)
- some bugs corrected (I have to change inner format ...)
- some problems with vfw interface solved

[video_magic]

Thank You for this promising codec. I intend to try it for my next lot of captures, and I wondered whether a P4 version could be made available and if this would give better performance (less CPU usage when capturing)?
Thanks again.

[MiroLx]

Thank You for this promising codec. I intend to try it for my next lot of captures, and I wondered whether a P4 version could be made available and if this would give better performance (less CPU usage when capturing)?
Thanks again.

Hello video_magic.

No I don't plan version for Pentium4, sorry...I have some plans for new version (MP and also 64bit version), but for some time vesion 0.3 should be the last.

[video_magic]

Okay, that's great - I appreciate what You have done.

Does 'MP' mean multi-processor, and if so, does that mean it will take advantage of the Hyper-threading of my P4 Prescott which can show up as 2 CPUs in task-manager?

Thankyou again for the really nice codec.

I use Windows XP SP2.

[MiroLx]

Okay, that's great - I appreciate what You have done.

Does 'MP' mean multi-processor, and if so, does that mean it will take advantage of the Hyper-threading of my P4 Prescott which can show up as 2 CPUs in task-manager?

Thankyou again for the really nice codec.

I use Windows XP SP2.
Thanks for Your comment. Yes with 'MP' I ment multi-processor and this will probably increase speed also on P4 Prescott.

[3ngel]

Hi,

i'm testing your codec, and it seems play well.

What about adding as soon the MP (at least on the decoding side) so it can be used for realtime playback?

Thanks and keep going so.

[MiroLx]

Hi,

i'm testing your codec, and it seems play well.

What about adding as soon the MP (at least on the decoding side) so it can be used for realtime playback?

Thanks and keep going so.

Hello.

Yes I know, playback is little bit slower, but multi-threaded version will take some time. At least I have to know that single-threaded version is really stable, it is much simpler to fix problems here...

[MiroLx]

Hello.

Version 0.4 of codec is now available.

I have done some speed optimizations. The biggest changes are in the fastest mode. High compression modes are not modified. Codec is backward compatible with previous version.

[MiroLx]

Hello.

New version (0.5) for summer. Multithreading option is now available. So codec will be faster on processors with more cores, maybe also on HT processors (didn't try). If you find any problems please let me know.

[Audionut]

2100 frames ---------------------- Time ------ Size

Alpy max speed ------------------- 10 ---- 301,742,080
Huff YV12 adaptive --------------- 11 ---- 261,971,968
Lagarith YV12 - Multi threaded ---- 15 ---- 245,825,536
MLC Max speed - Multi threaded --- 17 ---- 225,757,184
MLC High Speed - Multi threaded --- 28 ---- 198,580,224
FFV1 ----------------------------- 29 ---- 219,985,920
X264 Lossless Megui Profile ------- 30 ---- 200,042,474
MLC Max comp - motion fastest --- 51 ---- 187,492,352
Alpy max comp ------------------- 1:00 ---- 266,919,936
MLC Balanced - Multi threaded ---- 1:40 ---- 182,087,680
MLC Max comp - Multi threaded --- 8:34 ---- 175,843,328

Max cpu usage with my core2duo was about 70%. I guess a little more tweaking can be done.

Best speed/compression for me is MLC in high speed mode.

[video_magic]

I have tried the newest version and have to report that it is too slow to capture with for me, even using the 'Maximum Speed' and multi-threading options.

This was trying to capture 720x480 29.97 with a P4 SL9KE 3.2ghz HT 2mb.
My pc system is running clean & tidy and defragged.
It just dropped too many frames to be usable. But if a newer version of MLC comes out I look forward to trying it! Thanks.

[johnsonlam]

2100 frames ---------------------- Time ------ Size

Alpy max speed ------------------- 10 ---- 301,742,080
Huff YV12 adaptive --------------- 11 ---- 261,971,968
Lagarith YV12 - Multi threaded ---- 15 ---- 245,825,536
MLC Max speed - Multi threaded --- 17 ---- 225,757,184
MLC High Speed - Multi threaded --- 28 ---- 198,580,224
FFV1 ----------------------------- 29 ---- 219,985,920
X264 Lossless Megui Profile ------- 30 ---- 200,042,474
MLC Max comp - motion fastest --- 51 ---- 187,492,352
Alpy max comp ------------------- 1:00 ---- 266,919,936
MLC Balanced - Multi threaded ---- 1:40 ---- 182,087,680
MLC Max comp - Multi threaded --- 8:34 ---- 175,843,328
mode.

Sorry, I'm stupid. It's hard to see from the chart above.
Did you have direct FPS or fixed size/time video to compare?

[Audionut]

Did you have direct FPS or fixed size/time video to compare?

I don't understand.

It was 2100 frames, at 23.976fps iirc.

[johnsonlam]

I don't understand.

It was 2100 frames, at 23.976fps iirc.

Sorry, the text mess-up (not in fixed-width font) so I have difficulty reading.

Now I understand ...

Which version of HuffYUV you use?
Did you try 2.30 by Squid_80 (Andrew Dunstan).

Thanks.

[Audionut]

huffyuv_ccesp-patch_0.2.5


Did you mean here. http://forum.doom9.org/showthread.php?t=113889

I dropped off following that thread. Last I read it wasn't real stable.

but the files will be incompatible with older huffyuv builds (and ffdshow).

[MiroLx]

2100 frames ---------------------- Time ------ Size

Alpy max speed ------------------- 10 ---- 301,742,080
Huff YV12 adaptive --------------- 11 ---- 261,971,968
Lagarith YV12 - Multi threaded ---- 15 ---- 245,825,536
MLC Max speed - Multi threaded --- 17 ---- 225,757,184
MLC High Speed - Multi threaded --- 28 ---- 198,580,224
FFV1 ----------------------------- 29 ---- 219,985,920
X264 Lossless Megui Profile ------- 30 ---- 200,042,474
MLC Max comp - motion fastest --- 51 ---- 187,492,352
Alpy max comp ------------------- 1:00 ---- 266,919,936
MLC Balanced - Multi threaded ---- 1:40 ---- 182,087,680
MLC Max comp - Multi threaded --- 8:34 ---- 175,843,328

Max cpu usage with my core2duo was about 70%. I guess a little more tweaking can be done.

Best speed/compression for me is MLC in high speed mode.

Hello.

Max cpu usage with my core2duo was about 70%...
You are right, but the YV12 space is worst optimized for more cores. But I'll try to improve iit n next version.

[MiroLx]

I have tried the newest version and have to report that it is too slow to capture with for me, even using the 'Maximum Speed' and multi-threading options.

This was trying to capture 720x480 29.97 with a P4 SL9KE 3.2ghz HT 2mb.
My pc system is running clean & tidy and defragged.
It just dropped too many frames to be usable. But if a newer version of MLC comes out I look forward to trying it! Thanks.

Hello.

It is little suprising that You have dropped frames with your PC. If I remember I could capture full PAL with no dropped frames with my old 2,4 GHz AMD single core processor. (CPU usage was almost 90%). One question - does multithreading option helped with P4 HT processor?

[Audionut]

huffyuv_ccesp-patch_0.2.5

My bad. I actually used huffyuv from ffdshow. Would have been latest try-out build from the time i did the encode.

[video_magic]

I cannot notice any obvious benefit from Multi-Threading - although I don't use any proper utilities to monitor the difference, sorry, but it was just too slow to capture full-frame :(

I would like to try MLC again if you bring out a new version - the compression ratios certainly look impressive.


Hello.

It is little suprising that You have dropped frames with your PC. If I remember I could capture full PAL with no dropped frames with my old 2,4 GHz AMD single core processor. (CPU usage was almost 90%). One question - does multithreading option helped with P4 HT processor?

[MiroLx]

Version 0.6 is now available. Added support for RGB32. Also
corrected one error in multithreading mode. I have tested that RGB32 only in VDub so if you find any problem with it please let me know.

[MiroLx]

New version 0.61 is available for some time. Only improved
memory managment (format compatible with 0.6 version), solved
problems with TMPGEnc.

[MiroLx]

Version 0.7 is ready for downloaded. Tried to improve multithreading mode - little bit more challenging than I expected but now 4 independent threads are supported. So hope it will be usefull for people with 4 (or more) core proccessors.
Some tweaks between speed and compression efficiency, key frames are now faster. New instalation program added for newer versions of Windows. If you find any problem, please let me know...

[MiroLx]

Hello.
Version 0.8 is available, if anybody's interested. Changed internal memory model. New model uses less memory. Part of it was used to improve compression in modes not using ME. Speed is also better in this case..

[Boulder]

Always appreciated :)

[MiroLx]

Version 1.0 is available. Improved compression in higher comression (slower) modes. Also MLC codec is now open source under GNU license.

[MiroLx]

You can download 64 bit version of codec. It is slightly faster than 32 bit version (little surprise). Tested only in VirtualDub64, don't know many 64 bit applications using VfW...

[MiroLx]

Version 1.1. is ready for download. Modifications in motion estimation - better compression, but slower. 64 bit version is not available (now), I don't know if anybody use it.

[Leeloo Minaï]

I am interested by a x64 build of MLC codec v1.1 !

About version 1.1 : is it normal that v1.1 is slower than v1.0 for the same settings (i.e. : maximum speed preset) ? Even without using motion estimation ?

I also noticed that x64 builds of VirtualDub don't remember multithreading option when i check it (this doen't happen with other codecs like Lagarith or UT), can you verify if it is a MLC related problem ?

Thanks

[MiroLx]

Hello Leeloo Minaï.

I have not modified the fastest mode, but still it is
possible that there are small changes in speed. It happened also to me (with older version) and I don't know exact reason, probably it depends on memory allocotion in cache as I changed inner codec buffer dimensions. It also depends on processor type, when I tested it on my computer the speed was +- the same.

To your second question I am not able to repeat your problem
with multithreading option. But I noticed that when you change settings in MLC32 than also MLC64 is affected.
In any case I will prepare 64 bit version soon..

[MiroLx]

Version 1.1 64 bit is available.

[Asmodian]

Thanks! :thanks:

Edit:

Doing some tests with interlaced YUY2 and YV12 720x480 from a very noisy VHS capture. Using Win7 x64 and VirtualDub 1.9.11.

Compressing 4000 frames of uncompressed YUY2 or YV12 off of and to a ram drive with an i7 980@4Ghz.

MLC v1.1 32bit Maximum Speed
YUY2: ~190fps / 21sec 667MB, cpu @ 19%

MLC v1.1 32bit High Speed
YUY2: ~107fps / 37sec 662MB, cpu @ 19-20%

MLC v1.1 32bit Balanced
YUY2: ~16.5fps / 4:01 634MB, cpu @ 13-14%, one core at 100%

Using Lagarith 1.3.27 32 bit
YUY2: ~360fps / 11sec 753MB, cpu @ 15% - 16%

Using Huffyuv 2.1.1 32 bit Predict median
YUY2: ~540fps / 7sec 806MB, cpu @ 9-10%

MLC v1.1 64bit Maximum Speed
YUY2: ~226fps / 17sec 667MB, cpu @ 17% - 18%
YV12: ~250fps / 15sec 533MB, cpu @ 15%

MLC v1.1 64bit High Speed
YUY2: ~118fps / 33sec 662MB, cpu @ 18%

MLC v1.1 64bit Balanced
YUY2: ~17.8fps / 3:42 634MB, cpu @ 12-13%, one core at 100%
YV12: ~17.9fps / 3:39 510MB, cpu @ 10-11%, one core at 100%

Using Lagarith 1.3.27 64 bit
YUY2: ~344fps / 11sec 753MB, cpu @ 16% - 17%
YV12: ~350fps / 11sec 602MB, cpu @ 12%

Using UT Video 10.2.4 64 bit Predict median (frame divide 12)
YUY2: ~770fps / 5sec 777MB, cpu @ 25-30%, very even cpu usage on all 12 logical cores

MLC is very impressive, and with the option to turn up compression if you have extra cpu time. :D

Still room to improve SMP in the higher compression modes though. In all these tests the hyper threaded cores were all parked with light activity on five cores and most of the work on one. No core at 100% unless stated.

Great Codec!

edit: I just meant to show how much faster 64 bit MLC was vs 32 bit. A bunch of other codecs added now. :rolleyes:

[kolak]

Try UtVideo :)

[MiroLx]

Asmodian

Thank for your comment.

Still room to improve SMP in the higher compression modes though

You are right. SMP support is limited in this codec (max. 4 cores). Have some ideas, if they work I will try to implemented them in future version (maybe:sly:).

[Asmodian]

I added UT Video. It has very good multi-threading, I assume due to frame dividing. It is also even further up the speed vs. size curve. 32 bit UT was about the same, this clip might be too short.

[Asmodian]

Have some ideas, if they work I will try to implemented them in future version (maybe:sly:).

Sounds good, I'll keep and eye out. :D

I also wonder if you handle UVYV input? I ask because that is the only 4:2:2 format my capture drivers will deliver.

At least MLC doesn't show up when selecting a codec in capture mode in VirtualDub 1.9.11 and UVYV support is my best guess. :o

[MiroLx]

No not support for UVYV. Never heard abot that format -

http://www.fourcc.org/yuv.php. Maybe you meant UYVY.

Avisynth can help you.

[Asmodian]

Oops yes, UYVY of course. I do use avisynth but I still need to capture to something else.

[Asmodian]

Sorry to ask again but do you support UYVY? (you only mentioned you didn't support the fictional format UVYV)

I would love to be able to capture to MLC.

from your www.fourcc.org link:

"UYVY is probably the most popular of the various YUV 4:2:2 formats. It is output as the format of choice by the Radius Cinepak codec and is often the second choice of software MPEG codecs after YV12."

Which makes me think support for this pixel format would help more than just me. I assume the conversion to YUY2 internally would be straight forward?

Thanks!

[mariush]

That text is ancient ... Radius Cinepak codec was popular when Windows 95 and Windows 98 were all the rage. Things have evolved since then.

I would say the most common color spaces used nowadays are YV12 and NV12, simply because they're very well supported in hardware by video cards and their internal hardware video decoders. Sure, technically UYVY does hold more color information but only if your source actually has that information (you capture the screen, games or you have a camera that doesn't record straight to MPEG2 or MPEG4. Even in this case, if you upload the video somewhere like Youtube, they're going to recompress it and convert it to YV12..

[MiroLx]

Hello Asmodian.

Sorry for late answer. No MLC does not support this format - only YUY2 and YV12 from YUV family. Difference between this format and YUY2 is minimal while the last mentioned is much more popular. Still if there is hardware or software which only supports UYVY I can consider adding support to next release of codec.

[Asmodian]

I would say the most common color spaces used nowadays are YV12 and NV12, simply because they're very well supported in hardware by video cards and their internal hardware video decoders. Sure, technically UYVY does hold more color information but only if your source actually has that information (you capture the screen, games or you have a camera that doesn't record straight to MPEG2 or MPEG4. Even in this case, if you upload the video somewhere like Youtube, they're going to recompress it and convert it to YV12..

Of course YV12/NV12 is much more widely supported as the standard color space for DVDs & blurays. However my Blackmagic Intensity Pro will only give UYVY to Virtualdub. Also as this is interlaced material I would not want to subsample to YV12 until after deinterlacing, not that I can anyway. I do think UYVY support would help more than just me.
Still if there is hardware or software which only supports UYVY I can consider adding support to next release of codec.

Please consider supporting UYVY but no need to keep it as UYVY, as you said YUY2 is the standard 4:2:2 format for the current video tools. Blackmagic's capture cards are one of the few good consumer options for lossless video capture in Windows 7 x64.

[MiroLx]

Please consider supporting UYVY but no need to keep it as UYVY, as you said YUY2 is the standard 4:2:2 format for the current video tools. Blackmagic's capture cards are one of the few good consumer options for lossless video capture in Windows 7 x64.

You mean convert UYVY to YUY2 at input and then use only this format. Looks like a good idea (also keeps things consistent), I will very probably add it to the next release of codec.

[Asmodian]

You mean convert UYVY to YUY2 at input and then use only this format. Looks like a good idea (also keeps things consistent), I will very probably add it to the next release of codec.

That sounds wonderful! :thanks:

[MiroLx]

OK. Version 1.11 is available. Added support for UYVY as discussed above. Doesn't have many chances to test it, so if you find any problem please let me know.

[Asmodian]

Thanks a lot! That was fast.

Sorry I have been having some trouble with my capture computer but will try it out as soon as I get it back up and running.

:thanks:

[therobin]

Can you add HDYC input support for use with Blackmagic capture devices? I'd like to find something other than huffyuv I can use to capture HD footage.

[MiroLx]

HDYC.

Looks like very similar to UYVY. Only interpretation of colors is different, which is transparent for codec. In this case I can add support for this colorspace. Problem is how to test it.

[therobin]

I can test it for you since I have blackmagic hardware.

[MiroLx]

OK. I will prepare some test version soon.

[MiroLx]

Hello therobin.

You can download new experimental version with support for HDYC at:

http://www.linek.sk/mlc/mlcv112zzz.zip

I really don't know if it works or not. I am not able to test in
even in VDub (it looks that it always convert this format to YUY2 before calling codec). Only 32 bit version is available, hope it will be sufficient. Waiting for your reply.

[therobin]

Unfortunately, it did not work.

I see "MLC v1.12" in VDub when video editing. But when I am in capture mode, it does not show up.

http://desmond.imageshack.us/Himg11/scaled.php?server=11&filename=nomlc.png&res=landing

[MiroLx]

Thanks.

There was an error in the codec. Made a new version with the same name. Hope this time it will be better.

(created 1.5.2012).

[therobin]

Ok, it shows up now.

http://desmond.imageshack.us/Himg705/scaled.php?server=705&filename=mlcz.png&res=landing

However, it just crashes when I try to capture something. But it still works fine in video editing mode.

[MiroLx]

However, it just crashes when I try to capture something.

This is a kind of answer I was affraid of.:o

Tried that capture mode in Vdub with MLC and my Lifeview capture card. It worked, don't know colorspace but probably YUY2. But codec behaves very similar with different colorspaces.
Have you also tried some different codecs when capturing -
FFv1 in ffdshow for example?

[therobin]

ffdshow does not support HDYC FourCC input.

I have huffyuv, the blackmagic codec and raw. All which work perfectly fine.

I can feed MLC a HDYC raw video in video edit mode and it works fine. It only crashes when capturing.

[MiroLx]

therobin
Thank you. It is more clear now. As I am not able to repeat
your crash it is very difficult to fix it. It looks that the problem
lies somewhere in the Vfw interface as the edit mode works. Maybe the time will help.
Still I plan to release version 1.12 (also 64bit) in this state soon.

Best regards

[MiroLx]

Version 1.2 is ready.

Added HDYC colorspace support. Also finished changes in ME modes started in previous version (1.1). The inner data format is now stabilized if no problems occur.