Lagarith lossless codec

[ChiDragon]

Do you have any use for crummy-looking 10-bit footage? I can send you some VHS captures in v210, but I haven't figured out how to verify if the devices are actually filling those bits.

[Keiyakusha]

I need real footage.
Define "real". Every 10bit footage is artificially produced one way or another. If Ill take some video, apply to it 16bit color correction, dither it down to 10bit - would that be real enough for you?

Edit: actually sorry. something like that was already suggested by zerowalker. Only that "Broadcast tape" is not any different and just as "real" as the footage people here proposed.

[ChiDragon]

Howso? If color-correcting a 10-bit source was no different than starting from an 8-bit source, there would be no high-depth film scanners.

[Keiyakusha]

Howso? If color-correcting a 10-bit source was no different than starting from an 8-bit source, there would be no high-depth film scanners.

If you answered to me, whatever you don't understand what I'm saying or I poorly explained it. By "If Ill take some video" I mean if I'll take 8-bit video. And after color correction, new color values will be created. Resulting footage is just as "real" as any other 10bit footage. And in fact may be even better, as most of the 10bit footage is created during production and not by scanners. Of course 3DCG rendered at 10+bits is an option too.

[SirLagsalot]

Define "real".
Either unprocessed source from a high-bitdepth camera, film scanner or similar capture source, or complex CG renderings done in high-bitdepth. Footage from an analog capture source is what I am most interested in, since I want to see how noisy such sources are. I figure if Lagarith can handle those types of footage well, it'll shouldn't have any trouble on upsampled video.

Links
Those look like they should be useful.

Do you have any use for crummy-looking 10-bit footage? I can send you some VHS captures in v210, but I haven't figured out how to verify if the devices are actually filling those bits.
Sure, I can check if it is using the full bit-depth.

[zerowalker]

Wait, are Broadcast Tapes 10bit upsampled?
Aren´t those originally in 10bit, but everything that actually shows to the World (Cinema, TV, DVD you name it) is 8bit, so the actual original (At least for broadcasts) are never actually seen?
It makes no sense if they just have 10bit tapes for the heck of it, if all material is actually 8bit, and when they are going to show it, they dither it down after it was upsampled once.


EDIT:

Not adding Rec.709 etc will actually be worse than not adding it.

As if you have RGB and save it to YV something, you will automatically convert it to Rec.601, so if that´s wrong the only way to solve it is through Avisynth and Colormatrix conversion, which is a lot more advanced, and also not perfect.
I think you could add it as, "Expert", or something, cause Decoding will not solve this if it´s converted, as it will stay Rec.601 all the time.

[benwaggoner]

Wait, are Broadcast Tapes 10bit upsampled?
Aren´t those originally in 10bit, but everything that actually shows to the World (Cinema, TV, DVD you name it) is 8bit, so the actual original (At least for broadcasts) are never actually seen?
It makes no sense if they just have 10bit tapes for the heck of it, if all material is actually 8bit, and when they are going to show it, they dither it down after it was upsampled once.
Storing in higher precision makes it possible to do overlays, color correction, etcetera without having to add more dither with every single filter. The same reason professional formats are all 4:2:2 even though all delivery formats are 4:2:0.

[zerowalker]

I see, so 4:2:2 are also upsampled?
Though i am pretty sure that thing were broadcasted as 4:2:2 before Digital took over.

[Keiyakusha]

zerowalker
You are doing too strong accent on the word "upsampled". For some reason I feel some kind of disappointment in your posts. It doesn't matters if it's upsampled or not. Also even though video often starts as 8bit, if some 16/32 bit processing applied to it (after which it is dithered to 10bit for archiving, delivery, whatever), you can no longer call it "upsampled". It became "real".

I figure if Lagarith can handle those types of footage well, it'll shouldn't have any trouble on upsampled video.
If you're talking about upsampled video that was simply converted 8->16bit without any further modifications.... why would someone want to compress it with lagarith? I don't see the reason for that. I only happens when you was requested to deliver some footage in some "standard" format, which happened to be 10bit. In case of lagarith one would compress it before upsampling and upsample when needed. But I also don't think one would use 10bit lagarith for raw noisy captures. Not in professional world. And most likely more casual users won't do it either.
If you're talking about upsampled video that received some kind of processing on top of that, as I said above, I don't think you can call it "upsampled" anymore. This is exactly the kind of content for which you want to optimize the encoder. And not for some random noise... unless for lagarith it doesn't matter which content you have. But then there is even less reasons to optimize for something exotic

[Lenchik]

why would someone want to compress it with lagarith?
For finding best x264 settings while not loosing time on repeating previoud work.

[Keiyakusha]

For finding best x264 settings while not loosing time on repeating previoud work.
This is not what I'm talking about. If you're doing some 16bit processing, your video will no longer be simply upsampled.
Anyway, that was a rhetorical question.

[zerowalker]

Keyakusha,

What i simply meant is, i thought those tapes, or rather what was in the content of the tape was 10bit.
For example, let´s say Lord of the Rings, it was recorded in 10bit and that´s it´s original source (an example here).
Then they are going to broadcast it on TV etc. And they get a copy of that source, meaning they put it on the broacast tape, so it isn´t upsampled, it stays the same.

Same goes for everything else, but of course if there is 8bit stuff it get´s upsampled.

But if Nothing is 10bit which seems to be the case, than that´s my disappointment.

But of course i know that 10bit itself helps a lot if post-processing has been done at something above 8bit, as less information will be removed, the leap between 8bit to 10bit is quite high, though 10bit to 16bit is enormous, but luckily the improvement in actual quality is much less from my understanding, which makes it "a waste".

[foxyshadis]

Keyakusha,

What i simply meant is, i thought those tapes, or rather what was in the content of the tape was 10bit.
For example, let´s say Lord of the Rings, it was recorded in 10bit and that´s it´s original source (an example here).
Then they are going to broadcast it on TV etc. And they get a copy of that source, meaning they put it on the broacast tape, so it isn´t upsampled, it stays the same.

Same goes for everything else, but of course if there is 8bit stuff it get´s upsampled.

But if Nothing is 10bit which seems to be the case, than that´s my disappointment.

Nothing consumer-side is is more than 8bits is what he meant, I'm sure. Studio-side, it's tremendously different; stuff is generated and normally archived in various camera raw formats during production at a minimum of 10bit (HDCAM SR) up to 16bit (RED). Only older original HDCAM and DVCPRO used 8bit. The transmission pipelines vary widely between fully analog and 8bit digital and up. The difficulty is in obtaining samples of those masters, not whether they exist. They are usually compressed, but at such wildly high bitrates that it's not how we think of compression (like DV vs VCD).

Consumers can generate 10bit video with prosumer cameras, like Canon XL H1 or Sony HVR-V1U, but it might be better to ask on a videographer enthusiast forum than here.

[Keiyakusha]

Well, I tried to explain what I mean but it seems i failed at it so I've withdraw from this discussion. But let's try once more ^^
For some reason you got an impression that I'm saying that there are no >8bit material or something... this is not it. Of course there are >8bit things produced by some hardware and not generated on PC. For the movies like LotR it is of course more likely that they could be >8bit starting from the camera shooting stage. But if we are talking about global scale, which includes TV shows and various other material, then most of the content was never 10bit to begin with. It is whatever simply upsampled because standard requires it/someone requested it this way or it was modified/edited at some point.
That said, I believe the codec should be optimized for more common usage scenarios. Especially that for big studios who have things in >8 bit from the beginning there are 0 reason to use lagarith. I bet many of them never even heard of it and there is no reason for them to do such a research, they already have all the tools they need. Not to mention that many of them simply refuse to work with anything free/opensource. So why optimize codec for them and not for the world of "mere mortals"? (rhetorical question. I don't need an answer)

Edit: anyway, sorry for going offropic. Personally I'm not even interested in lagarith, it is too slow, so feel free to optimize for whatever you want.

[zerowalker]

Ah well than i had understood it correctly, at least half way.
That Tapes (Though not Broadcast tapes) can be done in higher bit depth.

But as you say, it´s impossible to obtain them, the only way to get such a "master tape" would be to either have the resources and contacts for it.
Or that it the company owning it has gone bankruptcy and it´s sold or something.

Keyakusha,

I fully understand what you mean, and as you say, the reason to even have 10bit as lossless currently, doesn´t even exist, the only reason for it i can think of with is for experimentations and certain individuals.
But i however don´t by any means want it NOT to be there, support and optimization in any case is always good, it doesn´t matter if the thing is useless now or not, If it can be added and someone wants to, then by all means do so!

And can also agree with it being slow. But for 2D (Pixel and simple stuff) it´s totally unbeatable, you can´t even compare it with UT Video Codec which is the rival i think.
On "Normal" videos however, Lagarith isn´t that great in terms of speed, but i think it can be optimized as it has been laying around for quite some time compared to UT which has been updated all the time.


But anyway, if i can help with anything for updating Lagarith i am on. Sadly Programming wise i am at a total newbie level, but making tests samples or something is something i should be able to do.

[raffriff42]

Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth. The format should generally be avoided if possible.*This* is my issue with Lagarith. I get too many glitches.

[zerowalker]

Wait what??
Errors as in, what?

[LoRd_MuldeR]

Wait what??
Errors as in, what?

He's referring to the "random" glitches (artifacts) that some people seem to be getting with Lagarith.

AFAIK, Lagarith uses arithmetic coding, which - in its most simple form - uses infinite-precision real numbers. Computers don't have that. So, in practice, we usually use a "range coder" instead of a plain arithmetic coder. It is essentially one and the same thing, yes. But the "range coder" works entirely on integer math (rather than using real numbers). This makes it more suitable for implementation on a real-world computer. Also ensure that the results are always deterministic.

Lagarith, on the other hand, seems to implement arithmetic coding based on limited-precision floating point math. This is kind of "delicate" with respect to rounding errors and stuff. And this is what may cause nondeterministic behavior :eek:

[zerowalker]

But it does not effect Encoding does it?
Meaning all encodes movies will always be 100% alright.

But decoding them CAN fail, but redoing it CAN succeed, meaning it´s not a permanent error?


Weird that it uses such system if it isn´t perfect as these stuff are supposed to Always work in All scenarios.

[LoRd_MuldeR]

But it does not effect Encoding does it?
Meaning all encodes movies will always be 100% alright.

But decoding them CAN fail, but redoing it CAN succeed, meaning it´s not a permanent error?

Well, we have a problem as soon as the encoder and the decoder "desynchronize" in some way, i.e. do not agree on the exact input/output values any longer. Whether the encoder or the decoder (or both) made the "mistake" that caused the desynchronization doesn't matter much. What does matter is that, after a desynchronization, we don't get back the same values from the decoder that we originally fed into the encoder. So we might get some ugly artifacts!

(Or in other words: Regardless of weather the video already was encoded "wrongly", or whether it was encoded "correct" but now can't be decoded "correctly", the end result is always the same. You can't get the correct output now!)

Weird that it uses such system if it isn´t perfect as these stuff are supposed to Always work in All scenarios.

Personally, I don't use Lagarith extensively. So I'm just trying to sum up the issues that some people have reported. There also seem to be many users which use Lagarith with no problem at all.

Also keep in mind that the "perfect" (error free) software doesn't exist in reality. We usually accept that a "high quality" software has about one bug per 1000 lines of code ;)

[zerowalker]

But if the error occurs (Artifact) is it always there, or does it appear from time to time depending if the decoder fails to reach the correct number?

And well yeah "error free" is more of a term what should be striven for. But something that´s fundamentally wrong like this, isn´t what i would expect from a Codec.

[LoRd_MuldeR]

But if the error occurs (Artifact) is it always there, or does it appear from time to time depending if the decoder fails to reach the correct number?

That's the "nice" thing about undefined behavior: Everything is possible. You just don't know what is going to happen. And the result can (but doesn't have to) be different each time :scared:

[zerowalker]

Well if the Possibility exist, then you can at least Save the content by converting it. If the error Always occurs that´s impossible.
But i find it weird that i myself have never been inflected by this, i tend to use Lagarith quite a lot.

[qyot27]

you can´t even compare it with UT Video Codec which is the rival i think.
No, the closest 'rival' to Lagarith is actually FFV1. Ut Video mostly lies in the chasm between HuffYUV and Lagarith.

FFV1, for the record, also uses arithmetic coding (it's integer-based, though, not floating point). It loses to Lagarith in a couple of areas though - mainly solid color frames* and things like line drawings (and speed; Lagarith is faster than FFV1). FFV1 also supports all those high bit depth pixel formats.

*unless you use the lower efficiency VLC coder for FFV1 - then it actually gets close to Lagarith for solid color frames (because it switches to inter-frame? that's...unexpected, and feels wrong - it's what VirtualDub seems to report, though). Lagarith stores solid color frames as flags and presents them to the decoder as real intra frames.

Case in point, a simple BlankClip script (RGBA, 10 seconds, 640x480 @ 24 fps):
Lagarith: 15 KB
FFV1 (-coder 1 -context 1): 1.85 MB
FFV1 (-coder 1 -context 0): 1.85 MB
FFV1 (-coder 0 -context 1): 70.1 KB
FFV1 (-coder 0 -context 0): 70.1 KB

-coder 0 = VLC
-coder 1 = AC

[foxyshadis]

Well if the Possibility exist, then you can at least Save the content by converting it. If the error Always occurs that´s impossible.
But i find it weird that i myself have never been inflected by this, i tend to use Lagarith quite a lot.

It's more that floating point isn't entirely perfect, so things might be +1 or -1 from where they should be when decoded; those errors can add up, but they're pretty rare and hard to notice usually. It has C, MMX, SSE, and SSE2 versions of a lot of functions, all of which may have different rounding characteristics; it's impossible to tell since there's no testbed. Then again, there might just plain plain old bugs, too.

[LoRd_MuldeR]

It's more that floating point isn't entirely perfect, so things might be +1 or -1 from where they should be when decoded; those errors can add up, but they're pretty rare and hard to notice usually.

But for an entropy coder it can mean everything. Even the slightest difference in the value may cause a number to fall into a different range/bucket, and then the decoded symbol might be a completely different one.

And because the probability model is usually updated after each symbol, one wrong symbol can cause all future symbols to be wrong too :scared:

BTW: This is also the reason why, when dealing with floating point numbers, we never check for equality directly, but instead assume the numbers are "equal" if the absolute difference is below a certain threshold.

[foxyshadis]

But for an entropy coder it can mean everything. Even the slightest difference in the value may cause a number to fall into a different range/bucket, and then the decoded symbol might be a completely different one.

And because the probability model is usually updated after each symbol, one wrong symbol can cause all future symbols to be wrong too :scared:

BTW: This is also the reason why, when dealing with floating point numbers, we never check for equality directly, but instead assume the numbers are "equal" if the absolute difference is below a certain threshold.

The entropy coder portion is all integer and only done in C. That at least doesn't seem to be the problem.

[LoRd_MuldeR]

Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth. The format should generally be avoided if possible.
The entropy coder portion is all integer and only done in C. That at least doesn't seem to be the problem.

:confused:

[SirLagsalot]

So why optimize codec for them and not for the world of "mere mortals"?
I expect that high-bitdepth will become more common over time with consumer-level gear, so I would like to aim Lagarith where the technology will be.


Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth.

This is not correct. Lagarith uses an integer-only range coder to handle the compression, floating-point is only used to scale the symbol probability tables to a power of 2. Floating point math on the x86/x64 is deterministic for a given input and order of operations, so it does not randomly introduce error. To my knowledge, there has only been one error related to the use of floating-point math, and that was caused when the calling program changed the floating-point rounding mode or precision. Lagarith now checks the floating-point state before each frame, and adjusts it if need be to prevent this. I do plan to remove floating-point math in the rewrite though, in order to simplify porting and hopefully kill off that myth.

[LoRd_MuldeR]

This is not correct. Lagarith uses an integer-only range coder to handle the compression, floating-point is only used to scale the symbol probability tables to a power of 2.

Well, as long as the encoder and the decoder need to do this step in the exactly same way, things could go still go wrong and break decoding, right?

Floating point math on the x86/x64 is deterministic for a given input and order of operations, so it does not randomly introduce error. To my knowledge, there has only been one error related to the use of floating-point math, and that was caused when the calling program changed the floating-point rounding mode or precision. Lagarith now checks the floating-point state before each frame, and adjusts it if need be to prevent this. I do plan to remove floating-point math in the rewrite though, in order to simplify porting and hopefully kill off that myth.

Hmm, to my knowledge there's at least the classical "x87" FPU with 80-Bit internal precision and the newer SIMD (MMX, SSE, etc) instructions with "only" 64-Bit precision. Even if all variables are 64-Bit (double) precision in memory, the compiler may still keep intermediate results in registers, which would then be either 80-Bit or 64-Bit. Furthermore, even if you don't use SIMD assembly/intrinsic explicitly, compilers may still generate such instructions when targeting CPU's with MMX/SSE support. Finally, there are various compiler settings that effect floating point math ("fast math" option, etc). So there's at least the danger of "portability" issues when the encoder/decoder are compiled with different compilers (compiler configuration).

[SirLagsalot]

Well, as long as the encoder and the decoder need to do this step in the exactly same way, things could go still go wrong and break decoding, right?

No, the code and the inputs are identical, so the output will be identical as long as the same floating-point state (rounding mode and precision) are used.


Hmm, to my knowledge there's at least the classical "x87" FPU with 80-Bit internal precision and the newer SIMD (MMX, SSE, etc) instructions with "only" 64-Bit precision. Even if all variables are 64-Bit (double) precision in memory, the compiler may still keep intermediate results in registers, which would then be either 80-Bit or 64-Bit. Furthermore, even if you don't use SIMD assembly/intrinsic explicitly, compilers may still generate such instructions when targeting CPU's with MMX/SSE support. Finally, there are various compiler settings that effect floating point math ("fast math" option, etc). So there's at least the danger of "portability" issues when the encoder/decoder are compiled with different compilers (compiler configuration).
Lagarith uses the 53-bit precision mode for floating point, so none of those variations in potential precision end up affecting the result. It also checks and sets the floating point state if need be, so even if Lagarith was compiled with the wrong settings by someone else, it would be corrected at run-time.

[ChiDragon]

Sure, I can check if it is using the full bit-depth.

Here you go, a VHS sample (http://chidragon.thedessie.com/Doom9/titanic-vhs-sample1.rar). Possibly not all that useful for you: WinRAR gave it a steady 51% compression, so I'm guessing it is 9-bit. That would also make sense because a guy opened up the next model in this line of AV syncs and found a Philips SAA7113 was the ADC.

[zerowalker]

So wait, does Lagarith not have the problem and just some people, or are you talking about something else (Lagalot?)

[SirLagsalot]

Zerowalker, I think early on some people were seeing glitches from other causes and assuming it was due to floating point math errors; and then the rumor stuck around long after the actual bugs were fixed.

[zerowalker]

Oh, well let´s hope that´s the case, really like the codec and some fundamental error like that would really ruin it.

[zerowalker]

It seems the "Floating Point" errors actually does exist, at least from my understanding in my discussion with an ffmpeg developer.

Not entirely sure, but if am understand things correctly, in normal circumstances the errors shouldn't appear.
But if it's not an x86 system it will, though that doesn't tell me much, as i am sure x64 is still an x86 system;S

The original codec implementation uses floating point arithmetic which will fail on processors != x86 and might fail if you use another compiler. FFmpeg's implementation is fixed-point (which is what you expect for a lossless codec) meaning you can compile it for any hardware with any (non-broken) C compiler and you will always get correct (bitexact) output. Fixed-point arithmetic is often slower than floating-point on typical hardware.
See also (for example):
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2009-September/079680.html
http://mod16.org/hurfdurf/?p=142



I personally have never seen any corrupt frames that i can think of, and i have been using Lagarith quite a bit, so i think i should have encountered it by now.
Though again, that depends what a "corrupt frame" is, if it's 1 pixel that's wrong, then it's not possible i have noticed it, but if it's an entire corrupted imaged, that's another story.

[Kein]

Anyone by any chance has some speed stats mt LAGS 1.3.27 vs Huffyuv?

[poisondeathray]

Anyone by any chance has some speed stats mt LAGS 1.3.27 vs Huffyuv?

http://forum.doom9.org/showthread.php?p=1668292#post1668292