Lagarith lossless codec [Archive]

Zarxrax

2nd December 2004, 03:10

I'm announcing this for my friend since he's never joined this forum. He has made a very nice lossless codec called Lagarith. It supports a number of colorspaces (RGB24, RGB32, RGBA, YUY2, and YV12). It runs at about half the speed of huffyuv, so it can work well for capturing video on most systems. The compression ratio is slightly better than corepng (I didnt enable delta frames in corepng in my test).
It is an interframe codec (all keyframes) so it works very well for editing. It does have a null frames option though, which you can enable to make it encode null frames if there is no difference from one frame to the next. It's also open source.
A reduced resolution mode is currently being implemented, which will be a extremely nice feature for video editors.

I personally use this as my standard lossless codec now, replacing huffyuv and vble. From all the lossless codecs that I have tried, it offers the best mix of compression, speed, and stability, in my opinion.

For more information and to download it, go here:
http://lags.leetcode.net/codec.html

Mug Funky

2nd December 2004, 05:11

looks good. you can never have too many lossless codecs :)

btw, love the avatar :)

akupenguin

2nd December 2004, 06:39

Cool, a new codec is always good news.

Nitpick posted here because I don't see any method of feedback on the site:
from http://lags.leetcode.net/codec.html
Pixel values are first predicted using Paeth prediction (the same method used when "Predict Median" is selected in Huffyuv). Huffyuv's prediction (which Lagarith also uses) is "median(L,T,L+T-LT)". Paeth (used in e.g. PNG) is "whichever of L,T,LT is nearest to median(L,T,L+T-LT)".

Soulhunter

2nd December 2004, 08:15

Wow, very nice codec... :)

Bye

akupenguin

2nd December 2004, 08:29

Since the compression method sounds similar to ffv1, lets see how they compare.

My benchmarks on a random chapter of a random DVD I had on hand (Ghost in the Shell 2; mostly CG, cropped but not filtered)
YUY2 (because that's what all these codecs have in common), 704x384, 2000 frames
There were no duplicate frames in this clip, so that feature of lagarith wasn't tested. But note that MEncoder has a similar duplicate frame remover that can be applied to any codec, so ffv1, ffvhuff, and ljpeg would benefit the same amount.
This is just one source; YMMV.
Codec Bitrate Enc fps Dec fps
raw: 103705
huffyuv 2.1.1: 31524 93.41 71.61
ljpeg: 28965 44.85 45.63
huffyuv ccesp: 28170 96.48 72.54
ffvhuff: 24308 86.81 70.42
lagarith: 23242 41.08 29.88
ffv1(cavlc): 19756 27.99 26.31
msu: 17676 0.83 10.63
ffv1(range coder): 17221 23.23 25.77

TEB

2nd December 2004, 19:06

good codec.. very stable..

but its about 20% worse at enc than ffv1 and around 3% slower at encoding.

Does anyone know how to force the encoder into 422 or 420 mode?

Shinobu

2nd December 2004, 23:09

A multi thread implmentation will be cool, i can beta test it (dual athlon mp on tyan thunder k7).
Also a "compression level" will be realy cool (don't know if it's easily implementable) so every computer could use it for capture ^^.

++

Atamido

3rd December 2004, 08:22

AFAIK, FFV1 only works in YV12, so if you're feeding it anything except that you are not encoding lossless. An AVISynth script could compare the original to decoded FFV1 to make sure.

Also, CorePNG with Auto-Delta frames at the maximum compression level would likely beat all of these for size, but unless you have an Athlon 30,000+, you aren't going to be capturing with it in real time.

akupenguin

3rd December 2004, 18:20

FFV1 works in YUV of any subsampling ratio (4:4:4, 4:2:2, 4:2:0, 4:1:1, 4:1:0) and also RGB. Yes, I made sure all of the encodes in my test are lossless.

akupenguin

4th December 2004, 04:06

Lagarith doesn't seem to be affected by the "null frames" option when encoding YUV. Is it supposed to be working? (Yes, I'm sure the frames are perfectly identical.) I also can't get it to encode RGB (vdub crashes when I begin encoding).

edit: SirLagsalot just fixed RGB.

Another chapter of another random DVD ("Ghiblies 2", 672x432, 2500 frames, cropped but not filtered).
Yes, I made sure all of these are really lossless.

YV12:
Codec Bitrate Enc fps Dec fps
raw 83529
corepng(6i) 23903 7.31 47.49
ffvhuff 22264 156.46 86.09
lagarith 20975 44.79 32.89
corepng(9p) 18007 0.68 51.09
ffv1 15368 28.62 29.80
snow 13838 7.46 15.43

With duplicate frame removal:
ffvhuff 18440 163.06 104.61
lagarith 17346 50.89 39.72
snow 13685 8.50 16.76
ffv1 12737 33.93 36.09

RGB:
Codec Bitrate Enc fps Dec fps
raw 167048
hyffyuv ccesp 54583 48.79 59.29
corepng(6i) 52360 3.82 35.49
corepng(6p) 41752 2.29 28.24
corepng(9p) 40122 0.55 29.04
lagarith 39539 19.42 17.56
ffv1 29937 13.86 14.76

With duplicate frame removal:
huffyuv ccesp 45388 58.86 71.53
lagarith 32771 23.93 20.80
ffv1 24891 16.68 18.00

where (6i) means CorePNG 0.8.2 on defaults, and (9p) means best + auto-delta.

CruNcher

4th December 2004, 16:28

2605 1:44 Pal 25 Fps 720x576 (Real) (cleaned DVD)

flawed look below for correct results

Zarxrax

4th December 2004, 18:14

After seeing the good results of ffv1, i decided to try a comparison myself, and see if I could actually get the codec working this time. I've never been able to get either ff huffyuv or ffv1 working the couple times that I tried them in the past.

Video is YV12, animated source, 1:40 in length.

Codec/settings Encode Speed Size
vble 0:48 395MB
lagarith 1:15 330MB
ffv1 vcl Context 0 kf dist 1 1:47 281MB
ffv1 ac Context 0 kf dist 1 2:40 268MB
ff huffyuv (resulting files are corrupt)

Well, at least ffv1 works now.
My results basically show a pretty linear relation between the amount of compression verses the encoding speed. Lagarith seems to be a good intermediate range codec: good compression, and good speed.

Atamido

4th December 2004, 19:21

Originally posted by akupenguin
FFV1 works in YUV of any subsampling ratio (4:4:4, 4:2:2, 4:2:0, 4:1:1, 4:1:0) and also RGBA. Yes, I made sure all of the encodes in my test are lossless. Interesting, I wonder when they added that? But if you checked to make sure the output was bit identical, it's all good. :)

akupenguin

4th December 2004, 21:01

Originally posted by Zarxrax
ff huffyuv (resulting files are corrupt)

What version of libavcodec? There have been some recent fixes and improvements. Last week it was split into two codecs: "ffhuffyuv" is compatible with the the original, and "ffvhuff" contains our improvements (YV12, adaptive huffman tables).

For best results, use vcodec=ffvhuff:vstrict=-1:pred=2:context=1

And you've tried ffv1 with context=1 and keyint=10 or 20 and decided that the compression gain is not worth the speed or the addition of P-frames? (FFV1 isn't actually temporally predicted, it just uses P-frames to avoid resetting CABAC contexts.)

Zarxrax

4th December 2004, 22:00

Originally posted by akupenguin
What version of libavcodec? There have been some recent fixes and improvements. Last week it was split into two codecs: "ffhuffyuv" is compatible with the the original, and "ffvhuff" contains our improvements (YV12, adaptive huffman tables).
My ffdshow is dated Dec 1, 2004.

Originally posted by akupenguin
And you've tried ffv1 with context=1 and keyint=10 or 20 and decided that the compression gain is not worth the speed or the addition of P-frames? (FFV1 isn't actually temporally predicted, it just uses P-frames to avoid resetting CABAC contexts.)
I actually wasn't sure what the context setting did, so I left it at the defaults, which I assumed would result in the best compression. I didn't enable P-frames because the other codecs that I tested do not support this feature, thus it would not be a fair comparison. Also, P-frames are worthless for the applications that I typically use lossless files for, so it does not interest me.

akupenguin

5th December 2004, 02:09

Looks like ffvhuff hasn't been included in ffdshow yet.

Originally posted by Zarxrax
I actually wasn't sure what the context setting did, so I left it at the defaults, which I assumed would result in the best compression. In all of lavc's codecs, the defaults are near the fastest settings. I don't know if this is intentional, or if they just didn't want to change the default behavior when adding a new feature.
I didn't enable P-frames because the other codecs that I tested do not support this feature, thus it would not be a fair comparison. What kind of reason is that? Different codecs have different features. You don't turn off FFV1's Arithmetic coding just because Huffyuv doesn't support it.
Also, P-frames are worthless for the applications that I typically use lossless files for, so it does not interest me. OK, if you need frame-accurate cutting and seeking. It was only about a 5% improvement.

RadicalEd

5th December 2004, 09:23

I've managed to get intra-only FFV1 to encode and decode at the same rates and faster compared to Laga, with decent compression gain. I hate to be such a dick, but I've never really seen the point of lagarith... the algorithms are pretty mundane, and it doesn't really fill any niche.

I'll post my test results tomorrow or Monday when I'm home.

CruNcher

5th December 2004, 12:14

Sorry but the results i posted before were borked here are the correct ones

2605 1:44 Pal 25 Fps 720x576 (Real) (cleaned DVD)
http://cruncher.mufflastig.com/real.png

size enc dec
ffv1 (AC) 293 MB 2:46 2:28/19 Fps
ffv1 (VLC) 316 MB 2:06 2:00/22 Fps
lagarith 356 MB 1:58 2:05/22 Fps
vble-gr3 368 MB 2:57 1:33/28 Fps
vble-gr2 369 MB 7:01 1:48/24 Fps
milk 371 MB 3:16 3:05/15 Fps
vble2-gr2-lite 375 MB 2:57 2:00/22 Fps
loco 401 MB 1:55 2:11/21 Fps
vble 416 MB 1:20 3:16/13 Fps
huffyuv12 median 468 MB 1:22 2:22/20 Fps
ffdshow huffyuv12 median 689 MB 1:50 2:50/18 Fps

4545 3:09 Film 24 Fps 640x480 (CG)
http://cruncher.mufflastig.com/cg.png

size enc dec
ffv1 (AC) 297 MB 3:40 2:51/30 Fps
ffv1 (VLC) 325 MB 2:46 2:20/33 Fps
milk 370 MB 4:00 3:17/23 Fps
lagarith 390 MB 2:28 2:20/33 Fps
vble-gr2 390 MB 9:20 2:10/36 Fps
vble-gr3 393 MB 3:47 1:50/38 Fps
vble2-gr2-lite 407 MB 3:56 2:26/33 Fps
loco 427 MB 2:33 2:32/32 Fps
vble 520 MB 1:50 3:47/20 Fps
huffyuv12 median 556 MB 1:48 3:14/24 Fps
ffdshow huffyuv12 median 872 MB 1:50 2:40/28 Fps

dragongodz

5th December 2004, 13:15

Lagarith doesn't seem to be affected by the "null frames" option when encoding YUV. Is it supposed to be working?
did you use version 1.2.4(updated 12-04-04, same build that has the RGB fix)) to test that ?

I hate to be such a dick, but I've never really seen the point of lagarith... the algorithms are pretty mundane, and it doesn't really fill any niche.
whats that to you ? the guy wanted to write a lossless codec and did. he has made it available for anyone that wants to try it or use it etc. he even made it open source so if anyone wants to improve it they can do that aswell. its not up to you or anyone else to say its pointless because it has cost you nothing and you are not being forced to use it.

RadicalEd

5th December 2004, 18:39

Well, if you like it, use it. I still don't see any practical use, FFV1 pretty much covers all bases. I guess I'm just too supportive of FFV1 :P Laga and me go back a ways though. The story of my distaste for it is too long and political to go into without being OT, so I'll step off here.

akupenguin

5th December 2004, 22:15

Originally posted by dragongodz
did you use version 1.2.4(updated 12-04-04, same build that has the RGB fix)) to test that ?

I just redid that test with 1.2.4, and null frames still has no effect on YUV.
(When I reported the compression with duplicates removed, that was using MEncoder's decimate filter, not Lagarith's.)

BlindWanderer

6th December 2004, 01:09

*poke*
1.2.5 released.
"Bug fixes: Released version did not mantain a copy of the previous frame for YV12/YUY2, resulting in null frames not working for those color spaces."

Atamido

8th December 2004, 15:52

Originally posted by akupenguin
Yes, I made sure all of these are really lossless. Would you provide an AVISynth script for testing this? I want to run some tests, but I'm really don't feel like figuring out AVISynth yet again.

akupenguin

8th December 2004, 18:43

I did it using "mplayer -vo md5sum", but if you want to use avisynth, the script could be
compare( avisource("raw.avi"), avisource("ffv1.avi"), logfile="ffv1.log" )

morsa

9th December 2004, 05:42

Could be Lagarith be modified to accept Bayer mosaic video?
For example a 1920x1080 Bayer mosaic can be decomposed to 4 960x540 planes as follows:

1) Odd lines Green
2) Even lines Green
3) Red
4) Blue

TEB

9th December 2004, 14:32

Anyone know how to set Snow to lossless mode? I cannot seem to find anyplace to do that. FFDSHOW from 5.12.2004.

My conclusion is that FFV1 AC mode is almost twice as good compression as lagarith.. and almost the same speed at enc and dec... Will test som more...
FFV can infact be an alternative to mpeg2-50mbit I-frame only 4:2:2 storage for video archives.. I tested some educational films at work and they were 40% smaller in FFV lossless mode 4:2:2 than mpeg2 HP@HL 50mbit i-fram only !! in other words.... that ROX!!! keep up the good work.

Tommy Carrot

9th December 2004, 14:49

Originally posted by TEB
Anyone know how to set Snow to lossless mode? I cannot seem to find anyplace to do that. FFDSHOW from 5.12.2004.
Just set the quality to 100.

Speaking of Snow, it's already quite good, often better than ffv1, but it could reach even better compression ratio if it would have a good frame-type decision algorithm implemented, because in lossless mode using keyframes instead of predicted frames could often save a lot of bits, especially in fast motion scenes, so the fixed frametype-pattern seriously hinders the lossless performance (and the lossy too to a lesser extent).

plonk420

19th December 2004, 13:10

dunno about you guys but i was getting 40FPS encoding to FFV1 (and the output was garbled, too). OTOH, i'm getting 120FPS encodign to Logarith and a projected filesize 1/6 the size of the FFV1...

PlazzTT

16th January 2005, 18:00

Version 1.3.1 is out now: http://lags.leetcode.net/codec.html

I'm having trouble with the new version. The old version I was using (1.2.4 I think, there was no way of checking the version number of it) was working great.

I installed the new version, and the codec seems to encode okay. I'm using VirtualVCR to capture live TV. But I can't seem to decode, view or open the encoded AVI afterwards. The 1.3.1 decoder can decode 1.2.4 clips okay, so the problem must be with the encoding.

The 1.3.1 AVI I have crashes MPC and Vdubmod. Using AviSynth's AviSource to read it, I get "Evaluate: Unrecognized exception!"

I opened a working 1.2.4 in GSpot and it says the container is OpenDML (AVI v2.0). Fine.

But when I open the 1.3.1 clip in GSpot, it says:
Multipart OpenDML AVI
(5807 frames in first part, 3 frames follow) :confused:

Maybe the problem is with VirtualVCR? Either way, does anyone know how I can open this AVI I recorded. It's not a very important clip to me, but I've been recording with 1.2.4 ever since.

TEB

16th January 2005, 21:00

1.3.2 is out fixing some crashing bugs :)

Sir_Lagsalot

16th January 2005, 21:30

Theres now also a recovery version of the codec for corrupted video from 1.3.0/1.3.1, sounds like the error you're getting: http://lags.leetcode.net/Lagarith_131_recovery.zip

PlazzTT

16th January 2005, 22:05

Originally posted by Sir_Lagsalot
Theres now also a recovery version of the codec for corrupted video from 1.3.0/1.3.1, sounds like the error you're getting: http://lags.leetcode.net/Lagarith_131_recovery.zip

Excellent! It works. Thanks.

EDIT: At least it seems to work. How slowly is this fix meant to decode? It seems to stop doing anything after 2 or 3 frames (Vdub goes to 0.0fps for at least 5 minutes)

Sir_Lagsalot

16th January 2005, 23:00

It should still get a few fps... Try downloading it again (I tweaked it a little), make sure multithreading is disabled, and make sure you are encoding to a different codec. If it still gets stuck, try manually removing the frames it stops on in virtualdub so they don't get decoded.

PlazzTT

16th January 2005, 23:06

Yep, much better now! I'm getting 9-10fps encoding 720x576 to Huffyuv.

Longinus

11th February 2005, 01:15

Hi!
Thanks for this codec. It's really good.
And it works... something I can't say for FFV1 and MSU.

I'm encoding 2K material (2048x1556) in RGB. FFV1 just creates a unusable video (if I set it to YUV, it works... but RGB to YUV conversion is so lossy that my eye hurts :p). And MSU and Alparysoft crashes in the middle of the encode.

CorePNG works, but it's slower than your codec... and I'm having problem importing it in Premiere. And your codecs compress a tiny little better than corepng (3D cel-shade material).
Huffyuv also works, but compress a LOT worse than both codecs.

Again, thanks for a very nice codec!

Longinus

11th February 2005, 03:39

Oohhh, I have one request, If I may...

The reduced resolution function is useafull, but not as fast as I wish it was. Perhaps because of the resize method? That seams to be some kind of bilinear resize.
What about an option to make a faster decode, with Nearest Neighbor or something like that for the resize?

Thanks!

Zarxrax

20th March 2005, 21:35

Lagarith 1.3.4 is released.

Now decoding speed has increased, and the resize method for reduced resolution mode is also faster.

Also, version 1.3.3 added the ability to downsample the colorspace when encoding, to use less space.

I tested the decoding speedup myself. The old version decoded a clip at an average speed of 18.4 fps. The new version decoded the same clip at an average speed of 21.1 fps. About 15% faster decoding speed on my clip.

http://lags.leetcode.net/codec.html

dragongodz

13th March 2006, 11:56

i notice lagarith is up to version 1.3.8 now so thought it may be worth mentioning here.

change log since 1.3.4

Version 1.3.8 released on 03-11-2006
Fixed a bug that caused Lagarith to attempt to decode any file encoded with an old version as YV12 when asked to suggest the format (affected AviSynth). Thanks to Zarxrax for reporting this error.
Fixed a bug that could cause video corruption or crashes when encoding certain types of YV12 source video.
Changed threading calls and synchronization methods; this may improve performance slightly when multithreading is used.
Merged various changes from the AMD 64 build back into the base code; hopefully I will be able to merge the builds as the AMD 64 version matures.
Support for non-MMX processors has been ended completely.

Version 1.3.7 released on 11-12-2005
The codec now writes extra information in the format header to indicate which mode the codec used when encoding. This allows the codec to decode to the format used internally even when 'Report Original Colorspace' is used.
Several other minor code changes and small optimizations.

Version 1.3.6 released on 05-26-2005
Decoding RGB24 as RGB32/RGBA now sets the alpha channel to 255 instead of 0; previously, such video would show up as solid black in Vegas because it was perceived as fully transparent (thanks to Stephen Schlueter for reporting this error).
Fixed RGB24 encoding for systems that do not have an MMX capable processor.
Fixed a bug that prevented encoding RGB32 as RGB24 when the "Report original colorspace" option was checked.

Version 1.3.5 released on 05-28-2005
Fixed a bug that caused solid color frames to only decode partially when exporting RGBA.
Added exception handling when freeing memory.

smok3

13th March 2006, 13:39

can somebody nice post some command lines to use for RGB and RGBA material, using mencoder and its lossless codecs on win32?

(and the same for decoding)?

akupenguin

13th March 2006, 21:22

mencoder -ovc lavc -lavcopts vcodec=ffv1:vstrict=-2:coder=1:context=1:format=bgr32 in.avi -o out.avi

Note that despite the "bgr32", it doesn't store an alpha channel.

smok3

13th March 2006, 21:26

a. tnx a lot, so i guess for huff it is similar? like :
vcodec=huffyuv:format=bgr32 - will that store alpha as well?

---

b. another question, what version of ffdshow should one get (and where?) for proper win playback of ffv1?

---

c. and another thing concerning mplayer and avs, is it possible that mplayer forces avisynth to always return only yv12?
(i was trying to use '-vo md5sum' to find out that:
original != ffv1 when
huff = ffv1
p.s. when i compare using avisynth compare function, then:
original = huff
at least i cant seem to find any other explanation.... )

akupenguin

14th March 2006, 11:38

a. ffmpeg's huffyuv does rgb decoding, but can only encode yv12 or yuy2.
b. ffv1 hasn't changed in a while, so any build >= 2005 february should work. I can't recommend one in particular.
c. Yes, mplayer invokes ConvertToYV12() if the avs isn't already. There's no particular reason for this, just whoever is in charge of demux_avs.c never implemented other colorspaces.

Ahmedbrave

5th September 2006, 09:42

I been tested Logarith codec (ver. 1.3.9) wich I downloaded from site (http://lags.leetcode.net).

And found that it have the same issue as 1.3.8 ver.: a bug that allowed YUY2 and YV12 video to be decoded as RGB16.

So i looked at files in archive wich I downloaded more carefully (it`s name was Lagarith_139.zip): in properties of lagarith.dll i looked at version, it was 1.3.8!!!

May be author made a wrong version build?... (or just uploaded a wrong file?...)
stupidies marked by gray...
i`m sorry.. codec work fine.. mixed out some thing...

wereHamster

7th September 2006, 09:47

What does lagarith do differently then huffyuv? Prediction? huffman encoding? I'm using a modified huffyuv codec with the median prediction and a simplified huffman table (one table for all three YUV planes) and it's working fine, but maybe I could 'borrow' something from lagarith to make my codec faster?

Ahmedbrave

7th September 2006, 13:07

wereHamster
...don't really know, what different, anyway compression is grater, but encoding is slower.

video_magic

13th November 2006, 22:57

Version 1.3.11 released on 11-12-2006

http://lags.leetcode.net/Changes.html
http://lags.leetcode.net/codec.html

akupenguin

14th November 2006, 01:01

What does lagarith do differently then huffyuv? Arithmetic coding instead of huffman.

TEB

14th November 2006, 21:10

Hi. I did some very rudementary tests here on my DualCore Athlon 4200 in VDUB 1.6.17 DVCAM AVI file 4:2:2 -->

Lagarith 1.3.11 RGB24 SMP 42 Sec 328MB
Lagarith 1.3.11 RGB24 NON-SMP 53 Sec 328MB
Lagarith 1.3.11 YV12 SMP 37 Sec 165MB
Lagarith 1.3.11 YV12 NON-SMP 38 Sec 165MB
UNCOMPRESSED RGB32 36 Sec 2324MB
MSU 0.6 Balanced, yv12 4:01 167MB
FFV1 RGB32 VLC 2.07 301MB
FFV1 RGB32 AC 2.05 297MB
FFV1 YV12 VLC 1:12 181MB
FFV1 YV12 AC 1:12 176MB
HUFFY YV12 : 35sec 522MB

Note:

The files from lagarith differ in size based on using SMP or not, just a 200-300bytes, but still worrying

Anyway, Lagarith in yv12 is VERY impressive imo. ITs just a 2 sec slower than huffy and 3times as efficient

This gotta be the worst comparison ever but i thought id share my results ;)

teb

TEB

14th November 2006, 21:31

btw, anyone know the parameters for running x264 in lossless mode?

Inventive Software

14th November 2006, 22:19

Constant Quality mode 0. Think the commandline's:

x264.exe -q0

But search for it, or do:

x264.exe --help

totya

17th November 2006, 01:53

This is very-very good codec, fast and result small file, uses 2 processors too... But very often freeze if "use multithreaded" is checked. I use only YUY2 mode.

Well I see, new version 1.3.11, Good news. :)

Problem with new codec (and last version) : codec uninstall not working (under programs add/remove)

Sorry my poor english, and thanks of developer : this is very good lossless video codec.

johnsonlam

17th November 2006, 04:26

I mean, with HuffYUV.

Still lots of people didn't know Lagarith, a direct comparison table can easily let them choose the codec they want. I'm capturing with Lagarith and it works great.

TEB

19th November 2006, 20:43

Lagarith 1.3.12 is out ;)

johnsonlam

25th November 2006, 20:36

Lagarith 1.3.12 is out ;)

I found that if using Largarith with some filter combination in Vdub, the output will have a lot of vertical and horizontal color lines, up to now I can't exactly reproduce it, seems it happen under certain criteria ...

Will report here if I have any progress.

Chainmax

14th December 2006, 02:38

I recently used Lagarith v1.3.12 to make an intermediate file. The process went through perfectly, but when trying to make the final MPEG2 I get two consecutive error messages:

http://img213.imageshack.us/img213/1508/errmsgsok9.png (http://imageshack.us)

That happens when loading the plain intermediate AVI file and a simple script with a couple extra filtering steps. Any ideas as to what could be causing this? I'm using AviSynth_160906 and VDubMod v1.5.4.1b2066.

johnsonlam

18th December 2006, 11:02

I recently used Lagarith v1.3.12 to make an intermediate file. The process went through perfectly, but when trying to make the final MPEG2 I get two consecutive error messages:

That happens when loading the plain intermediate AVI file and a simple script with a couple extra filtering steps. Any ideas as to what could be causing this? I'm using AviSynth_160906 and VDubMod v1.5.4.1b2066.

Sometimes many filters and scripts together will conflict, did you try removing the filter one by one, or just replacing Lagarith with Huffyuv to see still crash or not?

video_magic

21st July 2007, 09:23

Lagarith codec updated:

http://lags.leetcode.net/codec.html

totya

21st July 2007, 10:29

Lagarith codec updated:

http://lags.leetcode.net/codec.html

Good news, thank you!

Fixed a bug that would cause crashes when downsampling to YV12. - This is occurred me with v1.3.13.

GrofLuigi

14th April 2008, 16:27

Codec updated:

Homepage (http://lags.leetcode.net/codec.html)

Changelog (http://lags.leetcode.net/Changes.html)

GL

LeeWilson

1st May 2008, 04:30

Hi,

I am currently trying to implement AVI support into a OpenGL game engine. I have successfully got the engine to decompress AVI frames and render them in RGB. For various reasons I really need to get the alpha channel. I cannot seem to get a reliable AVI file, with alpha and Lagarith compression. I tried VirtualDub-1.7.8 and it seems that it does not support alpha channels in its conversions. I tried the Color Depth option for 32 bit RGB (888) (dummy alpha channel) and it made my movie go all semi transparent with vertical black lines. I also tried creating an uncompressed .mov with alpha, and using the movtoavi from http://lags.leetcode.net/Codec_mac.html and the avi result was black and white gibberish.

Does anyone know a reliable way of creating Lagarith compressed AVI's with the alpha channel?

Cheers in advance,

Lee

Inventive Software

1st May 2008, 14:43

Firstly, try VirtualDub 1.8. It's unstable, but has better colourspace support. Secondly, in Lagarith's config, select RGBA as the output.

LeeWilson

6th May 2008, 04:24

Hi, thanks for the reply.

I tried VirtualDub 1.8.0 with the same results as 1.7.8. It still says Dummy Alpha Channel in the color depth options for 32 bit RGB. And I have the mode set to RGBA in the Lagarith configuration.

I suppose I will have to look writing an app to compress my own avi from images or uncompressed avi with Lagarith.

[P]ako

6th May 2008, 16:14

Avisynth supports RGBA and you could use avs2avi (http://forum.doom9.org/showthread.php?s=&threadid=71493) to encode it to lagarith, in that very same thread you can also find a GUI for it (avs2avidrop).

It just occurred to me, did you set virtualdub to work in fast processing mode?

Zarxrax

7th May 2008, 11:56

I have often created lagarith files containing alpha channel with Adobe After Effects or Adobe Premiere. I think free trial versions may be available. Avisynth can also generate video with alpha channel like [P]ako said, and thats probably the easier way to just test it.

video_magic

26th August 2008, 00:20

I have noticed that the Lagarith codec has been updated to version 1.3.18! Thanks for the updates Ben Greenwood

http://lags.leetcode.net/codec.html

totya

26th August 2008, 00:42

Hi!

Version 1.3.17, and 1.3.18 uninstall function is bad.

if I use lagarith.inf form 1.3.15, uninstall works.
Different is "setupapi.dll" - "setupx.dll"

My system is XP, 32 bit.

benwaggoner

3rd September 2008, 07:06

So, I'm very annoyed by the Adobe video apps (I've seen this in Premiere Pro and After Effects CS3) inabilty to actually take settings with Lagarith. I can choose the options like normal, but when I close the dialog and go back, it's always RGB without multithreading, and regardless of what I set the options to, the output is always RGB single-threaded.

Anyone know of any workaround for this? A reg key I can set for default behavior or something?

squid_80

3rd September 2008, 07:12

Take a look at c:\windows\lagarith.ini.

video_magic

5th September 2008, 23:16

I have been trying out the newest version of MLC and the newest Lagarith, and Lagarith has been my winner!
Capturing from a NTSC Laserdisc at 29.97 720x480 YV12 using virtualvcr and through S-Video to S-Video via BT878a based board. Capturing audio as PCM 44100 stereo.
My PC has a SL9KE chip which is a P4 3.2ghz HT 2mb and 1gb PC3200 2-2-2-5 ram. XP SP2 running clean and defragged.

39 minutes of video resulted in a 12.6GB file! :)

MLC was too slow to be usable even on the fastest setting. Lagarith is excellent!

Capture Stats
Frames Captured: 70261
Frames Dropped1: 0
Frames Dropped2: 1
Time : 00:39:04
Time Left: 07:08:19
Free Space: 148,803,293,184
Video Bytes: 13,163,442,847
Audio Bytes: 413,569,800
Total Bytes: 13,577,012,647
Video Bytes Sec: 5,613,729
Audio Bytes Sec: 176,372
Total Bytes Sec: 5,790,101
Video Compression: 2.767
Video Rate: 29.970013
Audio Rate: 44100.241432
AV Diff: -0.000005
AV Adjust: 0.000000
AV Actual: 0.000000

benwaggoner

10th September 2008, 17:29

Take a look at c:\windows\lagarith.ini.
I don't see any way to set default modes in there, but in my ~\AppData\Local\VirtualStore I see

[settings]
multithreading=1
lossy_option=3

Anyone know what the right numbers are for multithreaded and YV12?

squid_80

10th September 2008, 19:11

I don't see any way to set default modes in thereWell, wherever settings go when WritePrivateProfileInt/String is used.Anyone know what the right numbers are for multithreaded and YV12?
multithreading=1 means use multithreading (who would have guessed?).
lossy_option=3 means YV12. (0:RGBA, 1:RGB (Default), 2:YUY2, 3:YV12, 4:Reduced Resolution)
suggest=1 means always suggest RGB for output.
nullframes=1 means enable Null frames.

From the sounds of it this isn't going to fix your problem though.

benwaggoner

10th September 2008, 19:15

Bummer. So I've got it set correctly, but Premiere and After Effects are somehow overriding the defaults!

smok3

10th September 2008, 19:29

my workaround is to use huffyuv with adobe apps (for RGB and for RGBA), after the project is finnished i would reconvert those huffs to lags for backups.
(ask for bat which controls vdub.exe and converts *.avi to *_lags.avi))

henryho_hk

11th September 2008, 05:18

smok3, you can use the command-line utility avs2avi.exe (http://www.moitah.net/).

I've drafted a batch file (totally untested):

SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION

if "%~1"=="" (
echo Usage: "%~f0" <source avi>
echo Defaults to RGBA output
echo For RGB24, name it as "blahblah_rgb.avi"
goto :EOF
)

set HOMEDIR=%~sdp0
set WORKDIR=%~sdp1

set PIXTYPE=%~n1
set PIXTYPE=%PIXTYPE:~-4%

if /i "%PIXTYPE%"=="_RGB" (
set PIXTYPE=RGB24
) else (
set PIXTYPE=RGB32
)

set SRCFILE=%~sf1
set AVSFILE=%~sdpn1_%PIXTYPE%.avs
set DSTFILE=%~sdpn1_lags_%PIXTYPE%.avi
set LAGSCFG=%HOMEDIR%lags_%PIXTYPE%.cfg

echo AVISource^("%SRCFILE%",pixel_type="%PIXTYPE%"^)>"%AVSFILE%"

if exist "%LAGSCFG%" (
"%HOMEDIR%avs2avi.exe" "%AVSFILE%" "%DSTFILE%" -w -p 0 -l "%LAGSCFG%"
) else (
"%HOMEDIR%avs2avi.exe" "%AVSFILE%" "%DSTFILE%" -w -p 0 -s "%LAGSCFG%"
)

:EOF

smok3

11th September 2008, 08:24

henryho_hk: oh, sorry for confusion, i allready have a batch, what i meant is if somebody needs an example, i can provide. However, good batches are always welcome.

edit1: why do you need avisynth for this?
edit2: ok, i just realized my batches wont do RGBA to RGBA but for some reason it ends with RGB lags, must take a look into this..

henryho_hk

12th September 2008, 01:37

why do you need avisynth for this?

I use AVISynth because I use avs2avi. :devil: :devil: It seems difficult to fiddle with Virtualdub's (<1.6) VFW config in batch files. Mencoder supports VFW compression too but for unknown reasons it runs slower than avs2avi in my computer.

lilhobo

21st September 2008, 18:05

how can i install for vista64?? the bat wont work for me :(

henryho_hk

22nd September 2008, 07:01

how can i install for vista64?? the bat wont work for me :(

Sorry, I dun have any Vista (32 or 64) for testing. But you can send me the error messages (seems out-of-topic for this thread) and see if I can help.

lilhobo

24th September 2008, 23:32

well vista u need the admin rights or something which the bat doesnt give

LoRd_MuldeR

25th September 2008, 00:12

well vista u need the admin rights or something which the bat doesnt give

What if you run it from an Admin account?

lilhobo

25th September 2008, 11:31

the first user on a fresh vista install is the admin i would guess

foxyshadis

30th September 2008, 11:46

how can i install for vista64?? the bat wont work for me :(

Same way you install huffyuv in win64 (xp or vista): http://www.dvinfo.net//conf/what-happens-vegas/126319-installing-huffyuv-vista-64-bit.html

totya

10th October 2008, 10:11

Hi!

v1.3.19 uninstall function is bad under XP32 again (and again and again and again...)

I wrote this (http://forum.doom9.org/showpost.php?p=1175081&postcount=66) before.

Thx.

ipanema

23rd November 2009, 18:49

Does anyone know if Lagarith is still being developed? I submitted a bug report a couple of times to the author a few months ago but there was no reply.

I'll pasted the details below in the hope that he may see it here.

I'm getting crashes with the Lagarith encoder when it is configured to YUY2 or YV12 in its settings. This only happens when the input video size is 1280x720, there is no problem when it is 1920x1080.

I'm feeding 1280x720 RGB24 video frames from Mainconcept or Elecard H.264 decoders to Lagarith for encoding. Both of these H.264 decoders cause Lagarith to crash with an access violation at the first sample delivered - here is a stack traceback when using the Mainconcept decoder:

lagarith.dll!0b581b7b()
lagarith.dll!0b584e9c()
lagarith.dll!0b5852d2()
lagarith.dll!0b587863()
msvfw32.dll!6c2718e6()
msvfw32.dll!6c274fb7()
qcap.dll!6c32a394()
qcap.dll!6c349455()
qcap.dll!6c349840()
......

However, when the ffdshow H.264 decoder filter is feeding Lagarith then there is NO crash and the filter graph runs fine. Having looked at the media types it looks like the sample size and actualdatalength for ffdshow are set to

sample actual data length 2764800
sample size 3686400

but for Mainconcept and Elecard, the sample size and actualdatalength are BOTH set to 2764800.

I therefore wonder if Lagarith is making an incorrect calculation of the sample size giving it a size beyond 2764800 which leads to an access voliation for Mainconcept and Elecard; but not with ffdshow because its sample size is much larger at 3686400.

If lagarith is set to RGB in its options sheet then this also avoids the crash and the filter graph runs OK. So maybe lagarith is getting the data size incorrect when doing its internal RGB to YUY2 conversion? Strange that this doesn't happen for 1920x1080 sized video though.

I also noticed that one of the last fixes is "Fixed a bug that would corrupt video when downsampling to YUY2 and the resolution was not a multiple of 32". Notice that 720 is not an exact multiple of 32, so maybe this has something to do with the problem too.

Also if "Use multithreading" is selected in Lagarith's options when the above crash happens, the program seems to terminate but Task Manager shows that there is a thread still running inside Lagarith and using approx 40% of CPU time. The only way to get rid of this is the End Process button in Task Manager.

If "Use multithreading" is NOT selected then this vestigial thread is not left running and the process disappears completely from Task Manager's list.

weaker

23rd November 2009, 23:22

The homepage http://lags.leetcode.net/codec.html had an update in July 2009, so I guess it is still developed.

Jim_Pansen

24th May 2010, 15:12

Hi folks,

is there any chance to see the Lagarith Codec for MAC in future? :)

For easier handshakes on exchanging material PC <--> MAC it would be a great plus for the workflow.

THX a lot for developing this great codec.

Jim

johnsonlam

25th May 2010, 10:56

Mac OSX should have some lossless codec, both video and audio!

Jim_Pansen

25th May 2010, 20:21

Mac OSX should have some lossless codec, both video and audio!

My question was related to interoperability, not to the general question "Is there any losless MAC codec"!

Your answer doesn't match my question.

Jim

Boolsheet

25th May 2010, 22:59

I believe the lagarith source itself is Windows specific. I also read it uses a floating point arithmetic (or range?) coder, which makes portability less fun. Nathan Caldwell started a yv12 decoder for FFMpeg and they came up with something to emulate the problem with the entropy coder. The git of the fork is here (http://repo.or.cz/w/FFMpeg-mirror/lagarith.git). Looks like everyone is busy with higher priorities, so it's more likley the far future. ;)
Once it gets finished, it probably gets comitted to FFMpeg and usable with perian (I'm assuming that's the ffdshow equivalent on OS X) or something.

I don't know if there are any other porting projects for lagarith.

Jim_Pansen

27th May 2010, 13:43

benwaggoner

3rd January 2011, 22:06

Another Lagarith question: why are settings not persistant in some apps? I can set Colorspace to YV12 and Multithreading on in VirtualDub 32-bit just fine. But if I "Okay" and go back to the dialog in VDub 64-bit or any Adobe video apps, everything is back at the defaults when I return.

Any ideas/workarounds?

mariush

4th January 2011, 10:21

Settings are saved in C:\Windows\Lagarith.ini - maybe the 64bit windows API functions used by Lagarith report a different folder instead of "C:\Windows", like "c:\windows\Sys64" or something like that.

Use Filemon from SysInternals Suite to see where's is the codec trying to find the ini file.

totya

4th January 2011, 14:20

Run these 64bit apps with admin rights (run with admin), and u see, settings is saved.

My idea: try x264 vfw codec, lossless mode is very good.

Another Lagarith question: why are settings not persistant in some apps? I can set Colorspace to YV12 and Multithreading on in VirtualDub 32-bit just fine. But if I "Okay" and go back to the dialog in VDub 64-bit or any Adobe video apps, everything is back at the defaults when I return.

Any ideas/workarounds?

qyot27

9th January 2011, 09:11

The decoder has been committed to ffmpeg:
http://git.ffmpeg.org/?p=ffmpeg;a=commit;h=2588d70388459e1f6958ee4f0c092d643006af3c

If only I had been aware of this going on on the mailing list over the last few days, I wouldn't have gone to the trouble of adjusting a local clone of the repo.or.cz branch to current ffmpeg manually last night (to use with FFMS2), just 12 hours before the official commit occurred.

Reimar

10th January 2011, 13:41

The decoder has been committed to ffmpeg:
http://git.ffmpeg.org/?p=ffmpeg;a=commit;h=2588d70388459e1f6958ee4f0c092d643006af3c

Please consider it experimental though, it's been committed due to demand and too little time of the original author, but it does have some issues (I am not aware of it decoding anything incorrectly though, but I wouldn't 100% exclude the possibility either).

qyot27

10th January 2011, 17:34

Please consider it experimental though, it's been committed due to demand and too little time of the original author, but it does have some issues (I am not aware of it decoding anything incorrectly though, but I wouldn't 100% exclude the possibility either).
I'm sure it's already known, but there is one specific case that I've seen it choke on. From what I can recall of how Lagarith's methodology was described, if it detects frames of a solid color it stores them as a flag or something rather than a full frame (the video generated by the below script at 640x480 results in a 10KB Lagarith file, as does that same script with a resize to 1920x1080). This behavior can't be turned off, so it can occur in the middle of any given sort of video material. But the fflagarith decoder outputs green frames instead of the intended color.

It can be duplicated by artificial means.
BlankClip(40) ++ BlankClip(40).Invert()
would be sufficient.

Reimar

10th January 2011, 20:57

Upload a sample, preferably as an attachment to an FFmpeg bug report.
A report on this board with instructions to reproduce that require Windows when basically none of the developers (including me) use Windows, if they can avoid it, is unlikely to help getting a fix any time soon.

qyot27

11th January 2011, 05:29

Report submitted with attached sample. Hopefully I did it correctly.

Reimar

11th January 2011, 09:50

Report submitted with attached sample. Hopefully I did it correctly.

I think so. It's very strange that you do not get any error messages in the output though, there's probably more than one bug there.

video_magic

14th February 2011, 12:43

New version 1.3.21
http://lags.leetcode.net/codec.html

The changelog looks like it delivers significant performance improvements.

Version 1.3.21 released on 2-13-2011
Fixed a bug that would cause the codec to crash when downsampling certain resolutions. Thanks to Richard Jones for reporting this bug and tracking down the cause.
Several speed improvements:
- Integrated RLE restoration into range decoder so decoding only takes one pass through the data.
- Increased range decoder hash table size.
- Removed RLE level 2, testing indicates that it doesn't offer any real benefit verses levels 1 or 3.
- RLE compression now encodes all levels in parallel and selects the best one rather than perform an estimation run and then an RLE run.
- RLE compression now has a faster SSE version for processors that support SSE.
- Added and improved existing MMX/SSE/SSE2 optimizations for several functions relating to median prediction and image layout.
Overall I typically see about a 10-30% speed improvement.
Tweaked how RLE level is selected to improve compression by about 1%.
Tweaked how the work is distributed in multithreading to reduce wasting CPU time.
Removed Reduced Resolution mode, I don't think it is useful enough to justify maintaining.

benwaggoner

18th February 2011, 19:34

Sounds like a good release. No new issues for me in limited testing, but After Effects still doesn't retain settings correctly.

totya

25th February 2011, 17:30

Sounds like a good release. No new issues for me in limited testing, but After Effects still doesn't retain settings correctly.

After Effects is x64 version?

SirLagsalot

12th May 2011, 02:05

Lagarith 1.3.25 has been released, with several performance improvements. Overall, I see roughly a 20% encoding speed increase, and a 30% decoding speed verses version 1.3.24.
Below is a speed comparison of recent builds, with the time in seconds it took for Virtualdub to do a video analysis pass:

Compression:
Threads Mode 1.3.20 1.3.22 1.3.24 1.3.25
single RGB 180 146 145 118
multi RGB 146 123 121 104
single RGBA 218 177 177 143
multi RGBA 179 163 148 131
single YUY2 116 97 97 83
multi YUY2 102 87 87 76
single YV12 100 84 84 71
multi YV12 89 76 75 64

Decompression:
Threads Mode 1.3.20 1.3.22 1.3.24 1.3.25
single RGB 203 134 127 96
multi RGB 151 102 88 72
single RGBA 266 177 169 127
multi RGBA 187 133 122 103
single YUY2 115 86 78 61
multi YUY2 93 77 69 55
single YV12 94 85 77 59
multi YV12 71 76 68 49

The test video was a 3000 frame 960x554 video. Encoding was done from a 12500kbps Xvid source; for RGBA it was fed through an Avisynth script which used mask to add an alpha channel based on the image. This biases the encoding results somewhat, since the Xvid decoding and mask times are included in the results. The test machine is an AMD Athlon 64 X2 4200+, using the 32bit build of Lagarith. Newer machines should see slightly better performance since the Athlon doesn't benefit from the added SSE2 routines.
The other significant change in this release is that settings are now stored in the Windows registry, this should make it more Vista and Windows 7 friendly.

At this point, I feel I've plateaued on optimizing Lagarith, and I am considering where to go next:
- Lend my assistance to FFMpeg to add full Lagarith decoding support.
- Start developing Lagarith 2.0, with the aims of supporting higher-bit depth colorspaces such as 48bit RGB, improved multithreading to take advantage of an arbitrary number of processors, and better performance overall.

Please let me know what you think of these options, as well as any other comments or suggestions about Lagarith.

benwaggoner

12th May 2011, 04:51

After Effects is x64 version?
Yep. I filed a bug on this eight and then six years ago, for the 32-bit version :).

3ds

12th May 2011, 11:06

- Lend my assistance to FFMpeg to add full Lagarith decoding support.
Put Lagarith to other packs like FFMPEG is a very good idea! I found this: https://github.com/justinruggles/FFmpeg-ac3enc/blob/ac3enc2/libavcodec/lagarith.c So it this work already done?

- Start developing Lagarith 2.0, with the aims of supporting higher-bit depth colorspaces such as 48bit RGB, improved multithreading to take advantage of an arbitrary number of processors, and better performance overall.
I think the performance of Lagarith it very good and it which scenario is 48bit RGB good?

IMHO a good idea is to improve the compression ratio. Don't know if it make sence to look at Ut Video Codec Suite (http://forum.doom9.org/showthread.php?t=143624) or to 7zip?

Some ideas (maybe some bad ideas ;) ):
* Try to use identical picture parts from previous frames.
* Improve NULL Frames:
** Make them more compatibly (the readme says that some programs can't handle them)
** Use not only the last frame. Make it possible to use not only the last frame as NULL Frame. Store some reference frames which can be reuses later.

TheRyuu

12th May 2011, 11:13

In your chart, do you by any chance have single/multi-threading reversed, all the single threaded numbers are higher than the multi-threaded ones.

SirLagsalot

12th May 2011, 22:50

Put Lagarith to other packs like FFMPEG is a very good idea! I found this: https://github.com/justinruggles/FFmpeg-ac3enc/blob/ac3enc2/libavcodec/lagarith.c So it this work already done?

Right now they only have decoding support for YV12 encodings.

Some ideas (maybe some bad ideas ;) ):
* Try to use identical picture parts from previous frames.
* Improve NULL Frames:
** Make them more compatibly (the readme says that some programs can't handle them)
** Use not only the last frame. Make it possible to use not only the last frame as NULL Frame. Store some reference frames which can be reuses later.

I'm leery of adding delta frames (besides null frames, which are part of the AVI spec), since it makes the codec less useful for video editing, and would significantly increase the complexity of the code.

In your chart, do you by any chance have single/multi-threading reversed, all the single threaded numbers are higher than the multi-threaded ones.

The numbers are the time in seconds it took to run, so smaller is better.

TheRyuu

12th May 2011, 23:21

The numbers are the time in seconds it took to run, so smaller is better.

Oh right, seems I overlooked that somehow. Thanks for the update.

Boogerlad

27th May 2011, 20:35

lagarith 2.0! I would go for improved compression ratio though, because my cpu is fast enough, but my raid0 array lags behind.

ianken

14th June 2011, 21:30

SirLagsalot

15th June 2011, 00:22

Possible newb question: I seem to always get RGB otput from the decoder. It encodes and contains YUV, and I can use it in (for example) AVISynth with no problems. but playing the file directly or in graphedit shows it outputting RGB even though YV12 is set in it's config page (via Virtual Dub).

Is this just a DSHOw interop issue or is there some setting I can tweak?

The filter chain is probably just trying RGB before YV12. Try checking the 'Prevent Upsampling' box in the config dialog, that should prevent it from converting YV12 video to RGB on playback.

Chikuzen

15th June 2011, 09:54

When VCM is used as a source filter in directshow, the graph cannot be constructed if VCM doesn't convert RGB.
thus, it is impossible to play Lagarith excluding RGB output on directshow mediaplayers(WMP, MPC, etc...) at now.

DMO decoder is required to construct the graph with YUV output.

ianken

16th June 2011, 00:42

3ds

16th September 2011, 20:33

16th September 2011, 23:03

Is there any news about Lagarith on OSX? I found this:
http://trac.perian.org/ticket/198

The last comment on that ticket is three years old. FFmpeg/libav got a Lagarith decoder two years ago, but I guess it's still missing a few things, like support for other than YV12 colorspace (which SirLagsalot mentions above). YMMV.

qyot27

17th September 2011, 03:53

The last comment on that ticket is three years old. FFmpeg/libav got a Lagarith decoder two years ago, but I guess it's still missing a few things, like support for other than YV12 colorspace (which SirLagsalot mentions above). YMMV.
The decoder was committed to SVN in January just before the project-splitting mess occurred, although the relevant code at commit time hadn't changed since the repo.or.cz Lagarith branch's last update in October 2009 (as observed above, a problem with solid color frames was fixed after commit).

FWIW, it works through mplayer/mplayer2 if you use a libavcodec that's newer than January 2011, so it's really down to getting mplayer/mplayer2 to build (Python was having some issues on OSX the last time I tried to build mplayer2 there). I attempted to build Perian a few months ago and see if I could expose the Lagarith decoder, but the build failed and I didn't feel like trying to troubleshoot it any further.

tuqueque

26th September 2011, 18:14

New Lagarith version 1.3.26
http://lags.leetcode.net/codec.html

Cheers.
tuqueque.

benwaggoner

4th October 2011, 20:18

Woah! The switch from config to registry settings looks like it's fixed the ANCIENT bug in Adobe products where options weren't honored and everything was always RGB. I'm happily making a YV12 LAGS file from After Effects right now!

So many years of pain from that bug...

dlnalover94

28th October 2011, 23:14

is there any directshow filter that supports this codec for decoding?

Chikuzen

29th October 2011, 07:43

is there any directshow filter that supports this codec for decoding?

ffdshow and LAV Video Decoder

Pat357

9th November 2011, 04:25

ffdshow and LAV Video Decoder
LAV Video works indeed, but very very slooow on playback, even on my i7 990X@4Ghz. (it looks like the decoding is only single treaded)

I can't make it work in FFSshow ... how is the codec called in FFDshow ? I see nothing that looks like Lagarith decoder in the decoder list.

If I write a singe line AVS script like :
AVISource("filename") and open this, the playback is smooth and much faster (the original vfw is than used to decode, multithreaded)

Opening the AVI in FFmpeg (FFPlay) gives me green colored decoded frames (!!) ... both with the AVI saved as YV12 and RGB32.
Is the FFmpeg LAC decoder faulty are am I doing something wrong ?

Chikuzen

9th November 2011, 10:08

I can't make it work in FFSshow ... how is the codec called in FFDshow ? I see nothing that looks like Lagarith decoder in the decoder list.
oops, sorry. I mistook.
ffdshow doesn't support lagarith yet.
Opening the AVI in FFmpeg (FFPlay) gives me green colored decoded frames (!!) ... both with the AVI saved as YV12 and RGB32.
Is the FFmpeg LAC decoder faulty are am I doing something wrong ?
If it becomes so, ffmpeg may have some bugs.
I found two bugs of libav's UtVideo decoder about ten days ago.
Since I sent small samples to the developer at #libav-devel(IRC channel) , they were fixed without passing half a day.
http://lists.libav.org/pipermail/libav-devel/2011-October/013542.html
http://lists.libav.org/pipermail/libav-devel/2011-October/013607.html
If libavcodec's Lagarith decoder is important for you, you should do a bug report.

qyot27

9th November 2011, 11:24

Pat357

9th November 2011, 16:13

The Lagarith decoder in libavcodec supports YV12 only. The green frame issue depends on what the underlying stream is like - if these were solid-colored frames, then it would seem that you just happen to be unlucky enough to be using an FFmpeg build from the gap time between when the decoder was committed and the solids-rendered-as-green issue was fixed, which was a week later.
I always update to the latest FFmpeg from Zeranoe : http://ffmpeg.zeranoe.com/builds/
I disabled null frames when encoding in Lagarith, so that could also not be the reason for the green decoded movie.
The movie was a about 2m 1080p HD track to test lagarith.
If it's not a problem with solids, then my only other guess is, maybe null frames? I really don't know how the decoder treats those. And if it's not solids or nulls, are you sure you're using a build new enough to support Lagarith? It was only committed to SVN this past January (although the branch had been around for a couple years before that).
My current build is only a couple of days old !!
I see Zeranoe has updated his builds : latest is dd. 2011-11-06
I will download and try this build.

The slowness of the fflagarith decoder could also possibly be due to not being assembly-optimized, not just multi-threading stuff.
When I use the original VfW lagarith to decode (open video by opening AviSynth 2.6 MT), I get smooth playback and see that several cores are used to decode.
Opening the AVI directly with FFplay gives me very choppy playback <5 fps with only 1 of my 6 available cores (i7 990x) used and at 100%. The other 5 cores are at < 1 %.
My command line is: "FFplay -threads 8 -i %1"

I found another build that doesn't show green video :
The green color could be a problem with the Zeranoe builds, I want to try other very recent builds.
The oldest build I have is this one :
ffmpeg\bin>ffplay.exe -threads 8 -loop 0 "K:\film\vdubtest\lagarith_UV12.avi"
FFplay version SVN-r26397, Copyright (c) 2003-2011 the FFmpeg developers
built on Jan 17 2011 04:07:25 with gcc 4.4.2
configuration: --enable-gpl --enable-version3 --enable-libgsm --enable-libvorbis --enable-libtheora --enable-libspeex --enable
-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libopencore_amrwb --enable-libopencore_amrnb --enable-libvpx
--disable-decoder=libvpx --arch=x86 --enable-runtime-cpudetect --enable-libxvid --enable-libx264 --enable-librtmp --extra-libs='
-lrtmp -lpolarssl -lws2_32 -lwinmm' --target-os=mingw32 --enable-avisynth --enable-w32threads --cross-prefix=i686-mingw32- --cc=
'ccache i686-mingw32-gcc' --enable-memalign-hack
libavutil 50.36. 0 / 50.36. 0
libavcore 0.16. 1 / 0.16. 1
libavcodec 52.108. 0 / 52.108. 0
libavformat 52.93. 0 / 52.93. 0
libavdevice 52. 2. 3 / 52. 2. 3
libavfilter 1.74. 0 / 1.74. 0
libswscale 0.12. 0 / 0.12. 0
Input #0, avi, from 'K:\film\vdubtest\lagarith_UV12.avi':
Duration: 00:00:30.03, start: 0.000000, bitrate: 169237 kb/s
Stream #0.0: Video: lagarith, yuv420p, 1920x1080, 29.97 tbr, 29.97 tbn, 29.97 tbc
16.54 A-V: 0.000 s:22.8 aq= 0KB vq= 3529KB sq= 0B f=0/0 /0
The quality is OK : no green decoded frames anymore!! The problem with this one is that it either hangs on playback or is *extremely* slow : I only see the first couple of frames (maybe 50 or so) and then playback stops.

The Zeranoe build on the same file :
K:\programs\ffmpeg-git-8475ec1-win32-static\bin>ffplay -threads 8 -i "K:\film\vdubtest\lagarith_UV12.avi"
ffplay version N-34318-g8475ec1, Copyright (c) 2003-2011 the FFmpeg developers
built on Oct 31 2011 17:50:05 with gcc 4.6.1
configuration: --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --en
able-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg
--enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --e
nable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
libavutil 51. 22. 0 / 51. 22. 0
libavcodec 53. 26. 0 / 53. 26. 0
libavformat 53. 18. 0 / 53. 18. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 45. 3 / 2. 45. 3
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[avi @ 04422FC0] parser not found for codec lagarith, packets or times may be invalid.
Input #0, avi, from 'K:\film\vdubtest\\lagarith_UV12.avi':
Duration: 00:00:30.03, start: 0.000000, bitrate: 169546 kb/s
Stream #0:0: Video: lagarith (LAGS / 0x5347414C), yuv420p, 1920x1080, 29.97 tbr, 29.97 tbn, 29.97 tbc
6.33 A-V: 0.000 fd= 0 aq= 0KB vq= 4608KB sq= 0B f=0/0 0/0

This build decodes much faster, but the green color is there.
It looks like the Y channel is OK but the chroma is completely messed up.
Also notice the "parser not found for codec lagarith, packets or times may be invalid" message".
I have to try newer build, but not from Zeranoe.

qyot27

10th November 2011, 03:01

When I use the original VfW lagarith to decode (open video by opening AviSynth 2.6 MT), I get smooth playback and see that several cores are used to decode.
Opening the AVI directly with FFplay gives me very choppy playback <5 fps with only 1 of my 6 available cores (i7 990x) used and at 100%. The other 5 cores are at < 1 %.
My command line is: "FFplay -threads 8 -i %1"
The reference Lagarith codec has both multithreading and assembly optimizations. Lacking multithreading capability will lead to a drop in speed, but if it's an extremely large drop in speed, it's more than likely due to not having the relevant assembly code to speed things up.

1) Assembly+Multithreading=Fast
2) Assembly+No Multithreading=Fast, but not as fast as 1) if the code in 1) is optimized for multiple threads
3) No Assembly+Multithreading=Slow, even if it can use threads
4) No Assembly+No Multithreading=Slowest option, but likely only by a small degree

This is a comparison using avsmeter to compare the performance of the VfW Lagarith decoder (1.3.26) to the one in libavcodec (via FFMS2):
$ avsmeter lagtest-avisource.avs

AVSMeter v1.12 (Aug 2 2011) by Groucho2004
AviSynth 2.60, build:May 25 2011 [19:58:41]

Number of frames: 2013
Length (h:m:s.ms): 0:02:14.200
Frame width: 512
Frame height: 288
Framerate: 15.000 (1000000000 / 66666501)
Progressive: Yes
Colorspace: YV12

Hit ESC to exit...
Frame 2013/2013, fps (min/max/avg): 52.44 | 178.80 | 71.07

$ avsmeter lagtest-ffms2.avs

AVSMeter v1.12 (Aug 2 2011) by Groucho2004
AviSynth 2.60, build:May 25 2011 [19:58:41]

Number of frames: 2013
Length (h:m:s.ms): 0:02:14.200
Frame width: 512
Frame height: 288
Framerate: 15.000 (15015 / 1001)
Progressive: Yes
Colorspace: YV12

Hit ESC to exit...
Frame 2013/2013, fps (min/max/avg): 29.98 | 126.88 | 44.40
As you can see, the VfW decoder is, on average, a little less than twice as fast as the libavcodec decoder is. My computer does not have multithreading ability at all, and because it's using a Celeron processor based on the Pentium III Coppermine, it's not got the luxury of newer instruction sets like SSE2 or higher. That speed difference in my example is therefore probably entirely due to the libavcodec decoder simply not being as optimized, not from the lack of multithreading.

Note, of course, that the sample I was using was only 512x288. I don't expect the speed to scale linearly up to 1080p, nor to scale linearly across processors, but it still illustrates the performance gap.

The other possibility, is that if you're using 64-bit builds, they're getting miscompiled, which would explain why 32-bit builds don't show the problem. I just downloaded the latest win32-static build from Zeranoe and the sample from the above test played back fine (albeit with the same quirks I also noticed in FFMS2, which coincided with some warnings I got back about length and output bytes or something).

zambelli

22nd March 2013, 23:27

Why no Lagarith encoder in Ffmpeg? Lagarith is released under GPL, so from a licensing perspective it should be OK, right?

LoRd_MuldeR

22nd March 2013, 23:37

Why no Lagarith encoder in Ffmpeg? Lagarith is released under GPL, so from a licensing perspective it should be OK, right?

Might be related to the fact that the original Lagarith code is Windows-only and implemented as a VfW Codec.

Also, if you need a lossless encoder, there already are some alternatives available in FFmpeg, such as HuffYUV, FFV1, x264...

Leeloo Minaï

23rd March 2013, 17:22

LoRd_MuldeR

24th March 2013, 22:23

@LoRd_MuldeR : where is the problem with the fact Lagarith is in VfW format ?
HuffYUV codec was originally released a VfW codec, so "windows only", but also as open source, all like Lagarith, and now HuffYUV is part of FFmpeg...

The "problem" is that you don't have a general purpose cross-platfrom encoder library that you can just take "as is" and plug it into FFmpeg ;)

Instead you have to "extract" (decouple) the actual encoder code from the VfW Codec wrapper. Depending on how "clean" the code is written/designed (haven't looked at in detail) this can be straight-forward or really annoying!

Also, code that compiles in Visual Studio and that never has been tested with other compilers is likely to not compile with GCC, especially if a lot of Win32 API functions/types or MSVC-specifics intrinsic are used. It's not fun to fix that :rolleyes:

After all, this all should be very doable, but somebody has to do it! Feel free to port the Lagarith encoder and submit a patch to the FFmpeg and/or Lavc guys...

foxyshadis

26th March 2013, 01:59

Porting might not be too difficult, but it isn't in there because no one ever did it, the author doesn't want to, and it must not be a high priority for volunteers.

Lagarith itself is written with intrinsics, which is always nice and makes porting easy, but also includes a bunch of intel-format custom asm code to do colorspace conversion. It also sets and reads options in a very windows-specific way, via the registry. Since ffdshow is primarily made for MSVC it could be included nearly as-is, the way libdts and faad are, but it wouldn't make it into ffmpeg. I don't think it'd be very hard, colorspace conversion is something better handled in libswscale than lavc anyway. Another serious problem is that some features are based on x86 quirks, like the way it performs floating point operations, that aren't be 100% lossless on non-x86 platforms without reimplementing floating-point logic in software. That's a huge part of the slowdown, and one reason some of the intrinsics can't be used.

(The FFDShow includion would be time-consuming more due to creating the GUI than anything.)

tuqueque

27th March 2013, 00:04

I'm no coder... But as a user, I LOVE Lagarith to record video tutorials, intermediate lossless content and stuff like that. I'd love to have Lagarith on FFmpeg. It compresses better than almost any other (lossless) codec and I'll use anything but x264/H.264 in my work. I think the best way to support real open source projects, is to avoid stuff based on proprietary/patented stuff (not trying to initiate a debate, I'm just exposing the searons why I use Lagarith).

In any case, we may wait for VP9's lossless mode.

Dark Shikari

27th March 2013, 00:16

FFV1 and Dirac are both (probably) royalty-free and should compress much better than Lagarith.

zambelli

27th March 2013, 22:40

Lagarith is royalty-free too (at least there's no mention of any licensing on its homepage). "This codec was build using the Huffyuv source as a template, and uses some Huffyuv code, most notably the routine to upsample YUY2 video to RGB and to perform pixel prediction on YUY2 video. Other colorspace conversion routines were taken from AviSynth. Lagarith is released under the GPL."

You're correct regarding compression efficiency, but performance wise Lagarith kicks FFV1's butt on a modern Intel i7 quadcore system. Here's a quick test:

Lagarith v1.3.27 x86
FFV1 (Ffdshow) Rev4499 20130104 x86
Source: 1920x1080 29.97 fps YV12

==========================
Lagarith = 0.601x realtime, 33.11% file size
FFV1 AC Large = 0.306x realtime, 27.95% file size
FFV1 VLC Small = 0.408x realtime, 30.04% file size

As you can see, FFV1 is 15% more efficient than Lagarith in its most complex mode (AC Large), but half the speed. IMHO, that's not worth it.

Another advantage of Lagarith over FFV1 is that it can be installed as a standalone codec in Windows, making it very easy to distribute Lagarith-compressed videos to people who aren't well acquainted with codecs, VfW, DirectShow, etc. Telling somebody to "download and install this one executable" is way easier than teaching somebody how to find the latest stable build of FFdshow, install it, and then configure to decode the right codecs.

Asmodian

28th March 2013, 00:39

Doesn't this speed/size ratio look even better for huffyuv? I remember it is much faster than Lagarith and only somewhat larger? I think it is all about picking the option that is "fast enough" while also being "small enough". Of course "enough" is project dependent.

benwaggoner

29th March 2013, 20:23

Another advantage of Lagarith over FFV1 is that it can be installed as a standalone codec in Windows, making it very easy to distribute Lagarith-compressed videos to people who aren't well acquainted with codecs, VfW, DirectShow, etc. Telling somebody to "download and install this one executable" is way easier than teaching somebody how to find the latest stable build of FFdshow, install it, and then configure to decode the right codecs.
And that is exactly why I use Lagarith instead of FFV1. I need to be able to send someone an .avi file and have them easily be able to do whatever with it in their existing Windows tools.

If there was a stand-alone installer for FFV1 that provided 32-bit and 64-bit VfW/DirectShow/MediaFoundation encoders and decoders, I'd be over the moon. If it had QuickTime components for Mac and Windows, I'd never use anything else.

zerowalker

2nd April 2013, 20:57

Is it possible to optimize the Lagarith code to use AVX instructions and stuff?
The encoder is fast as it is, but it would be awesome if it´s possible to improve it.

But i don´t know much of programming, so can´t say if it´s possible or not.

Dark Shikari

2nd April 2013, 21:02

Almost all the time in something like Lagarith is going to be spend in the arithmetic coder, which is almost necessarily not-SIMDable.

zerowalker

2nd April 2013, 21:04

So, in other words, instructions doesn´t help Lagarith?

Dark Shikari, you wouldn´t know how to implement YCoCg support in Lagarith?

foxyshadis

3rd April 2013, 02:36

Almost all the time in something like Lagarith is going to be spend in the arithmetic coder, which is almost necessarily not-SIMDable.

It's actually a reasonably fast range coder, and keeps separate state in both threads; most of the time seems to be spent in prediction, which is already SIMD'd to hell and back. (There is no straight-C fallback code, just SIMD for most operations.)

Dark Shikari, you wouldn´t know how to implement YCoCg support in Lagarith?

It's not knowing how, it's someone willing to do it. Although it might not be too hard to rip the guts out of ConvertToYCgCo and insert it into another conversion routine, it would be incompatible with every other Lagarith install out there right now.

I did do some testing with ConvertToYCgCo, though, and it does seem to help a little bit if you don't mind subsampling, but if color fidelity is important to you, YCgCo24 (or YV24) gains you almost nothing size-wise over RGB. Maybe 3-5%.

zerowalker

3rd April 2013, 03:27

Oh well if it´s just 3-5% less size, then it´s pretty much useless, as it´s not perfect, which RGB is.

Is there a way to make Lagarith convert to YV12 correctly?

ConvertToYV12(matrix="Rec709")

For example, if the original need to use this matrix to be correct.
Lagarith will use Rec601 always it seems.

And i don´t think you can solve it afterwards, though if i am wrong, it would be nice.

EDIT:

Okay it seems to be fixable with ColorMatrix afterwards.
But would be nice for Lagarith to do it for many reasons.

Dark Shikari

3rd April 2013, 04:48

It's actually a reasonably fast range coder, and keeps separate state in both threads; most of the time seems to be spent in prediction, which is already SIMD'd to hell and back.That seems kind of hard to believe -- median prediction only takes a few SIMD operations per 16 pixels, while even a highly optimized arithmetic coder like x264's takes a dozen or two cycles per output bit. Is something else odd going on here?

zerowalker

4th April 2013, 01:01

Is anyone able to decode Lagarith with the 64 bit version of Avisynth?
I get FourCC error.

I can decode it if i use Lavfilter´s decoder for it though, but it´s a bit problematic.

sh0dan

3rd July 2013, 22:13

A kind soul would add multithreaded colorspace rgb<->yuv conversion as well - a fairly straightforward optimization, which would make both encoding+decoding faster.

Lenchik

5th November 2013, 04:09

Can anyone add YV24 support to Lagarith, please?

SirLagsalot

16th November 2013, 19:07

I am experimenting with re-writing Lagarith to support high-bitdepth (10 bits/color channel or more). To do this though, I need to know what color formats would be the most useful, and sample videos in those formats. Hopefully, some people in this thread would be willing to provide such video. My criteria for test video is as follows:

Must be from a source that uses the full bitdepth (upsampled video will not properly test the compression).
Should not have lossy compression applied
Should not have been resized
Uncompressed AVI would be the desired format, since it guarantees that I will be able to access the video
Should be short to keep the file size reasonable (100 frames or so is plenty)

zerowalker

16th November 2013, 19:49

Don´t have any sample videos, but can say state some color formats.

First YV24, even though it´s mostly unnecessary, it shouldn´t be to hard to implement while you are at it.
And then Rec.709, which is extremely useful but very rarely supported in a lossless codec.

Well that´s probably about it now that i think about it, i hope that was something you wanted to know, or perhaps you wanted to know something about bitdepth and such, if so i provided the wrong information.

I hope you see this through, Lagarith is an underestimated codec, and what´s it mostly is lacking is support and perhaps tweaks for encoding/decoding with newer CPU instructions.

EDIT:

Original 10bit videos will probably be quite hard to obtain, but it´s fairly simple to make one.

Take a video with much darkness and gradients. Fix it with Avisynth and work in 16bit.
Dither it to 10bit.

That way it won´t be upsampled from 8bit for no reason, and there should be pretty useful information in the last 2 bit that should have impact on compression.

But of course a 10bit Broadcast tape would be preferred, but can´t see a way to obtain one if you don´t have some good contacts.

qyot27

17th November 2013, 00:51

The better reason to support YV24 (and YV16, YV411, and Y8) is because that rounds out all the colorspaces that AviSynth 2.6 supports.

And really, if Rec.709 support gets added, might as well future-proof it a bit and also add Rec.2020.

raffriff42

17th November 2013, 00:57

ffmpeg can make nice high bit depth / deep color test videos...
EDIT test ramps seem to be generated with 8-bit math (http://www.ffmpeg.org/doxygen/1.0/vsrc__testsrc_8c-source.html#l00454) - sorry for not checking previously.
Try this thread: http://forum.doom9.org/showthread.php?t=158836

SirLagsalot

17th November 2013, 20:45

I can use upsampled or generated high-bitdepth video for developing and testing for losslessness, but to really tune the compression algorithms and speed optimizations, I need real footage. The upsampled/generated video simply won't have the same statistical distribution for the synthesized bits, and that could skew the results. I don't need test video immediately either, just in the next couple of months.

Since this would be a re-write, I would like to simplify how color formats are handled so it is easier to support more of them. YV24 and YV8 would probably be added, and Lagarith can already support encoding YV16.

I don't think I will add an option for Rec.709 (or Rec.2020), since it is another option the user has to get right or they can introduce noticeable loss. I feel that is an option better left to the encoding or decoding program.

raffriff42

17th November 2013, 22:00

OK try: http://media.xiph.org/tearsofsteel/ ("Tears of Steel" raw files)
http://blender-mirror.kino3d.org/peach/ ("Big Buck Bunny" raw files)
http://www.digitalbolex.com/d16-in-the-wild/ (Digital Bolex test footage)
EDIT http://blender-mirror.kino3d.org/peach/peach/filmmore/dpx-log/ (frame of "Big Buck Bunny" in a variety of deep color pixel formats)
http://blender-mirror.kino3d.org/peach/peach/filmmore/greyscale/ (grayscale test images in a variety of deep color pixel formats)
http://djv.sourceforge.net/ (DJV, an excellent deep color viewer & converter for images, image sequences, .MP4s)

ChiDragon

17th November 2013, 23:04

Do you have any use for crummy-looking 10-bit footage? I can send you some VHS captures in v210, but I haven't figured out how to verify if the devices are actually filling those bits.

Keiyakusha

17th November 2013, 23:56

I need real footage.
Define "real". Every 10bit footage is artificially produced one way or another. If Ill take some video, apply to it 16bit color correction, dither it down to 10bit - would that be real enough for you?

Edit: actually sorry. something like that was already suggested by zerowalker. Only that "Broadcast tape" is not any different and just as "real" as the footage people here proposed.

ChiDragon

18th November 2013, 01:40

Howso? If color-correcting a 10-bit source was no different than starting from an 8-bit source, there would be no high-depth film scanners.

Keiyakusha

18th November 2013, 04:27

Howso? If color-correcting a 10-bit source was no different than starting from an 8-bit source, there would be no high-depth film scanners.

If you answered to me, whatever you don't understand what I'm saying or I poorly explained it. By "If Ill take some video" I mean if I'll take 8-bit video. And after color correction, new color values will be created. Resulting footage is just as "real" as any other 10bit footage. And in fact may be even better, as most of the 10bit footage is created during production and not by scanners. Of course 3DCG rendered at 10+bits is an option too.

SirLagsalot

20th November 2013, 04:27

Define "real".
Either unprocessed source from a high-bitdepth camera, film scanner or similar capture source, or complex CG renderings done in high-bitdepth. Footage from an analog capture source is what I am most interested in, since I want to see how noisy such sources are. I figure if Lagarith can handle those types of footage well, it'll shouldn't have any trouble on upsampled video.

Links
Those look like they should be useful.

Do you have any use for crummy-looking 10-bit footage? I can send you some VHS captures in v210, but I haven't figured out how to verify if the devices are actually filling those bits.
Sure, I can check if it is using the full bit-depth.

zerowalker

24th November 2013, 11:05

Wait, are Broadcast Tapes 10bit upsampled?
Aren´t those originally in 10bit, but everything that actually shows to the World (Cinema, TV, DVD you name it) is 8bit, so the actual original (At least for broadcasts) are never actually seen?
It makes no sense if they just have 10bit tapes for the heck of it, if all material is actually 8bit, and when they are going to show it, they dither it down after it was upsampled once.

EDIT:

Not adding Rec.709 etc will actually be worse than not adding it.

As if you have RGB and save it to YV something, you will automatically convert it to Rec.601, so if that´s wrong the only way to solve it is through Avisynth and Colormatrix conversion, which is a lot more advanced, and also not perfect.
I think you could add it as, "Expert", or something, cause Decoding will not solve this if it´s converted, as it will stay Rec.601 all the time.

benwaggoner

24th November 2013, 20:08

Wait, are Broadcast Tapes 10bit upsampled?
Aren´t those originally in 10bit, but everything that actually shows to the World (Cinema, TV, DVD you name it) is 8bit, so the actual original (At least for broadcasts) are never actually seen?
It makes no sense if they just have 10bit tapes for the heck of it, if all material is actually 8bit, and when they are going to show it, they dither it down after it was upsampled once.
Storing in higher precision makes it possible to do overlays, color correction, etcetera without having to add more dither with every single filter. The same reason professional formats are all 4:2:2 even though all delivery formats are 4:2:0.

zerowalker

25th November 2013, 01:49

I see, so 4:2:2 are also upsampled?
Though i am pretty sure that thing were broadcasted as 4:2:2 before Digital took over.

Keiyakusha

25th November 2013, 02:14

zerowalker
You are doing too strong accent on the word "upsampled". For some reason I feel some kind of disappointment in your posts. It doesn't matters if it's upsampled or not. Also even though video often starts as 8bit, if some 16/32 bit processing applied to it (after which it is dithered to 10bit for archiving, delivery, whatever), you can no longer call it "upsampled". It became "real".

I figure if Lagarith can handle those types of footage well, it'll shouldn't have any trouble on upsampled video.
If you're talking about upsampled video that was simply converted 8->16bit without any further modifications.... why would someone want to compress it with lagarith? I don't see the reason for that. I only happens when you was requested to deliver some footage in some "standard" format, which happened to be 10bit. In case of lagarith one would compress it before upsampling and upsample when needed. But I also don't think one would use 10bit lagarith for raw noisy captures. Not in professional world. And most likely more casual users won't do it either.
If you're talking about upsampled video that received some kind of processing on top of that, as I said above, I don't think you can call it "upsampled" anymore. This is exactly the kind of content for which you want to optimize the encoder. And not for some random noise... unless for lagarith it doesn't matter which content you have. But then there is even less reasons to optimize for something exotic

Lenchik

25th November 2013, 04:19

why would someone want to compress it with lagarith?
For finding best x264 settings while not loosing time on repeating previoud work.

Keiyakusha

25th November 2013, 05:53

For finding best x264 settings while not loosing time on repeating previoud work.
This is not what I'm talking about. If you're doing some 16bit processing, your video will no longer be simply upsampled.
Anyway, that was a rhetorical question.

zerowalker

25th November 2013, 12:59

foxyshadis

1st December 2013, 08:34

Keyakusha,

What i simply meant is, i thought those tapes, or rather what was in the content of the tape was 10bit.
For example, let´s say Lord of the Rings, it was recorded in 10bit and that´s it´s original source (an example here).
Then they are going to broadcast it on TV etc. And they get a copy of that source, meaning they put it on the broacast tape, so it isn´t upsampled, it stays the same.

Same goes for everything else, but of course if there is 8bit stuff it get´s upsampled.

But if Nothing is 10bit which seems to be the case, than that´s my disappointment.

Nothing consumer-side is is more than 8bits is what he meant, I'm sure. Studio-side, it's tremendously different; stuff is generated and normally archived in various camera raw formats during production at a minimum of 10bit (HDCAM SR) up to 16bit (RED). Only older original HDCAM and DVCPRO used 8bit. The transmission pipelines vary widely between fully analog and 8bit digital and up. The difficulty is in obtaining samples of those masters, not whether they exist. They are usually compressed, but at such wildly high bitrates that it's not how we think of compression (like DV vs VCD).

Consumers can generate 10bit video with prosumer cameras, like Canon XL H1 or Sony HVR-V1U, but it might be better to ask on a videographer enthusiast forum than here.

Keiyakusha

1st December 2013, 08:52

Well, I tried to explain what I mean but it seems i failed at it so I've withdraw from this discussion. But let's try once more ^^
For some reason you got an impression that I'm saying that there are no >8bit material or something... this is not it. Of course there are >8bit things produced by some hardware and not generated on PC. For the movies like LotR it is of course more likely that they could be >8bit starting from the camera shooting stage. But if we are talking about global scale, which includes TV shows and various other material, then most of the content was never 10bit to begin with. It is whatever simply upsampled because standard requires it/someone requested it this way or it was modified/edited at some point.
That said, I believe the codec should be optimized for more common usage scenarios. Especially that for big studios who have things in >8 bit from the beginning there are 0 reason to use lagarith. I bet many of them never even heard of it and there is no reason for them to do such a research, they already have all the tools they need. Not to mention that many of them simply refuse to work with anything free/opensource. So why optimize codec for them and not for the world of "mere mortals"? (rhetorical question. I don't need an answer)

Edit: anyway, sorry for going offropic. Personally I'm not even interested in lagarith, it is too slow, so feel free to optimize for whatever you want.

zerowalker

1st December 2013, 14:04

Ah well than i had understood it correctly, at least half way.
That Tapes (Though not Broadcast tapes) can be done in higher bit depth.

But as you say, it´s impossible to obtain them, the only way to get such a "master tape" would be to either have the resources and contacts for it.
Or that it the company owning it has gone bankruptcy and it´s sold or something.

Keyakusha,

I fully understand what you mean, and as you say, the reason to even have 10bit as lossless currently, doesn´t even exist, the only reason for it i can think of with is for experimentations and certain individuals.
But i however don´t by any means want it NOT to be there, support and optimization in any case is always good, it doesn´t matter if the thing is useless now or not, If it can be added and someone wants to, then by all means do so!

And can also agree with it being slow. But for 2D (Pixel and simple stuff) it´s totally unbeatable, you can´t even compare it with UT Video Codec which is the rival i think.
On "Normal" videos however, Lagarith isn´t that great in terms of speed, but i think it can be optimized as it has been laying around for quite some time compared to UT which has been updated all the time.

But anyway, if i can help with anything for updating Lagarith i am on. Sadly Programming wise i am at a total newbie level, but making tests samples or something is something i should be able to do.

raffriff42

1st December 2013, 14:18

Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth. The format should generally be avoided if possible.*This* is my issue with Lagarith. I get too many glitches.

zerowalker

1st December 2013, 14:20

Wait what??
Errors as in, what?

LoRd_MuldeR

1st December 2013, 14:54

Wait what??
Errors as in, what?

He's referring to the "random" glitches (artifacts) that some people seem to be getting with Lagarith.

AFAIK, Lagarith uses arithmetic coding, which - in its most simple form - uses infinite-precision real numbers. Computers don't have that. So, in practice, we usually use a "range coder" instead of a plain arithmetic coder. It is essentially one and the same thing, yes. But the "range coder" works entirely on integer math (rather than using real numbers). This makes it more suitable for implementation on a real-world computer. Also ensure that the results are always deterministic.

Lagarith, on the other hand, seems to implement arithmetic coding based on limited-precision floating point math. This is kind of "delicate" with respect to rounding errors and stuff. And this is what may cause nondeterministic behavior :eek:

zerowalker

1st December 2013, 15:15

LoRd_MuldeR

1st December 2013, 15:41

But it does not effect Encoding does it?
Meaning all encodes movies will always be 100% alright.

But decoding them CAN fail, but redoing it CAN succeed, meaning it´s not a permanent error?

Well, we have a problem as soon as the encoder and the decoder "desynchronize" in some way, i.e. do not agree on the exact input/output values any longer. Whether the encoder or the decoder (or both) made the "mistake" that caused the desynchronization doesn't matter much. What does matter is that, after a desynchronization, we don't get back the same values from the decoder that we originally fed into the encoder. So we might get some ugly artifacts!

(Or in other words: Regardless of weather the video already was encoded "wrongly", or whether it was encoded "correct" but now can't be decoded "correctly", the end result is always the same. You can't get the correct output now!)

Weird that it uses such system if it isn´t perfect as these stuff are supposed to Always work in All scenarios.

Personally, I don't use Lagarith extensively. So I'm just trying to sum up the issues that some people have reported. There also seem to be many users which use Lagarith with no problem at all.

Also keep in mind that the "perfect" (error free) software doesn't exist in reality. We usually accept that a "high quality" software has about one bug per 1000 lines of code ;)

zerowalker

1st December 2013, 15:45

But if the error occurs (Artifact) is it always there, or does it appear from time to time depending if the decoder fails to reach the correct number?

And well yeah "error free" is more of a term what should be striven for. But something that´s fundamentally wrong like this, isn´t what i would expect from a Codec.

LoRd_MuldeR

1st December 2013, 15:51

But if the error occurs (Artifact) is it always there, or does it appear from time to time depending if the decoder fails to reach the correct number?

That's the "nice" thing about undefined behavior: Everything is possible. You just don't know what is going to happen. And the result can (but doesn't have to) be different each time :scared:

zerowalker

1st December 2013, 15:52

qyot27

1st December 2013, 16:17

you can´t even compare it with UT Video Codec which is the rival i think.
No, the closest 'rival' to Lagarith is actually FFV1. Ut Video mostly lies in the chasm between HuffYUV and Lagarith.

FFV1, for the record, also uses arithmetic coding (it's integer-based, though, not floating point). It loses to Lagarith in a couple of areas though - mainly solid color frames* and things like line drawings (and speed; Lagarith is faster than FFV1). FFV1 also supports all those high bit depth pixel formats.

*unless you use the lower efficiency VLC coder for FFV1 - then it actually gets close to Lagarith for solid color frames (because it switches to inter-frame? that's...unexpected, and feels wrong - it's what VirtualDub seems to report, though). Lagarith stores solid color frames as flags and presents them to the decoder as real intra frames.

Case in point, a simple BlankClip script (RGBA, 10 seconds, 640x480 @ 24 fps):
Lagarith: 15 KB
FFV1 (-coder 1 -context 1): 1.85 MB
FFV1 (-coder 1 -context 0): 1.85 MB
FFV1 (-coder 0 -context 1): 70.1 KB
FFV1 (-coder 0 -context 0): 70.1 KB

-coder 0 = VLC
-coder 1 = AC

foxyshadis

1st December 2013, 16:26

Well if the Possibility exist, then you can at least Save the content by converting it. If the error Always occurs that´s impossible.
But i find it weird that i myself have never been inflected by this, i tend to use Lagarith quite a lot.

It's more that floating point isn't entirely perfect, so things might be +1 or -1 from where they should be when decoded; those errors can add up, but they're pretty rare and hard to notice usually. It has C, MMX, SSE, and SSE2 versions of a lot of functions, all of which may have different rounding characteristics; it's impossible to tell since there's no testbed. Then again, there might just plain plain old bugs, too.

LoRd_MuldeR

1st December 2013, 16:49

It's more that floating point isn't entirely perfect, so things might be +1 or -1 from where they should be when decoded; those errors can add up, but they're pretty rare and hard to notice usually.

But for an entropy coder it can mean everything. Even the slightest difference in the value may cause a number to fall into a different range/bucket, and then the decoded symbol might be a completely different one.

And because the probability model is usually updated after each symbol, one wrong symbol can cause all future symbols to be wrong too :scared:

BTW: This is also the reason why, when dealing with floating point numbers, we never check for equality directly, but instead assume the numbers are "equal" if the absolute difference is below a certain threshold.

foxyshadis

1st December 2013, 18:48

But for an entropy coder it can mean everything. Even the slightest difference in the value may cause a number to fall into a different range/bucket, and then the decoded symbol might be a completely different one.

And because the probability model is usually updated after each symbol, one wrong symbol can cause all future symbols to be wrong too :scared:

BTW: This is also the reason why, when dealing with floating point numbers, we never check for equality directly, but instead assume the numbers are "equal" if the absolute difference is below a certain threshold.

The entropy coder portion is all integer and only done in C. That at least doesn't seem to be the problem.

LoRd_MuldeR

1st December 2013, 18:58

Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth. The format should generally be avoided if possible.
The entropy coder portion is all integer and only done in C. That at least doesn't seem to be the problem.

:confused:

SirLagsalot

1st December 2013, 21:31

So why optimize codec for them and not for the world of "mere mortals"?
I expect that high-bitdepth will become more common over time with consumer-level gear, so I would like to aim Lagarith where the technology will be.

Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth.

This is not correct. Lagarith uses an integer-only range coder to handle the compression, floating-point is only used to scale the symbol probability tables to a power of 2. Floating point math on the x86/x64 is deterministic for a given input and order of operations, so it does not randomly introduce error. To my knowledge, there has only been one error related to the use of floating-point math, and that was caused when the calling program changed the floating-point rounding mode or precision. Lagarith now checks the floating-point state before each frame, and adjusts it if need be to prevent this. I do plan to remove floating-point math in the rewrite though, in order to simplify porting and hopefully kill off that myth.

LoRd_MuldeR

2nd December 2013, 01:59

This is not correct. Lagarith uses an integer-only range coder to handle the compression, floating-point is only used to scale the symbol probability tables to a power of 2.

Well, as long as the encoder and the decoder need to do this step in the exactly same way, things could go still go wrong and break decoding, right?

Floating point math on the x86/x64 is deterministic for a given input and order of operations, so it does not randomly introduce error. To my knowledge, there has only been one error related to the use of floating-point math, and that was caused when the calling program changed the floating-point rounding mode or precision. Lagarith now checks the floating-point state before each frame, and adjusts it if need be to prevent this. I do plan to remove floating-point math in the rewrite though, in order to simplify porting and hopefully kill off that myth.

Hmm, to my knowledge there's at least the classical "x87" FPU with 80-Bit internal precision and the newer SIMD (MMX, SSE, etc) instructions with "only" 64-Bit precision. Even if all variables are 64-Bit (double) precision in memory, the compiler may still keep intermediate results in registers, which would then be either 80-Bit or 64-Bit. Furthermore, even if you don't use SIMD assembly/intrinsic explicitly, compilers may still generate such instructions when targeting CPU's with MMX/SSE support. Finally, there are various compiler settings that effect floating point math ("fast math" option, etc). So there's at least the danger of "portability" issues when the encoder/decoder are compiled with different compilers (compiler configuration).

SirLagsalot

2nd December 2013, 02:42

Well, as long as the encoder and the decoder need to do this step in the exactly same way, things could go still go wrong and break decoding, right?

No, the code and the inputs are identical, so the output will be identical as long as the same floating-point state (rounding mode and precision) are used.

Hmm, to my knowledge there's at least the classical "x87" FPU with 80-Bit internal precision and the newer SIMD (MMX, SSE, etc) instructions with "only" 64-Bit precision. Even if all variables are 64-Bit (double) precision in memory, the compiler may still keep intermediate results in registers, which would then be either 80-Bit or 64-Bit. Furthermore, even if you don't use SIMD assembly/intrinsic explicitly, compilers may still generate such instructions when targeting CPU's with MMX/SSE support. Finally, there are various compiler settings that effect floating point math ("fast math" option, etc). So there's at least the danger of "portability" issues when the encoder/decoder are compiled with different compilers (compiler configuration).
Lagarith uses the 53-bit precision mode for floating point, so none of those variations in potential precision end up affecting the result. It also checks and sets the floating point state if need be, so even if Lagarith was compiled with the wrong settings by someone else, it would be corrected at run-time.

ChiDragon

4th December 2013, 04:35

Sure, I can check if it is using the full bit-depth.

Here you go, a VHS sample (http://chidragon.thedessie.com/Doom9/titanic-vhs-sample1.rar). Possibly not all that useful for you: WinRAR gave it a steady 51% compression, so I'm guessing it is 9-bit. That would also make sense because a guy opened up the next model in this line of AV syncs and found a Philips SAA7113 was the ADC.

zerowalker

4th December 2013, 11:21

So wait, does Lagarith not have the problem and just some people, or are you talking about something else (Lagalot?)

SirLagsalot

5th December 2013, 00:25

Zerowalker, I think early on some people were seeing glitches from other causes and assuming it was due to floating point math errors; and then the rumor stuck around long after the actual bugs were fixed.

zerowalker

9th December 2013, 13:09

Oh, well let´s hope that´s the case, really like the codec and some fundamental error like that would really ruin it.

zerowalker

26th May 2014, 22:17

It seems the "Floating Point" errors actually does exist, at least from my understanding in my discussion with an ffmpeg developer.

Not entirely sure, but if am understand things correctly, in normal circumstances the errors shouldn't appear.
But if it's not an x86 system it will, though that doesn't tell me much, as i am sure x64 is still an x86 system;S

The original codec implementation uses floating point arithmetic which will fail on processors != x86 and might fail if you use another compiler. FFmpeg's implementation is fixed-point (which is what you expect for a lossless codec) meaning you can compile it for any hardware with any (non-broken) C compiler and you will always get correct (bitexact) output. Fixed-point arithmetic is often slower than floating-point on typical hardware.
See also (for example):
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2009-September/079680.html
http://mod16.org/hurfdurf/?p=142

I personally have never seen any corrupt frames that i can think of, and i have been using Lagarith quite a bit, so i think i should have encountered it by now.
Though again, that depends what a "corrupt frame" is, if it's 1 pixel that's wrong, then it's not possible i have noticed it, but if it's an entire corrupted imaged, that's another story.