Lagarith lossless codec

[benwaggoner]

Sounds like a good release. No new issues for me in limited testing, but After Effects still doesn't retain settings correctly.

[totya]

Sounds like a good release. No new issues for me in limited testing, but After Effects still doesn't retain settings correctly.

After Effects is x64 version?

[SirLagsalot]

Lagarith 1.3.25 has been released, with several performance improvements. Overall, I see roughly a 20% encoding speed increase, and a 30% decoding speed verses version 1.3.24.
Below is a speed comparison of recent builds, with the time in seconds it took for Virtualdub to do a video analysis pass:


Compression:
Threads Mode 1.3.20 1.3.22 1.3.24 1.3.25
single RGB 180 146 145 118
multi RGB 146 123 121 104
single RGBA 218 177 177 143
multi RGBA 179 163 148 131
single YUY2 116 97 97 83
multi YUY2 102 87 87 76
single YV12 100 84 84 71
multi YV12 89 76 75 64


Decompression:
Threads Mode 1.3.20 1.3.22 1.3.24 1.3.25
single RGB 203 134 127 96
multi RGB 151 102 88 72
single RGBA 266 177 169 127
multi RGBA 187 133 122 103
single YUY2 115 86 78 61
multi YUY2 93 77 69 55
single YV12 94 85 77 59
multi YV12 71 76 68 49


The test video was a 3000 frame 960x554 video. Encoding was done from a 12500kbps Xvid source; for RGBA it was fed through an Avisynth script which used mask to add an alpha channel based on the image. This biases the encoding results somewhat, since the Xvid decoding and mask times are included in the results. The test machine is an AMD Athlon 64 X2 4200+, using the 32bit build of Lagarith. Newer machines should see slightly better performance since the Athlon doesn't benefit from the added SSE2 routines.
The other significant change in this release is that settings are now stored in the Windows registry, this should make it more Vista and Windows 7 friendly.


At this point, I feel I've plateaued on optimizing Lagarith, and I am considering where to go next:
- Lend my assistance to FFMpeg to add full Lagarith decoding support.
- Start developing Lagarith 2.0, with the aims of supporting higher-bit depth colorspaces such as 48bit RGB, improved multithreading to take advantage of an arbitrary number of processors, and better performance overall.

Please let me know what you think of these options, as well as any other comments or suggestions about Lagarith.

[benwaggoner]

After Effects is x64 version?
Yep. I filed a bug on this eight and then six years ago, for the 32-bit version :).

[3ds]

- Lend my assistance to FFMpeg to add full Lagarith decoding support.
Put Lagarith to other packs like FFMPEG is a very good idea! I found this: https://github.com/justinruggles/FFmpeg-ac3enc/blob/ac3enc2/libavcodec/lagarith.c So it this work already done?

- Start developing Lagarith 2.0, with the aims of supporting higher-bit depth colorspaces such as 48bit RGB, improved multithreading to take advantage of an arbitrary number of processors, and better performance overall.
I think the performance of Lagarith it very good and it which scenario is 48bit RGB good?

IMHO a good idea is to improve the compression ratio. Don't know if it make sence to look at Ut Video Codec Suite (http://forum.doom9.org/showthread.php?t=143624) or to 7zip?

Some ideas (maybe some bad ideas ;) ):
* Try to use identical picture parts from previous frames.
* Improve NULL Frames:
** Make them more compatibly (the readme says that some programs can't handle them)
** Use not only the last frame. Make it possible to use not only the last frame as NULL Frame. Store some reference frames which can be reuses later.

[TheRyuu]

In your chart, do you by any chance have single/multi-threading reversed, all the single threaded numbers are higher than the multi-threaded ones.

[SirLagsalot]

Put Lagarith to other packs like FFMPEG is a very good idea! I found this: https://github.com/justinruggles/FFmpeg-ac3enc/blob/ac3enc2/libavcodec/lagarith.c So it this work already done?

Right now they only have decoding support for YV12 encodings.



Some ideas (maybe some bad ideas ;) ):
* Try to use identical picture parts from previous frames.
* Improve NULL Frames:
** Make them more compatibly (the readme says that some programs can't handle them)
** Use not only the last frame. Make it possible to use not only the last frame as NULL Frame. Store some reference frames which can be reuses later.

I'm leery of adding delta frames (besides null frames, which are part of the AVI spec), since it makes the codec less useful for video editing, and would significantly increase the complexity of the code.

In your chart, do you by any chance have single/multi-threading reversed, all the single threaded numbers are higher than the multi-threaded ones.

The numbers are the time in seconds it took to run, so smaller is better.

[TheRyuu]

The numbers are the time in seconds it took to run, so smaller is better.

Oh right, seems I overlooked that somehow. Thanks for the update.

[Boogerlad]

lagarith 2.0! I would go for improved compression ratio though, because my cpu is fast enough, but my raid0 array lags behind.

[ianken]

Possible newb question: I seem to always get RGB otput from the decoder. It encodes and contains YUV, and I can use it in (for example) AVISynth with no problems. but playing the file directly or in graphedit shows it outputting RGB even though YV12 is set in it's config page (via Virtual Dub).

Is this just a DSHOw interop issue or is there some setting I can tweak?

[SirLagsalot]

Possible newb question: I seem to always get RGB otput from the decoder. It encodes and contains YUV, and I can use it in (for example) AVISynth with no problems. but playing the file directly or in graphedit shows it outputting RGB even though YV12 is set in it's config page (via Virtual Dub).

Is this just a DSHOw interop issue or is there some setting I can tweak?

The filter chain is probably just trying RGB before YV12. Try checking the 'Prevent Upsampling' box in the config dialog, that should prevent it from converting YV12 video to RGB on playback.

[Chikuzen]

When VCM is used as a source filter in directshow, the graph cannot be constructed if VCM doesn't convert RGB.
thus, it is impossible to play Lagarith excluding RGB output on directshow mediaplayers(WMP, MPC, etc...) at now.

DMO decoder is required to construct the graph with YUV output.

[ianken]

When VCM is used as a source filter in directshow, the graph cannot be constructed if VCM doesn't convert RGB.
thus, it is impossible to play Lagarith excluding RGB output on directshow mediaplayers(WMP, MPC, etc...) at now.

DMO decoder is required to construct the graph with YUV output.

AVISynth seems to be able to output YUV into graphedt, is that not VFW/VCM based?

[3ds]

I believe the lagarith source itself is Windows specific. I also read it uses a floating point arithmetic (or range?) coder, which makes portability less fun. Nathan Caldwell started a yv12 decoder for FFMpeg and they came up with something to emulate the problem with the entropy coder. The git of the fork is here (http://repo.or.cz/w/FFMpeg-mirror/lagarith.git). Looks like everyone is busy with higher priorities, so it's more likley the far future. ;)
Once it gets finished, it probably gets comitted to FFMpeg and usable with perian (I'm assuming that's the ffdshow equivalent on OS X) or something.

I don't know if there are any other porting projects for lagarith.

Is there any news about Lagarith on OSX? I found this:
http://trac.perian.org/ticket/198 but the decoder seems to be in libavcodec and perian:
https://github.com/FFmpeg/FFmpeg/tree/master/libavcodec/lagarith.c
http://trac.perian.org/browser/ffmpeg/libavcodec/lagarith.c
isn't it?

[nm]

Is there any news about Lagarith on OSX? I found this:
http://trac.perian.org/ticket/198

The last comment on that ticket is three years old. FFmpeg/libav got a Lagarith decoder two years ago, but I guess it's still missing a few things, like support for other than YV12 colorspace (which SirLagsalot mentions above). YMMV.

[qyot27]

The last comment on that ticket is three years old. FFmpeg/libav got a Lagarith decoder two years ago, but I guess it's still missing a few things, like support for other than YV12 colorspace (which SirLagsalot mentions above). YMMV.
The decoder was committed to SVN in January just before the project-splitting mess occurred, although the relevant code at commit time hadn't changed since the repo.or.cz Lagarith branch's last update in October 2009 (as observed above, a problem with solid color frames was fixed after commit).

FWIW, it works through mplayer/mplayer2 if you use a libavcodec that's newer than January 2011, so it's really down to getting mplayer/mplayer2 to build (Python was having some issues on OSX the last time I tried to build mplayer2 there). I attempted to build Perian a few months ago and see if I could expose the Lagarith decoder, but the build failed and I didn't feel like trying to troubleshoot it any further.

[tuqueque]

New Lagarith version 1.3.26
http://lags.leetcode.net/codec.html

Cheers.
tuqueque.

[benwaggoner]

Woah! The switch from config to registry settings looks like it's fixed the ANCIENT bug in Adobe products where options weren't honored and everything was always RGB. I'm happily making a YV12 LAGS file from After Effects right now!

So many years of pain from that bug...

[dlnalover94]

is there any directshow filter that supports this codec for decoding?

[Chikuzen]

is there any directshow filter that supports this codec for decoding?

ffdshow and LAV Video Decoder

[Pat357]

ffdshow and LAV Video Decoder
LAV Video works indeed, but very very slooow on playback, even on my i7 990X@4Ghz. (it looks like the decoding is only single treaded)

I can't make it work in FFSshow ... how is the codec called in FFDshow ? I see nothing that looks like Lagarith decoder in the decoder list.

If I write a singe line AVS script like :
AVISource("filename") and open this, the playback is smooth and much faster (the original vfw is than used to decode, multithreaded)

Opening the AVI in FFmpeg (FFPlay) gives me green colored decoded frames (!!) ... both with the AVI saved as YV12 and RGB32.
Is the FFmpeg LAC decoder faulty are am I doing something wrong ?

[Chikuzen]

I can't make it work in FFSshow ... how is the codec called in FFDshow ? I see nothing that looks like Lagarith decoder in the decoder list.
oops, sorry. I mistook.
ffdshow doesn't support lagarith yet.
Opening the AVI in FFmpeg (FFPlay) gives me green colored decoded frames (!!) ... both with the AVI saved as YV12 and RGB32.
Is the FFmpeg LAC decoder faulty are am I doing something wrong ?
If it becomes so, ffmpeg may have some bugs.
I found two bugs of libav's UtVideo decoder about ten days ago.
Since I sent small samples to the developer at #libav-devel(IRC channel) , they were fixed without passing half a day.
http://lists.libav.org/pipermail/libav-devel/2011-October/013542.html
http://lists.libav.org/pipermail/libav-devel/2011-October/013607.html
If libavcodec's Lagarith decoder is important for you, you should do a bug report.

[qyot27]

The Lagarith decoder in libavcodec supports YV12 only. The green frame issue depends on what the underlying stream is like - if these were solid-colored frames, then it would seem that you just happen to be unlucky enough to be using an FFmpeg build from the gap time between when the decoder was committed and the solids-rendered-as-green issue was fixed, which was a week later. If it's not a problem with solids, then my only other guess is, maybe null frames? I really don't know how the decoder treats those. And if it's not solids or nulls, are you sure you're using a build new enough to support Lagarith? It was only committed to SVN this past January (although the branch had been around for a couple years before that).

The slowness of the fflagarith decoder could also possibly be due to not being assembly-optimized, not just multithreading stuff.

[Pat357]

The Lagarith decoder in libavcodec supports YV12 only. The green frame issue depends on what the underlying stream is like - if these were solid-colored frames, then it would seem that you just happen to be unlucky enough to be using an FFmpeg build from the gap time between when the decoder was committed and the solids-rendered-as-green issue was fixed, which was a week later.
I always update to the latest FFmpeg from Zeranoe : http://ffmpeg.zeranoe.com/builds/
I disabled null frames when encoding in Lagarith, so that could also not be the reason for the green decoded movie.
The movie was a about 2m 1080p HD track to test lagarith.
If it's not a problem with solids, then my only other guess is, maybe null frames? I really don't know how the decoder treats those. And if it's not solids or nulls, are you sure you're using a build new enough to support Lagarith? It was only committed to SVN this past January (although the branch had been around for a couple years before that).
My current build is only a couple of days old !!
I see Zeranoe has updated his builds : latest is dd. 2011-11-06
I will download and try this build.

The slowness of the fflagarith decoder could also possibly be due to not being assembly-optimized, not just multi-threading stuff.
When I use the original VfW lagarith to decode (open video by opening AviSynth 2.6 MT), I get smooth playback and see that several cores are used to decode.
Opening the AVI directly with FFplay gives me very choppy playback <5 fps with only 1 of my 6 available cores (i7 990x) used and at 100%. The other 5 cores are at < 1 %.
My command line is: "FFplay -threads 8 -i %1"

I found another build that doesn't show green video :
The green color could be a problem with the Zeranoe builds, I want to try other very recent builds.
The oldest build I have is this one :
ffmpeg\bin>ffplay.exe -threads 8 -loop 0 "K:\film\vdubtest\lagarith_UV12.avi"
FFplay version SVN-r26397, Copyright (c) 2003-2011 the FFmpeg developers
built on Jan 17 2011 04:07:25 with gcc 4.4.2
configuration: --enable-gpl --enable-version3 --enable-libgsm --enable-libvorbis --enable-libtheora --enable-libspeex --enable
-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libopencore_amrwb --enable-libopencore_amrnb --enable-libvpx
--disable-decoder=libvpx --arch=x86 --enable-runtime-cpudetect --enable-libxvid --enable-libx264 --enable-librtmp --extra-libs='
-lrtmp -lpolarssl -lws2_32 -lwinmm' --target-os=mingw32 --enable-avisynth --enable-w32threads --cross-prefix=i686-mingw32- --cc=
'ccache i686-mingw32-gcc' --enable-memalign-hack
libavutil 50.36. 0 / 50.36. 0
libavcore 0.16. 1 / 0.16. 1
libavcodec 52.108. 0 / 52.108. 0
libavformat 52.93. 0 / 52.93. 0
libavdevice 52. 2. 3 / 52. 2. 3
libavfilter 1.74. 0 / 1.74. 0
libswscale 0.12. 0 / 0.12. 0
Input #0, avi, from 'K:\film\vdubtest\lagarith_UV12.avi':
Duration: 00:00:30.03, start: 0.000000, bitrate: 169237 kb/s
Stream #0.0: Video: lagarith, yuv420p, 1920x1080, 29.97 tbr, 29.97 tbn, 29.97 tbc
16.54 A-V: 0.000 s:22.8 aq= 0KB vq= 3529KB sq= 0B f=0/0 /0
The quality is OK : no green decoded frames anymore!! The problem with this one is that it either hangs on playback or is *extremely* slow : I only see the first couple of frames (maybe 50 or so) and then playback stops.

The Zeranoe build on the same file :
K:\programs\ffmpeg-git-8475ec1-win32-static\bin>ffplay -threads 8 -i "K:\film\vdubtest\lagarith_UV12.avi"
ffplay version N-34318-g8475ec1, Copyright (c) 2003-2011 the FFmpeg developers
built on Oct 31 2011 17:50:05 with gcc 4.6.1
configuration: --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --en
able-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg
--enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --e
nable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
libavutil 51. 22. 0 / 51. 22. 0
libavcodec 53. 26. 0 / 53. 26. 0
libavformat 53. 18. 0 / 53. 18. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 45. 3 / 2. 45. 3
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[avi @ 04422FC0] parser not found for codec lagarith, packets or times may be invalid.
Input #0, avi, from 'K:\film\vdubtest\\lagarith_UV12.avi':
Duration: 00:00:30.03, start: 0.000000, bitrate: 169546 kb/s
Stream #0:0: Video: lagarith (LAGS / 0x5347414C), yuv420p, 1920x1080, 29.97 tbr, 29.97 tbn, 29.97 tbc
6.33 A-V: 0.000 fd= 0 aq= 0KB vq= 4608KB sq= 0B f=0/0 0/0

This build decodes much faster, but the green color is there.
It looks like the Y channel is OK but the chroma is completely messed up.
Also notice the "parser not found for codec lagarith, packets or times may be invalid" message".
I have to try newer build, but not from Zeranoe.

[qyot27]

When I use the original VfW lagarith to decode (open video by opening AviSynth 2.6 MT), I get smooth playback and see that several cores are used to decode.
Opening the AVI directly with FFplay gives me very choppy playback <5 fps with only 1 of my 6 available cores (i7 990x) used and at 100%. The other 5 cores are at < 1 %.
My command line is: "FFplay -threads 8 -i %1"
The reference Lagarith codec has both multithreading and assembly optimizations. Lacking multithreading capability will lead to a drop in speed, but if it's an extremely large drop in speed, it's more than likely due to not having the relevant assembly code to speed things up.

1) Assembly+Multithreading=Fast
2) Assembly+No Multithreading=Fast, but not as fast as 1) if the code in 1) is optimized for multiple threads
3) No Assembly+Multithreading=Slow, even if it can use threads
4) No Assembly+No Multithreading=Slowest option, but likely only by a small degree


This is a comparison using avsmeter to compare the performance of the VfW Lagarith decoder (1.3.26) to the one in libavcodec (via FFMS2):
$ avsmeter lagtest-avisource.avs

AVSMeter v1.12 (Aug 2 2011) by Groucho2004
AviSynth 2.60, build:May 25 2011 [19:58:41]

Number of frames: 2013
Length (h:m:s.ms): 0:02:14.200
Frame width: 512
Frame height: 288
Framerate: 15.000 (1000000000 / 66666501)
Progressive: Yes
Colorspace: YV12

Hit ESC to exit...
Frame 2013/2013, fps (min/max/avg): 52.44 | 178.80 | 71.07

$ avsmeter lagtest-ffms2.avs

AVSMeter v1.12 (Aug 2 2011) by Groucho2004
AviSynth 2.60, build:May 25 2011 [19:58:41]

Number of frames: 2013
Length (h:m:s.ms): 0:02:14.200
Frame width: 512
Frame height: 288
Framerate: 15.000 (15015 / 1001)
Progressive: Yes
Colorspace: YV12

Hit ESC to exit...
Frame 2013/2013, fps (min/max/avg): 29.98 | 126.88 | 44.40
As you can see, the VfW decoder is, on average, a little less than twice as fast as the libavcodec decoder is. My computer does not have multithreading ability at all, and because it's using a Celeron processor based on the Pentium III Coppermine, it's not got the luxury of newer instruction sets like SSE2 or higher. That speed difference in my example is therefore probably entirely due to the libavcodec decoder simply not being as optimized, not from the lack of multithreading.

Note, of course, that the sample I was using was only 512x288. I don't expect the speed to scale linearly up to 1080p, nor to scale linearly across processors, but it still illustrates the performance gap.



The other possibility, is that if you're using 64-bit builds, they're getting miscompiled, which would explain why 32-bit builds don't show the problem. I just downloaded the latest win32-static build from Zeranoe and the sample from the above test played back fine (albeit with the same quirks I also noticed in FFMS2, which coincided with some warnings I got back about length and output bytes or something).

[zambelli]

Why no Lagarith encoder in Ffmpeg? Lagarith is released under GPL, so from a licensing perspective it should be OK, right?

[LoRd_MuldeR]

Why no Lagarith encoder in Ffmpeg? Lagarith is released under GPL, so from a licensing perspective it should be OK, right?

Might be related to the fact that the original Lagarith code is Windows-only and implemented as a VfW Codec.

Also, if you need a lossless encoder, there already are some alternatives available in FFmpeg, such as HuffYUV, FFV1, x264...

[Leeloo Minaï]

@LoRd_MuldeR : where is the problem with the fact Lagarith is in VfW format ?
HuffYUV codec was originally released a VfW codec, so "windows only", but also as open source, all like Lagarith, and now HuffYUV is part of FFmpeg...

[LoRd_MuldeR]

@LoRd_MuldeR : where is the problem with the fact Lagarith is in VfW format ?
HuffYUV codec was originally released a VfW codec, so "windows only", but also as open source, all like Lagarith, and now HuffYUV is part of FFmpeg...

The "problem" is that you don't have a general purpose cross-platfrom encoder library that you can just take "as is" and plug it into FFmpeg ;)

Instead you have to "extract" (decouple) the actual encoder code from the VfW Codec wrapper. Depending on how "clean" the code is written/designed (haven't looked at in detail) this can be straight-forward or really annoying!

Also, code that compiles in Visual Studio and that never has been tested with other compilers is likely to not compile with GCC, especially if a lot of Win32 API functions/types or MSVC-specifics intrinsic are used. It's not fun to fix that :rolleyes:

After all, this all should be very doable, but somebody has to do it! Feel free to port the Lagarith encoder and submit a patch to the FFmpeg and/or Lavc guys...

[foxyshadis]

Porting might not be too difficult, but it isn't in there because no one ever did it, the author doesn't want to, and it must not be a high priority for volunteers.

Lagarith itself is written with intrinsics, which is always nice and makes porting easy, but also includes a bunch of intel-format custom asm code to do colorspace conversion. It also sets and reads options in a very windows-specific way, via the registry. Since ffdshow is primarily made for MSVC it could be included nearly as-is, the way libdts and faad are, but it wouldn't make it into ffmpeg. I don't think it'd be very hard, colorspace conversion is something better handled in libswscale than lavc anyway. Another serious problem is that some features are based on x86 quirks, like the way it performs floating point operations, that aren't be 100% lossless on non-x86 platforms without reimplementing floating-point logic in software. That's a huge part of the slowdown, and one reason some of the intrinsics can't be used.

(The FFDShow includion would be time-consuming more due to creating the GUI than anything.)

[tuqueque]

I'm no coder... But as a user, I LOVE Lagarith to record video tutorials, intermediate lossless content and stuff like that. I'd love to have Lagarith on FFmpeg. It compresses better than almost any other (lossless) codec and I'll use anything but x264/H.264 in my work. I think the best way to support real open source projects, is to avoid stuff based on proprietary/patented stuff (not trying to initiate a debate, I'm just exposing the searons why I use Lagarith).

In any case, we may wait for VP9's lossless mode.

[Dark Shikari]

FFV1 and Dirac are both (probably) royalty-free and should compress much better than Lagarith.

[zambelli]

Lagarith is royalty-free too (at least there's no mention of any licensing on its homepage). "This codec was build using the Huffyuv source as a template, and uses some Huffyuv code, most notably the routine to upsample YUY2 video to RGB and to perform pixel prediction on YUY2 video. Other colorspace conversion routines were taken from AviSynth. Lagarith is released under the GPL."

You're correct regarding compression efficiency, but performance wise Lagarith kicks FFV1's butt on a modern Intel i7 quadcore system. Here's a quick test:

Lagarith v1.3.27 x86
FFV1 (Ffdshow) Rev4499 20130104 x86
Source: 1920x1080 29.97 fps YV12

==========================
Lagarith = 0.601x realtime, 33.11% file size
FFV1 AC Large = 0.306x realtime, 27.95% file size
FFV1 VLC Small = 0.408x realtime, 30.04% file size


As you can see, FFV1 is 15% more efficient than Lagarith in its most complex mode (AC Large), but half the speed. IMHO, that's not worth it.

Another advantage of Lagarith over FFV1 is that it can be installed as a standalone codec in Windows, making it very easy to distribute Lagarith-compressed videos to people who aren't well acquainted with codecs, VfW, DirectShow, etc. Telling somebody to "download and install this one executable" is way easier than teaching somebody how to find the latest stable build of FFdshow, install it, and then configure to decode the right codecs.

[Asmodian]

Doesn't this speed/size ratio look even better for huffyuv? I remember it is much faster than Lagarith and only somewhat larger? I think it is all about picking the option that is "fast enough" while also being "small enough". Of course "enough" is project dependent.

[benwaggoner]

Another advantage of Lagarith over FFV1 is that it can be installed as a standalone codec in Windows, making it very easy to distribute Lagarith-compressed videos to people who aren't well acquainted with codecs, VfW, DirectShow, etc. Telling somebody to "download and install this one executable" is way easier than teaching somebody how to find the latest stable build of FFdshow, install it, and then configure to decode the right codecs.
And that is exactly why I use Lagarith instead of FFV1. I need to be able to send someone an .avi file and have them easily be able to do whatever with it in their existing Windows tools.

If there was a stand-alone installer for FFV1 that provided 32-bit and 64-bit VfW/DirectShow/MediaFoundation encoders and decoders, I'd be over the moon. If it had QuickTime components for Mac and Windows, I'd never use anything else.

[zerowalker]

Is it possible to optimize the Lagarith code to use AVX instructions and stuff?
The encoder is fast as it is, but it would be awesome if it´s possible to improve it.

But i don´t know much of programming, so can´t say if it´s possible or not.

[Dark Shikari]

Almost all the time in something like Lagarith is going to be spend in the arithmetic coder, which is almost necessarily not-SIMDable.

[zerowalker]

So, in other words, instructions doesn´t help Lagarith?

Dark Shikari, you wouldn´t know how to implement YCoCg support in Lagarith?

[foxyshadis]

Almost all the time in something like Lagarith is going to be spend in the arithmetic coder, which is almost necessarily not-SIMDable.

It's actually a reasonably fast range coder, and keeps separate state in both threads; most of the time seems to be spent in prediction, which is already SIMD'd to hell and back. (There is no straight-C fallback code, just SIMD for most operations.)

Dark Shikari, you wouldn´t know how to implement YCoCg support in Lagarith?

It's not knowing how, it's someone willing to do it. Although it might not be too hard to rip the guts out of ConvertToYCgCo and insert it into another conversion routine, it would be incompatible with every other Lagarith install out there right now.

I did do some testing with ConvertToYCgCo, though, and it does seem to help a little bit if you don't mind subsampling, but if color fidelity is important to you, YCgCo24 (or YV24) gains you almost nothing size-wise over RGB. Maybe 3-5%.

[zerowalker]

Oh well if it´s just 3-5% less size, then it´s pretty much useless, as it´s not perfect, which RGB is.

Is there a way to make Lagarith convert to YV12 correctly?

ConvertToYV12(matrix="Rec709")

For example, if the original need to use this matrix to be correct.
Lagarith will use Rec601 always it seems.

And i don´t think you can solve it afterwards, though if i am wrong, it would be nice.

EDIT:

Okay it seems to be fixable with ColorMatrix afterwards.
But would be nice for Lagarith to do it for many reasons.

[Dark Shikari]

It's actually a reasonably fast range coder, and keeps separate state in both threads; most of the time seems to be spent in prediction, which is already SIMD'd to hell and back.That seems kind of hard to believe -- median prediction only takes a few SIMD operations per 16 pixels, while even a highly optimized arithmetic coder like x264's takes a dozen or two cycles per output bit. Is something else odd going on here?

[zerowalker]

Is anyone able to decode Lagarith with the 64 bit version of Avisynth?
I get FourCC error.

I can decode it if i use Lavfilter´s decoder for it though, but it´s a bit problematic.

[sh0dan]

A kind soul would add multithreaded colorspace rgb<->yuv conversion as well - a fairly straightforward optimization, which would make both encoding+decoding faster.

[Lenchik]

Can anyone add YV24 support to Lagarith, please?

[SirLagsalot]

I am experimenting with re-writing Lagarith to support high-bitdepth (10 bits/color channel or more). To do this though, I need to know what color formats would be the most useful, and sample videos in those formats. Hopefully, some people in this thread would be willing to provide such video. My criteria for test video is as follows:

Must be from a source that uses the full bitdepth (upsampled video will not properly test the compression).
Should not have lossy compression applied
Should not have been resized
Uncompressed AVI would be the desired format, since it guarantees that I will be able to access the video
Should be short to keep the file size reasonable (100 frames or so is plenty)

[zerowalker]

Don´t have any sample videos, but can say state some color formats.

First YV24, even though it´s mostly unnecessary, it shouldn´t be to hard to implement while you are at it.
And then Rec.709, which is extremely useful but very rarely supported in a lossless codec.

Well that´s probably about it now that i think about it, i hope that was something you wanted to know, or perhaps you wanted to know something about bitdepth and such, if so i provided the wrong information.

I hope you see this through, Lagarith is an underestimated codec, and what´s it mostly is lacking is support and perhaps tweaks for encoding/decoding with newer CPU instructions.

EDIT:

Original 10bit videos will probably be quite hard to obtain, but it´s fairly simple to make one.

Take a video with much darkness and gradients. Fix it with Avisynth and work in 16bit.
Dither it to 10bit.

That way it won´t be upsampled from 8bit for no reason, and there should be pretty useful information in the last 2 bit that should have impact on compression.

But of course a 10bit Broadcast tape would be preferred, but can´t see a way to obtain one if you don´t have some good contacts.

[qyot27]

The better reason to support YV24 (and YV16, YV411, and Y8) is because that rounds out all the colorspaces that AviSynth 2.6 supports.

And really, if Rec.709 support gets added, might as well future-proof it a bit and also add Rec.2020.

[raffriff42]

ffmpeg can make nice high bit depth / deep color test videos...
EDIT test ramps seem to be generated with 8-bit math (http://www.ffmpeg.org/doxygen/1.0/vsrc__testsrc_8c-source.html#l00454) - sorry for not checking previously.
Try this thread: http://forum.doom9.org/showthread.php?t=158836

[SirLagsalot]

I can use upsampled or generated high-bitdepth video for developing and testing for losslessness, but to really tune the compression algorithms and speed optimizations, I need real footage. The upsampled/generated video simply won't have the same statistical distribution for the synthesized bits, and that could skew the results. I don't need test video immediately either, just in the next couple of months.

Since this would be a re-write, I would like to simplify how color formats are handled so it is easier to support more of them. YV24 and YV8 would probably be added, and Lagarith can already support encoding YV16.

I don't think I will add an option for Rec.709 (or Rec.2020), since it is another option the user has to get right or they can introduce noticeable loss. I feel that is an option better left to the encoding or decoding program.

[raffriff42]

OK try: http://media.xiph.org/tearsofsteel/ ("Tears of Steel" raw files)
http://blender-mirror.kino3d.org/peach/ ("Big Buck Bunny" raw files)
http://www.digitalbolex.com/d16-in-the-wild/ (Digital Bolex test footage)
EDIT http://blender-mirror.kino3d.org/peach/peach/filmmore/dpx-log/ (frame of "Big Buck Bunny" in a variety of deep color pixel formats)
http://blender-mirror.kino3d.org/peach/peach/filmmore/greyscale/ (grayscale test images in a variety of deep color pixel formats)
http://djv.sourceforge.net/ (DJV, an excellent deep color viewer & converter for images, image sequences, .MP4s)