Most interesting open source

[jakor]

Hello guys !
What do you think, what the most interesting open source project in the field of video codecs and more general - image and video processing currently is ?
Do they need some programmer force to move on ?

[bratao]

The snow Wavelet codec is a very interesting project that need urgent a developer.
http://ffmpeg.mplayerhq.hu/

[jakor]

what tasks need to be done ?

[nm]

See http://wiki.multimedia.cx/index.php?title=FFmpeg_Summer_Of_Code_2008#Snow_Job

Apparently (http://code.google.com/soc/2008/ffmpeg/about.html) the task wasn't picked up by any student this year. If Snow doesn't sound interesting to you, there are many other tasks in FFmpeg, just waiting for people to grab them. Read the FFmpeg Summer of Code page for more suggestions.

[PatchWorKs]

Note that SNOW is open but not royalty free.

Choose BBC's Dirac Video Coded (http://dirac.sourceforge.net/), instead.

Or check out the Fluendo's high-speed implementation called Schrodinger (http://www.diracvideo.org/).

Xiph's Theora (http://www.theora.org/) is cool too, but a bit old...

[Dark Shikari]

Note that SNOW is open but not royalty free.Uh... what? Snow is free software. If you're meaning to say its not free of patents--well neither is most of anything else in video.
Choose BBC's Dirac Video Coded (http://dirac.sourceforge.net/), instead.Dirac has its own problems, like the fact that it only allows CQM-style adaptive quantization (subband-based), not spatial, or its absurdly complicated motion compensation filter.

The main problem with both Snow and Dirac is that wavelets have proven to be pretty useless in video coding for two reasons:

1. Spatial intra prediction is basically impossible.
2. Wavelets have this odd property of being extremely PSNR-optimal while looking like crap.

[jakor]

Dark Shikari, what would you recommend ?

[jakor]

Sun's OMF is going (promised) to be royalty free

[Dark Shikari]

Dark Shikari, what would you recommend ?x264? ;)

[jakor]

what tasks are left in x264 ?

[Inventive Software]

SVC, PAFF, making it the best encoder ever! :D

[Dark Shikari]

what tasks are left in x264 ?Plenty, if I had to name a few:

1. Better B-frame decision that takes into account pyramid
2. Multi-level B-pyramid
3. Optimizing the inter mode decision process
4. Psychovisual optimizations for mode decision and quantization
5. Adaptive MBAFF
6. 4:2:2 and 4:4:4 colorspace support

and of course a lot of smaller projects too, and loads of things that I couldn't name off the top of my head.

Oh, and MMCO and better use of weighted prediction.

Hop on #x264dev on Freenode if you're interested in helping with anything--we can always find you some sort of project.
making it the best encoder ever! :DAlready done that one.

[Inventive Software]

Not according to the hyperbole of other companies. :p

[Dark Shikari]

Not according to the hyperbole of other companies. :pAll commercial encoders are three times faster, twice the quality, and exponentially more reliable than all other commercial encoders.

[video_magic]

I give x264 my vote too - my archiving project from tapes to digital video files will use this codec. I enjoy the quality to filesize ratio but am keen to see it more mature and further optimised - I thank the people who have worked on it very much!

For container formats I vote for Matroska which I have followed with interest for maybe 7 years? or whenever I first heard of it. I have lost some interest because I think they want developers so many users can't simply just download and use it. How about helping the Matroska project?

[IgorC]

I totally agree that x264 is the most interesting and useful open source project on compression area.
There is also some work on AAC encoder.
http://codecs.multimedia.cx/?cat=18

I checked the latest version of Apple AAC with true VBR and optimal settings and I found that Apple LC-AAC produced very good quality even on 80-85 kbit/s on par with Nero 96 kbit/s. Even better than MP3 VBR 5 130-135 kbit/s. There is still *big* room for improvement. An open source LC-AAC could make even better.

SBR (HE-AAC) is making less scene each time. It's not transparent.
It's more preferable to rise a little bit bitrate (to 80-85 kbit/s) where LC-AAC has already good quality.

[MeteorRain]

The main problem with both Snow and Dirac is that wavelets have proven to be pretty useless in video coding for two reasons:

1. Spatial intra prediction is basically impossible.
2. Wavelets have this odd property of being extremely PSNR-optimal while looking like crap.
so does this mean wavelet based codec have no future?
and DCT is still the best choice?

[jakor]

what is the problem with faac and faad - why they need to be rewritten in ffmpeg ?

[nm]

what is the problem with faac and faad - why they need to be rewritten in ffmpeg ?
AFAIK, the main reason in case of FAAD is that it's licensed under GPL while FFmpeg prefers LGPL (or BSD/MIT). Many other decoders and encoders have been written from scratch for the same reason. A secondary reason for reimplementing codecs is that existing source code would need to be heavily modified and refactored anyway because FFmpeg's policy on code quality is pretty strict. This might be the case with FAAC along with its limited feature set.

Implementing a decoder or an encoder is also good practice for a student, and FFmpeg gladly accepts well written new codecs even when there isn't high demand for them.

[jakor]

I see... thanks for the explanation !
However, waiting from a student to write a complicated codec like this from scratch looks like a pretty hard problem for me ;-))
I've been teaching students in the field of multimedia compression for five years, giving them real life problems and codecs etc and from my experience the best students completed for several months just some part of an algorithm included in a video codec, like MC, or IDWT or deblock or whatever. But I had best students from one (not big) city, while google searches over the best students in the world ;-)

[CruNcher]

GPU support for X264 would be a nice addition, the quality of X264 is allready @ a very very good level now the speed side of things could get enhanced, ofcourse the more efficient decissioning of all kinds of stuff like Dark said is also something that needs to be worked on (for the perfect visual experience), but all in all H.264 and especialy X264 rocks currently everything. I even would say it will have a biger impact then Mpeg-2 had back the time. :)
Jesus just compare the times people had Mpeg-2 Decoder cards Gfx card accelleration was alot later and we allready have both Decoding/Encoding on 1 Card it's crazy compared with the old times. And the Quality i did Full SD back in ASP times with 1 Core in very good quality @ 15 fps (RD,Qpel) now we do better quality with the same MHZ and above 30 Fps and Dualcore 60 Fps it's just crazy and how fast X264 improves :)

[Dark Shikari]

so does this mean wavelet based codec have no future?
and DCT is still the best choice?At least right now, there are serious issues with wavelet/OBMC formats. I don't see any of them becoming major competitors in the next decade without a significant miracle.

[CruNcher]

At least right now, there are serious issues with wavelet/OBMC formats. I don't see any of them becoming major competitors in the next decade without a significant miracle.

Im sure Dirac will guarantee alot of "Fresh Blood" getting interested in Wavelet Video Codecs and trying to improve it in the coming years, at least 3 Next Gen Wavelet Codecs are available under Open Source to Research on Snow,Dirac and Rududu.
It will be interesting to see how Dirac and X264 are going to compete Visualy against each other and the improvements being made, X264 has still a head start compared to Dirac so let's wait and see whats coming :)

[Dark Shikari]

Im sure Dirac will guarantee alot of "Fresh Blood" getting interested in Wavelet Video Codecs and trying to improve it in the coming years, at least 3 Next Gen Wavelet Codecs are available under Open Source to Research on Snow,Dirac and Rududu.
It will be interesting to see how Dirac and X264 are going to compete Visualy against each other and the improvements being made, X264 has still a head start compared to Dirac so let's wait and see whats coming :)Dirac's bitstream format is finalized (I think?) and it has no spatial adaptive quantization.

Ergo, its doomed.

[smok3]

At least right now, there are serious issues with wavelet/OBMC formats. I don't see any of them becoming major competitors in the next decade without a significant miracle.

how about cineform (or the usage of wavelet codecs as intermediate codecs in general)?

[*.mp4 guy]

Waveletes do very well as intermediate formats, they perform well at the high end of the bpp scale compared to the alternatives, and when you are working with such high data rates, perceptual quality is less important then absolute difference from the source, all formats will be perceptually lossless, so the best choice is clearly the most objectively optimal option. This is why a lot of high data rate formats are based on Jpeg2K, D-cinema is supposed to be j2k (though isn't in practice), many high end raw formats are based on wavelets. The dct doesn't discretize very well, which tends to count against it when lossless, or neer lossless bitrates are used, the integer-only discrete aproximation used in mpeg4-avc solves this problem, but at the expense of some performance.

[smok3]

yes, i think that is also what Dirac is about (i do briefly remember two BBC guys from IBC showing a little blackbox with dirac painted onto it, saying it reduces bitrate to half, for intrahouse room2room transmition of HD data)

[akupenguin]

The dct doesn't discretize very well, which tends to count against it when lossless, or neer lossless bitrates are used, the integer-only discrete aproximation used in mpeg4-avc solves this problem, but at the expense of some performance.
The HCT isn't any better or worse at near-lossless than any other integer approximation of the DCT (such as is used by implementations of formats that officially ask for the real DCT). The benefits of HCT are: speed, and it's a specific approximation rather than letting each implementation choose their own. But aside from speed, standardizing any conventional DCT approximation works just as well (e.g. Theora picked the 16-bit Chen factorization).

[*.mp4 guy]

I was under the impression that some of the other aproximations of the dct performed significantly better then the hct. IIRC some mpeg2 encoders and decoders use very high precision tranforms. when I said the hct loses some performance, I meant across the board, not in any particular area. When I said it solved the problem, I meant that it doesn't have an upper quality bound dictated by rounding areas, like some standards, such as jpeg suffer from in practice.

[Dark Shikari]

I was under the impression that some of the other aproximations of the dct performed significantly better then the hct.If by "significantly better" you mean "about 1% better compression at most," yes.

IIRC some mpeg2 encoders and decoders use very high precision tranforms. when I said the hct loses some performance, I meant across the board, not in any particular area. When I said it solved the problem, I meant that it doesn't have an upper quality bound dictated by rounding areas, like some standards, such as jpeg suffer from in practice.As long as the forward and reverse transforms agree on the rounding error, there is no loss of precision.

[*.mp4 guy]

If by "significantly better" you mean "about 1% better compression at most," yes.
At the high end the difference can be as high as 2db, compared to the idealized dct. Though in practice, compared to other real world implementations, around 1% sounds right.

As long as the forward and reverse transforms agree on the rounding error, there is no loss of precision.mpeg4-avc is the first standard that agrees on rounding for forward and reverse transforms, all of the others don't, hence all of the others almost always have rounding based precision problems.

[Dark Shikari]

mpeg4-avc is the first standard that agrees on rounding for forward and reverse transforms, all of the others don't, hence all of the others almost always have rounding based precision problems.VC-1 also has a specified rounding transform.

[Manao]

mpeg4-avc is the first standard that agrees on rounding for forward and reverse transformsIt only specifies the reverse transform. Forward is used only by the encoder, and, as such, has no need to be specified.

[*.mp4 guy]

I had forgotten about vc-1, I guess MS learned from their mistakes in msmpeg4.

[jakor]

I came up to an interesting idea - is there an interest, to create a kind of a professional SVC encoder application based on x264 ? Is there someone wishing to sponsorship this project ?

[Manao]

Open source community, unless company-driven, has no use for a SVC encoder, since the usefulness of SVC resides only in some very specific use-cases, such as IPTV streaming.

A SVC decoder might be interesting, provided SVC does get use and doesn't end up like previous scalability attempts. But in the case of the decoder, the basis would be ffmpeg, not x264.

[CruNcher]

Manao i guess Avail has a very big interest in it the same for Joost, also seeing the fact that Microsoft has allready a very stable scaleable approach for high network congestion and their MBR switching, in it's newest incarnation shows that off pretty well on nbcolympics.com it's nice how it scales to the network in small steps that visualy dont interfer alot with the viewing experience :)

[MfA]

so does this mean wavelet based codec have no future?
Depends a bit on what you call a wavelet, it's a really broad term.

For instance read this paper :
http://www-ee.uta.edu/msp/pub/CAS_02_2.pdf

I wonder why lapped block transforms like in that paper weren't considered within H.26L ...

[akupenguin]

I wonder why lapped block transforms like in that paper weren't considered within H.26L ...
Same problem as conventional wavelets: requires slow OBMC, and incompatible with IIR intra prediction.

[MfA]

Same problem as conventional wavelets: requires slow OBMC
H.263 style OBMC is 4 shifts and 3 adds per pixel ... what's the problem? OBME is slow, but OBMC not so much.

There's not a whole lot of difference between the pre-filter and OBMC ... so I don't think you would require it with this transform though (ME remains harder).
and incompatible with IIR intra prediction.
Intra isn't that important, coefficent prediction also works to an extent and you get pure gradients for "free" from the regularity of the post filter ... without experiment I wouldn't count this as a definite loss.

If you really wanted you could break the causal bond by not letting the pre/post-filter work across boundaries to an intra block, mirror instead, then you could code them however you wanted (and use standard deblocking to clean them up).

[Dark Shikari]

Intra isn't that important, coefficent prediction also works to an extent and you get pure gradients for "free" from the regularity of the post filter ... without experiment I wouldn't count this as a definite loss.If someone figured out how to do proper intra prediction in a wavelet codec it would be pretty big news...

And intra is only unimportant if you're designing a wavelet codec and want an excuse for why your intra coding sucks ;) </cynic>

[rkalwaitis]

I just checked out snow, and thought it wasnt to bad. Perhaps a bit on the slow side, when crunched. Not near as fast as Xvid, but I bet its close in quality if not better especially at lower bitrates. Of course x264 is faster but I think if a bit of effort was put into snow, it could rival the best of them. I used AutoFF, here on doom(ffmpeg gui). Now saying that my knowledge of lavcopts are poor at best I muddled through it and AutoFF gave me a respectable video.

[MfA]

If someone figured out how to do proper intra prediction in a wavelet codec it would be pretty big news...
Has anyone even tried the naive solution? Simply add a variable per coefficient to indicate inter/intra (with a per coefficient flag you can't afford to to do greedy R/D optimization for this variable obviously, so that gets a bit complex). A specific intra context model taking into account the inter&intra band causal neighbours should take care of the prediction.

[Dark Shikari]

Has anyone even tried the naive solution? Simply add a variable per coefficient to indicate inter/intra/intra-predicted (with a per coefficient flag you can't afford to to do greedy R/D optimization for this variable obviously, so that gets a bit complex).Coefficient prediction is generally rather inefficient though; the only particularly effective intra prediction is spatial, a'la H.264.

[MfA]

I shouldn't have said prediction ... prediction is handled implicitly by the context model, all you need is the inter/intra flag. Obviously it works well enough, as pure intra coders wavelets are certainly competitive ... so inefficient is a bit strong.

I don't really care for the shape of wavelet artifacts for low bitrate coding, but then again I don't really care for deblocking which ignores quantization intervals either.

[Dark Shikari]

I shouldn't have said prediction ... prediction is handled implicitly by the context model, all you need is the inter/intra flag. Obviously it works well enough, as pure intra coders wavelets are certainly competitive ... so inefficient is a bit strong.Competitive? Hardly (http://upload.wikimedia.org/wikipedia/commons/5/51/JPEG_JFIF_and_2000_Comparison.png). When something as ancient and inefficient as JPEG is what you're "competitive" with, you're not really competitive at all.

H.264/AVC Intra is quite a bit better.

[rkalwaitis]

what about this approach (hoping to learn something from you guys :) ) 3d-ESCOT

http://research.microsoft.com/china/papers/Wavelet_Codec_Using_3D_ESCOT.pdf

[MfA]

Competitive? Hardly (http://upload.wikimedia.org/wikipedia/commons/5/51/JPEG_JFIF_and_2000_Comparison.png). When something as ancient and inefficient as JPEG is what you're "competitive" with, you're not really competitive at all.
JPEG2K isn't the be-all and end-all of wavelet coders, it adds a lot of compromises, bitplane coding and progressively decodeable bitstreams come at a cost. The older SFQ is significantly better. Nonlinear multiresolution transforms such as the peak transform perform better still ... although they aren't really wavelet transforms.

[zambelli]

Manao i guess Avail has a very big interest in it the same for Joost, also seeing the fact that Microsoft has allready a very stable scaleable approach for high network congestion and their MBR switching, in it's newest incarnation shows that off pretty well on nbcolympics.com it's nice how it scales to the network in small steps that visualy dont interfer alot with the viewing experience :)
Despite advances in streaming technologies over the past few years, a scalable codec would still be revolutionary for Internet video streaming. Current adaptive streaming technologies don't scale well on the encoding end because they require multiple re-encodes of the same source. If you want 15 bitrates - you have to encode the source 15 times. A scalable video codec, on the other hand, would allow you to only encode the source once but create a multi-layered compressed video.

A good SVC encoder would also save bandwidth because, for example, in order to transfer 4 profiles ranging from 1 Mbps to 4 Mbps - you'd only need 4 Mbps of bandwidth. If you wanted to do that today without a SVC you would need 1+2+3+4 = 10 Mbps.

[Dark Shikari]

A good SVC encoder would also save bandwidth because, for example, in order to transfer 4 profiles ranging from 1 Mbps to 4 Mbps - you'd only need 4 Mbps of bandwidth. If you wanted to do that today without a SVC you would need 1+2+3+4 = 10 Mbps.This makes the assumption the scalability is free--it isn't. From what I've heard it costs around 15-20% efficiency.

Also, for internet streaming, SVC is basically useless because you don't have multicast, so you don't save any bandwidth (you have to send one stream to every user anyways). I can see its potential use in broadcast situations, but here you usually only have two streams, one SD and one HD, and the SD one is going to be about 20% of the bitrate of the HD stream... which means that the entire advantage of SVC is going to be absorbed by the reduced efficiency.