Modification of huffyuv codec for YV12? [Archive] - Page 2

Boulder

4th January 2003, 18:31

Originally posted by HarryM
@Boulder: It is depend on content of video, of course. But (I hope) -h's YV12 modification is surely more sparing than Huffyuv_YUY2.
Only upgrading from YUY2 to YV12 (16bits<>12bits) make sparing effect=25%, theoretically.

I know, that's why I said estimates;)

At the moment I'm unwilling to use the YUY2 version for anything longer than 60 mins so hopefully I'll be able to extend the caps using -h's version to a full movie length:cool:

Suzahara

4th January 2003, 20:54

Originally posted by -h
Sure - it's finished, I just have to run some test video to produce optimized huffman codes. Then make sure it's actually faster than huffyuv :)

-h

Excellent news. I got kind of worried when this thread wasn't active for a while :D

wotef

7th January 2003, 14:33

sign me up, too, if you need help testing, heh

can't wait to get at this! :D

sh0dan

7th January 2003, 18:30

Don't know if this actually help, but here goes.

When you have a look at how U/V is mapped in typical movie (non-anime) sources, they are mostly mapped pretty close to 128. (Just try UtoY() in AviSynth 2.5).

Y is typically mapped between 16 and 128 (try histogram()).

Are the tables automatically adjusting to this, or could the tables be optimized for this situation?

IMO it could also be a very good thing to be able to clip invalid YUV values, so that Y and UV values below 16 are mapped to 16, Y values are maxed at 235 and UV are maxed out at 240. For sensible souls it could be made an option (for huffyuv to be truly lossless), but these values are invalid and does not contribute to the visual quality. A very fast ISSE implementation is in "levels.cpp" at the bottom.

Clipping could lead to better compression on sources that have been colorspace-converted, or have compression artifacts left.

-h

7th January 2003, 19:13

Are the tables automatically adjusting to this, or could the tables be optimized for this situation?

The prediction is just the median of the left, top and gradient (left + top - topleft) values, with the error being stored. Context modelling could be performed since as you point out some error values are not possible, however it would slow things down and wouldn't save many bits - typical error images are heavily weighted towards small errors (87% of errors are -2 <= x <= 2 in the Y plane, for the U and V planes 94% are -1 <= x <= 1!).

Clipping could lead to better compression on sources that have been colorspace-converted, or have compression artifacts left.

I hadn't considered adding a clipping option, that's a fine idea.

-h

Boulder

7th January 2003, 19:42

Let him finish the initial release before you postpone it by your marvellous (I really mean it!) ideas, sh0dan:devil:

sh0dan

7th January 2003, 19:55

Ideas never hurt - and I would like to see an YV12-version ASAP too. On the other hand, things like this must be in place before any formal release, otherwise you'll just end up having to do an incompatible version later - with a lot of angry users.

Just look at the 2.0 vs. 2.5 plugin compatibility - how many users haven't complained because the alpha version just crashed on 2.0 plugins - even though it has been clearly marked as alpha material - and I'm just awaiting the uproar, when old 2.5 plugins cannot be loaded because users haven't updated them or the developers haven't changed their code yet. (sorry - just had to get it out ;)

-h

7th January 2003, 20:16

I will definitely be releasing incompatible versions after the initial release (too many ideas to try), however new builds will always decode old versions.

I am assuming that people won't mind all that much, since lossless video codecs are rarely used for long-term archiving.

-h

Boulder

7th January 2003, 20:40

Originally posted by sh0dan
Ideas never hurt - and I would like to see an YV12-version ASAP too. On the other hand, things like this must be in place before any formal release, otherwise you'll just end up having to do an incompatible version later - with a lot of angry users.

It was pure sarcasm on my behalf, dear friend.. it's like we're all kids waiting for our delayed Christmas present here:D And kids can get so impatient at times!

cweb

9th January 2003, 16:51

With the current huffyuv the only valid depth is 24 bits. Could this
be extended to 32 bits (so I can use it in Alamdv2), for YV12
and the current implementation? I was wondering. Or is it the huffyuv which I installed which is perhaps an old one?

Boulder

15th January 2003, 21:37

How's the testing progress going, shall we soon be able to enjoy the benefits of YV12 captures?

Suzahara

19th January 2003, 04:52

Heh, I'm getting worried since there hasn't been status updates :o ...Is it near release time?

-h

19th January 2003, 05:19

I guess it is? I'm not terribly good at hitting release schedules, in case no one's noticed :)

This has actually been my first week at a new job and I've been having a blast, partying most nights and recovering most days. But I have all of tomorrow afternoon off, and I want to release both this and the MSMPEG4V3->MPEG4 convertor in a grand display of bugs and ignoring feedback. Should be a great show.

There is also another eerie secret project which I'm having a lot of fun working on, even though I know it's going to haunt me further down the line.

-h

Suzahara

19th January 2003, 05:22

Originally posted by -h
There is also another eerie secret project which I'm having a lot of fun working on, even though I know it's going to haunt me further down the line.

Eerie projects are always good...especially secret ones :D

Boulder

19th January 2003, 09:36

Originally posted by -h
There is also another eerie secret project which I'm having a lot of fun working on, even though I know it's going to haunt me further down the line.

-h [/B]

Don't worry, you can tell us - we're really good at keeping secrets:devil:

ales19

27th January 2003, 00:28

any news from the front? :D :D

-h

27th January 2003, 03:59

Yeah, I'm sick of seeing this in my dev/unfinished folder. Tomorrow I only work 11am-2pm, and have all of the day after off. I will finish this and the MSMPEG4V3->MPEG4 convertor. Or else.

-h

digitize

27th January 2003, 05:05

Im glad to hear it -h, and I thank you for all the hard work you've put in :)

-h

27th January 2003, 05:22

My working hard is a vicious rumour, and I'm yet to discover who started it.

-h

digitize

27th January 2003, 08:01

ah im sorry, well thank you for working period.

ales19

27th January 2003, 21:56

Originally posted by -h
Yeah, I'm sick of seeing this in my dev/unfinished folder. Tomorrow I only work 11am-2pm, and have all of the day after off. I will finish this and the MSMPEG4V3->MPEG4 convertor. Or else.

-h

yay, im really impatient to test this out :)
thanks man

wotef

28th January 2003, 13:44

Boulder

2nd February 2003, 09:32

*a shameless bump*

Boulder

5th February 2003, 11:36

Any news yet? :devil:

Suzahara

6th February 2003, 19:09

No news/release dates yet? :confused:

I'm starting to fear the worst again :( :scared:

Guest

6th February 2003, 19:35

POLITE MODERATOR REQUEST: Gents, please stop the pointless harrassing posts. When and if it is ready it will be announced I am sure. Thank you.

FishB8

7th February 2003, 20:50

The work on this lossless codec has me wondering, would it be much work to store this in an Ogg container? Xiph has lossless (FLAC) and lossey (Vorbis & Speex) audio, and lossey video (Theora & Tarkin) but they have no lossless video. If there's no patent issues with HuffYUV they might even consider sponsoring it if you could store it in an Ogg container.

Had to spend my 2¢ somewhere... :)

Suzahara

8th February 2003, 15:30

Originally posted by FishB8
The work on this lossless codec has me wondering, would it be much work to store this in an Ogg container? Xiph has lossless (FLAC) and lossey (Vorbis & Speex) audio, and lossey video (Theora & Tarkin) but they have no lossless video. If there's no patent issues with HuffYUV they might even consider sponsoring it if you could store it in an Ogg container.

Had to spend my 2¢ somewhere... :)

Well, I'd figure it's already storable in an Ogg Container. Divx and XVid are storable and I don't think any extra coding was put in just for those two codecs. Although I can't imagine why you'd want to do this. Huffyuv is extrememly high on size since it's around 2.0:1 on compression. This makes filesizes big really fast. So unless you've got a ton of space to use up, most people won't actually do this.

Belgabor

8th February 2003, 20:50

Well, what we all want to do at some point (at least I hope all want to..) is replace avi completely, so its perfectly legal to want it usable in other containers. But as you said, thats already possible :)
Another point you have to consider is that a) -h wanted to improve on compression and b) that YV12 in itself is already smaller than YUV2

Suzahara

9th February 2003, 16:05

Yeah, avi is an ancient format that needs to be replaced by more flexible formats.

But I still don't believe that you would want huffyuv or huffyv12 as a long term storage thing since I don't think you can improve compression all that much (meaning it will still take up GB's very quickly) since lossless requires space and until we can find a lossless solution that can compress as much as divx or xvid, huffyuy(yv12) will still only serve as an intermediate file between (unfortunately) lossy formats. :(

FishB8

10th February 2003, 05:04

But I still don't believe that you would want huffyuv or huffyv12 as a long term storage thing

No I wouldn't. I would want to use it more for a video editing purposes. I'm a linux user and right now all the video editing solutions use a hodge podge of avi, quicktime, DV, and various types of MPEG. I would prefer a container format more native to linux, such as Ogg. I know Ogg was origionally developed to be a delivery container, but I don't imagine it would be too tough to use it for editing.

what we all want to do at some point (at least I hope all want to..) is replace avi completely, so its perfectly legal to want it usable in other containers. But as you said, thats already possible

Exactly, but are there any tools that make this possible on linux...

^^-+I4004+-^^

11th February 2003, 00:43

"2003-01-23 07:06 milan_cutka

huffYV12 encoding, graph currently disabled"

from:
http://homepages.pathfinder.gr/ffvfw/

i didn't tested or checked anything,just saw this changelog...

jcsston

11th February 2003, 09:04

Originally posted by ^^-+I4004+-^^
"2003-01-23 07:06 milan_cutka

huffYV12 encoding, graph currently disabled"

from:
http://homepages.pathfinder.gr/ffvfw/

i didn't tested or checked anything,just saw this changelog... :( I tried this out, but I couldn't decode will anything. ffvfw gave me a distorted picture and YUV2 Huffyuv crashed

^^-+I4004+-^^

11th February 2003, 19:09

Originally posted by jcsston
:( I tried this out, but I couldn't decode will anything. ffvfw gave me a distorted picture and YUV2 Huffyuv crashed

he did put decoding capability few hours (i presume by the changelog)
afterwards,but anyhow,have you tried ffdshow?
(currently i don't have such recent versions of either ffvfw or ffdshow and ,frankly,i cooled down from all this huffyv12 stuff,as
anything larger than 2-3MB/s is too big for me so i'll stuck with mjpeg...)

fisix

14th February 2003, 09:44

in agony, i bump.

Marc FD

14th February 2003, 21:37

hi.

i'm currently playing with yv12 lossless compression.

about compressibility improvements with yv12.
IMHO, it's not big. because chroma planes are VERY compressible (ratios often over 4.0:1) and luma as hard to compress than in yuy2 ^^

if you have a very hard to compress frame, like one of the frame of my test clip, where luma is 1.6:2, chroma ~4:1, it gives around 2:1 in overall in yv12, but it would be better in yuy2 (~2.5:0), because the additionnal (4bpp) chroma data is more easy to compress ^^

Cheers,
MarcFD

Boulder

14th February 2003, 21:45

Marc,

what about near-lossless compression, would that be hard to accomplish? The main problem is that we don't have a YV12 codec for capturing so a possible minor loss of details wouldn't hurt. I believe there was some talk about that elsewhere in this thread. I think people use mainly MJPEG for capturing and with high enough quality settings, the result is near-lossless IMO.

What do you think? I think that -h's too busy nowadays so maybe you could come up with something for us capture freaks?

Richard Berg

15th February 2003, 04:50

fisix

16th February 2003, 13:08

Indeed, I recall -h saying that merely allowing for +/- 1 error would dramatically increase compressibility. Since YV12 already throws away substantial information with most capture cards vs. HuffYUV, I think such a codec would be a most useful compromise, at least until someone is able to make JPEG2000 run at 30fps

what are we giving away by using yv12 with capture cards? is this just saying that there is a conversion from yuv2 to yv12 when using most cards because they only output yuy2?

my card is one of the many bt8x8 cards from hauppauge, and the driver gives the option of 4:2:2 packed AND YUV12 planar. if the wintv cards support it, doesn't that almost mean most cards support it?

anyway, a codec that isn't lossless isn't huffyv12. its something else.

as an aside, i saw someone write something about poor decoding ability of huffyuv, and i wondered if the decoding speed was something -h was looking into also. see this thread, post by aktan:

http://forum.doom9.org/showthread.php?s=&threadid=35387

sh0dan

16th February 2003, 17:17

The decoding speed of huffyuv is quite good. The missing DirectShow interface is mainly what makes it appear to be slow.

Marc FD

16th February 2003, 22:19

>what about near-lossless compression, would that be hard to accomplish?

mhh. not really. dunno how huffyuv works. but i think it's useless.

BTW, the true-lossless thing is what we seek, because for lossy compression, MJPEG will _always_ win.

>The main problem is that we don't have a YV12 codec for capturing

my old PCTV gives me YV12/I420, so with a yv12 lossless codec, you should be able to capture in yv12 ^^.

>so a possible minor loss of details wouldn't hurt. I believe there
>was some talk about that elsewhere in this thread. I think people use
>mainly MJPEG for capturing and with high enough quality settings, the
>result is near-lossless IMO.

yes, but it _is_ lossy. if you reencode 10x a clip with a lossless codec, you still have full quality.

>What do you think? I think that -h's too busy nowadays so maybe you >could come up with something for us capture freaks?

héhé. -h was too slow, so i did something this WE ^^.
honestly, coding a codec from scratch in a WE is really not easy ^___^

compression algo is 100% mine, and it's really not bad (that why i did a codec with). i'll release an alpha for you all to play with tomorow.

VBLE Video Codec, sound good, no ? ^^

>Indeed, I recall -h saying that merely allowing for +/- 1 error would >dramatically increase compressibility.

maybe around 5-10%, but i don't think more. my algo achieve much more, and it's truly lossless.

Boulder

16th February 2003, 22:49

Originally posted by Marc FD

if you reencode 10x a clip with a lossless codec, you still have full quality.

compression algo is 100% mine, and it's really not bad (that why i did a codec with). i'll release an alpha for you all to play with tomorow.

I meant near-lossless codec for capturing. As an intermediate codec a true lossless one should definitely be used, I agree.

I can hardly wait, lots to capture:cool:

vidiot

16th February 2003, 22:50

Neuron did say that it won´t take long til we see you back MarcFD!

And I´m glad he is right!:D

Knowing all your filters I guess we´ll see a pretty fast "alpha" these days.

Curiously
Harald

digitize

17th February 2003, 20:29

Ah cool, i cannot wait to see marc fd's codec, he usually codes good stuff and Im sure this will be the same :D

McQuaid

17th February 2003, 21:58

In regards to capping, I checked and I don't have yv12 for my capture card but I do have yuv12 are they the same? Also the other post about near lossless wouldn't be useless for capturing to save space, would be handy feature. It would be nice to have something similar to mjpeg compression sizes while staying in the same colour space throughout.

Blight

18th February 2003, 01:19

Regarding decoding, FFDShow decodes huffyuv and is faster than using the huffyuv codec as it is directshow based.

Marc FD

18th February 2003, 19:37

>>Neuron did say that it won´t take long til we see you back MarcFD!

wow, wow, not so fast.
i never coded a codec, so i thought it would be fun.

but i'll get a dsl connection very soon, and i don't think i'll have 5 minutes free for devel then ^^.

here's a beta of VBLE. compression is close to huffyuv, with 25% more thanks to yv12 colorspace.

encoding part is pure SIMD (isse processor required, no check)
decoding is pure C, and hard to optimise.

it's way slower than huffyuv, but it should be fast enough for capture on modern cpus ^^.

(attached "VBLE Video Codec (beta version).zip")

Install instruction : dezip & install using the .inf file ^_^

enjoy,
MarcFD

>In regards to capping, I checked and I don't have yv12 for my capture
>card but I do have yuv12 are they the same? Also the other post about
>near lossless wouldn't be useless for capturing to save space, would
>be handy feature. It would be nice to have something similar to mjpeg
>compression sizes while staying in the same colour space throughout.

well, a hyper-fast yv12 native wavelet codec would be cool, sure, but my math skill is way too limited for that (remember i'm 17 years old)