Modification of huffyuv codec for YV12?

[digitize]

hmmm, guess we'll have to wait for a moderator for the attachment.

[Koepi]

Hm, it works quite well, not a single dropped frame (but i checked max 3 mins for that).

Data rate is a bit too high for my limited space though :-/

Do you think you can do something about that Marc?

Btw.: congrats, it's a nice codec indeed :) And that from a 17 years old... :)

Respect!

Regards
Koepi

PS: I could scan you some of the scientific papers which I had to work through in Uni, but I don't think they're of any use for you %)

[Guest]

@MarcFD

Ha! Back even sooner than I expected.

Big questions for you:

1. Does your codec use interlaced or progressive downsampling?

2. Can you add an option to select interlaced or progressive downsampling?

3. Any chance of you releasing the source code?

Thank you.

[Schlumpf]

Originally posted by Marc FD
well, a hyper-fast yv12 native wavelet codec would be cool, sure, but my math skill is way too limited for that (remember i'm 17 years old)

You know...I'm already 20 but still have to take tuition in mathematics if I don't want to fail my final exams and I understand less than a fraction of the technical stuff posted here...
Damn, I feel so useless! :(


Anyway, thanks for this sweet codec. I will test it this weekend :)

[Si]

@marcFD
Excellent :D

In a quick test against Huffyuv -
I get about 10% decrease in compression speed with 26% decrease in file size :D

Decompression is 30% slower.

I'm also very interested in answers to neuron2's questions.

regards

Simon

[Guest]

I did some experiments and am almost certain the codec is downsampling using the progressive algorithm. Since I know MarcFD is quite well aware that his beloved XviD (as well as DivX) does only progressive upsampling, it stands to reason that he would have done it that way.

The problem is that it leads to color corruption if the source is interlaced. To prove this, download the attached AVI and encode it using his codec. Then view the original AVI and the encoded one. You'll see that fields have been blended together (and it is easy to show theoretically that it occurs).

The codec desperately needs an option to specify the downsampling as either progressive-type or interlaced-type.

[Guest]

Originally posted by Marc FD
well, a hyper-fast yv12 native wavelet codec would be cool, sure, but my math skill is way too limited for that (remember i'm 17 years old) http://www.maven.de/code/#wavelet

Might get you started thinking about things. :)

[Boulder]

This is slightly OT, but how does one enable YV12 format in VirtualDubMod in Windows XP? I tried setting the custom format to YV12 but all I get is a green screen and a very slowly responsive VDubMod. My card's a Hauppauge WinTV Theatre and I'm using the BTWinCap drivers.

Switching the format works fine in Win98 with Hauppauge VfW drivers and the codec seems to work okay:) I just want to get rid of Win98 at last, video capturing is the only reason why I've still kept a dual-boot system.

[ACClarke]

@MarcFD
I see that you reinstall your Visual C++ :D
It's good to see you back

I'm sure you make a great job with this codec but I don't have a computer to test it......

I will be back in april, it's hard to wait !

[Marc FD]

>Data rate is a bit too high for my limited space though :-/

sure, a so simple lossless codec is not going to do miracles ^^.

>Do you think you can do something about that Marc?

well, the whole stuff is a choice. there are 3 parameters : quality, speed and size. you cannot have them together. here, the choice is quality (lossless ^^) and speed (70 cycles per pixel in encoding). size is of course terribly big ^^. but it's temporal storage (i hope ^^).

>Btw.: congrats, it's a nice codec indeed And that from a 17 years old...

^^. i'm more close to 18 than 17 ^^.

> Ha! Back even sooner than I expected.

maybe not, because i'm not really back ^^.

it was just to wait for my dsl line (because i'm in holidays) and you can guess than as soon as i get it, i'll (almost) disappear from devel stuff.

>Big questions for you:
>1. Does your codec use interlaced or progressive downsampling?

progressive.
oups, didn't even thinked at that !

>2. Can you add an option to select interlaced or progressive >downsampling?

er, doing a codec wasn't too hard, but my knowledge of winapi is _so_ bad than i don't even no how to draw a single dialog box. and to be honest, i simply don't want to know ^^.

my advice : use avisynth 2.5 beta, and you would be able to do interlaced downsampling then (i.e. with the stuff i gave to you) and to give yv12 to the codec, it's gonna be (almost) as fast.

>3. Any chance of you releasing the source code?

you know how lazy i am for doc/src/ect.. and for 1000 lines of crap like that, i don't think it does worth it ^^.

but since i'm using GPL code, and because i'm for opensource dev, i'll give the source if someone _really_ wants it.
but it'll take a lot of time, because i don't want to clean the stuff (and i can't release it without some cleaning) and i don't want to play with code (so i'll do it, but slowly ^^).

>I did some experiments and am almost certain the codec is downsampling >using the progressive algorithm. Since I know MarcFD is quite well >aware that his beloved XviD (as well as DivX) does only progressive >upsampling, it stands to reason that he would have done it that way.

i'm not sure, but try to activate interlacing in XviD. you'll be surprised, it's doing much more than only field-based DCT, if i rememeber well. (even if i never tested it ^^)

>The problem is that it leads to color corruption if the source is
>interlaced. To prove this, download the attached AVI and encode it
>using his codec. Then view the original AVI and the encoded one.

well, give interlaced yv12 and everything is okay. btw, if you encode with your TV card, yv12 should be interlaced. so, no problem ^^.

>You'll see that fields have been blended together (and it is easy to
>show theoretically that it occurs).

>The codec desperately needs an option to specify the downsampling as >either progressive-type or interlaced-type.

depends on what you want to do. but since colorspace convertions are not codec-dependant, it's easy to workaround this, so it's not desperate IMHO.

>I see that you reinstall your Visual C++

i never uninstalled it, it's just hidden to avoid tentation (asm virus ^^)

>It's good to see you back

i'm already gone again ^^

>I'm sure you make a great job with this codec but I don't have a
>computer to test it......

well, it's just simple code :

- median predict the image (isse), pack signed/unsigned into intensity (mmx), get the length by level (mmx), pack the bitstream (length+val) (mmx), and that's all. i tried a bit more advanced stuff, but it was too slow or to unefficient (less than -5%)

>I will be back in april, it's hard to wait !

lol ^^. btw, i re-encoded all the slayers try you gave me in xvid q6-9 asp + ogg for 3 cds (instead of the 7 original CDs), and it still looks good !

bye all,
Marc

[^^-+I4004+-^^]

on win98 can't get proper image (only different color
blocks.no real image.....) with YV12 or YUY2 as input..

on w2k crashed VDub, couldn't recover it in that
session........(used btwincap+YV12 as input...)

very poor documantation too.....(heh)

[Guest]

Originally posted by Marc FD
er, doing a codec wasn't too hard, but my knowledge of winapi is _so_ bad than i don't even no how to draw a single dialog box. and to be honest, i simply don't want to know ^^. Please release the source code and I'll add the dialog handling for you and then give it back.

my advice : use avisynth 2.5 beta, and you would be able to do interlaced downsampling then (i.e. with the stuff i gave to you) and to give yv12 to the codec, it's gonna be (almost) as fast.The stuff you gave me is compiled with AvisynthPluginInit(), making it unusable. And, of course, you didn't give me the source code. :) Finally, you told me not to redistribute it, so others cannot benefit.

i'm not sure, but try to activate interlacing in XviD. you'll be surprised, it's doing much more than only field-based DCT, if i rememeber well. (even if i never tested it ^^)I didn't see any interlaced upsampling code in XviD, but I'll test it just for fun.

Thank you for your response.

[^^-+I4004+-^^]

oups graft,you forgot he's 17?
he has yet to find where his heart is.......


but sure he should give source......(hehe)

[Kurosu]

@Neuron2
Just to clear *one* misunderstanding, although other matters are none of my business:


>what about near-lossless compression, would that be hard to accomplish?
mhh. not really. dunno how huffyuv works. but i think it's useless.

He only stated that doing near-lossless compression (for instance dropping one bit due to noise if it's usefull) would not produce a result worth the implementation for it.

Next point:

The stuff you gave me is compiled with AvisynthPluginInit()

If really compiled for AVS2.5, and if AvisynthPluginInit is the only difference, you can do that ugly hack: replace "AvisynthPluginInit@" ASCII string by "AvisynthPluginInit2@" in the binary (hexadecimal edit). Yes, it _is_ ugly, may not work, but I've just tested it as working right now on several plugins (both loading and processing).

On a side note, he has spent quite a lot of time on coding this last year, and he has some serious exams soon to pass, I guess. Maybe he's hadnling the whole matter a bit lightly, but in the end, wouldn't it produce more good than bad?

Last point, about the codec:
The decoding under W2K produced a totally garbled video, like would do the use of an uncorrect pitch. The decoding in Vdub(mod) was fine though. (Maybe software specific).
Test sequence: a 512x384 xvid encoding of a much cleaned and flat source (anime DVD).

|HUFYUV| VBLE
-----------+------
enc.| 65 | 55
fps | |
----+------+------
play| |
fps | 250 | 160
----+------+------
size| 64.2M| 96.9
----+------+------

I don't know where the size gain comes from, but it sure is impressive, although the material (not a TV cap) helps a lot.

[Guest]

OK, forgive my impatience. I'll edit my post. I hope you like it better.

[Guest]

@Kurosu

>If really compiled for AVS2.5, and if AvisynthPluginInit is the only
>difference, you can do that ugly hack: replace "AvisynthPluginInit@"
>ASCII string by "AvisynthPluginInit2@" in the binary (hexadecimal
>edit).

That worked. Thank you for suggesting it.

@MarcFD

Do you have any objection to making your yuy2toyv12i() available to other people? If you give us the source for the codec and for this filter we can integrate them with an option, if you are too busy to do it. Thank you.

[Marc FD]

well, i switched from 56k to 512k this morning, so i'm not gonna touch anything related to dev before several weeks at least. sorry, too much counter-strike to play and to much leeching to do ^^.

but i'll get bored a day, and i'll finalize things who needs to be finalized. BTW, the interlaced upsampling i gave you is in fact just a little piece of a motion compensated denoiser i did some month ago, and i think it's better not to distribute it.

i'll add a config box with interlaced/progressive up/downsampling someday, i could add RGB32/24 & YUY2 native encoding too, and release the code, but now, i'd like to do something else.

thx for comprehension.

bye,
Marc

[c0de_v0id]

case closed... :(

[Zarxrax]

Hrmm, anyone know what happened to -h? It was weird cause he said he would try and have it finished in like a day, then hasn't posted again for 1-2 weeks. I hope nothing has happened to him :eek:
I've seen a lot of people just suddenly up and disappear off the face of the internet, and it never gets out of the back of my head that these people could be dead or something :scared:

[sungey]

yaa ... i like to know that too ...

btw i just tested VBLE in my anime encoding .... the normal size for my huffyuv avi for the show was 5-6 gigs and with vble i got 4.4 gig nice :) ..

after that... i began checking the colorspaces using avisynth
info() ... i just found out that xvid,divx5 and vble are YV12 ( of course they are :) ) but divx3 is rgb32 ? ... i thought mpeg4 are all using YV12 ... ... i tried comparing the encoding speed betweeen huffyuv -> divx3 and vble -> divx3 ... seems like theres no speed gain .... ( or maybe im blind ).

Seems like colorspaces conversion in my divx3 anime encoding is unavoidable :( ... not a big prob though .. i do xvid most of the time now .... just sharing my experience :)

p/s : the source of my anime are usually DivX5 or xvid most of the time .. if divx3 is really rgb32 ... theres a waste of bits there..

[Belgabor]

All MPEG4 codec use YV12 internally, that doesn't mean they accept it. And btw you need a YV12 enabled version of VDub to process without colorspace conversion, which nandub certainly isn't (except someone modded it while i wasn't looking :p)

Cheers
Belgabor

P.S. Hm, I still fail to see why ppl insist on keeping to use Divx3...

[c0de_v0id]

yeah im pretty sure the codec itself is able to use most of the colorspaces. It's just nandub that can't encode any other colorspaces than rgb. i saw a post saying its because of anti-shit.

Also i saw some posts in forum saying nandub can handle yv12(???) with fast recompress+avisynth 2.5 (because ffdshow reports the nandub encode as yuv12)(?????)

btw, i looked at all sbc encodes i have and they are all rgb. i also did some test encodes myself with fast recompress sources being yv12
(one without any conversion, one with converttoyuv2), but they all came out as rgb.

can someone clarify this?

[sungey]

yeah right .... probably its nandub that cant work with YV12 ...
so it converts to RGB32 ... and leave the encode in RGB32 since reconverting to yv12 will make it slower and lowers quality .... that's just my guess though ...

[Si]

I still fail to see why ppl insist on keeping to use Divx3...

Because its got the best quality vs compression factor vs stability factor codec around maybe?

regards
Simon

[Guest]

But no B frames.

[^^-+I4004+-^^]

Originally posted by sungey
yeah right .... probably its nandub that cant work with YV12 ...
so it converts to RGB32 ... and leave the encode in RGB32 since reconverting to yv12 will make it slower and lowers quality .... that's just my guess though ...

don't know bout yv12,but i can say this:
mjpeg avi->avs205->ND(fastrepack)->ok...
so it's YUY2 input converted directly to yv12...
(off course,fastrepack means avs must do all the processing,and no
antishit...)
but that's no big news....

i think there was some talk on forum on this issue of yv12->yv12
ND.......(or just previewing it?)

[scmccarthy]

@c0de_vi0dbtw, i looked at all sbc encodes i have and they are all rgb. All mpeg-4 codecs use YV12, including divx3.11alpha. Once you've encoded it to sbc, it is always the same format and always has been. AviSynth has nothing to do with that.

On the other hand, players (rather the codecs) convert everything to rgb for the display. So everything gets encoded to YV12 and decoded to RGB.

How did you look at the sbc encodes? Did you open them in a hex viewer and read the fourcc codes?

Stephen

[c0de_v0id]

@siwalters& neuron2
divx3 may not be the best but ppl(at least in fansubs) still use it to keep the leechers happy. especially with xvid instability and leechers' stupidity ( they cant even install ffdshow right)
I use xvid for all my personal encodes btw.

@scmccarthy

How did you look at the sbc encodes? Did you open them in a hex viewer and read the fourcc codes?

No. I used avisynth info(). they are all in rgb colorspace.(unless info() is reporting wrong colorspaces ^^;; ) and their 4ccs are div3 (yes in hex viewer :) ).

here is what i did,
Source = small clip(xvid, yv12)
1) ndub, default settings, 1 pass, avisynth2.5 script with just avisource(clip)
2) ndub, default settings, 1 pass, avisynth2.5 script with avisource(clip) & converttoyuy2()
3) vdubmod, XviD default settings, 1 pass, avisynth2.5 script with just avisource(clip)
4) vdubmod, XviD default settings, 1 pass, avisynth2.5 script with avisource(clip) & converttoyuy2()
:edit: all using fast repack

Colorspace results: ( i used a 2 lines avs script with avisource and info() to check the colorspaces)
1)RGB
2)RGB
3)yv12
4)yv12 (??)
so questions to u guys,
for 1&2 the ndub only operate in RGB?
for 4, how come the output is in yv12? (codec converted it?)

I dont think there is away to find out colorspaces with hex viewer. is there?

@sung
whatchu encoding now man, i miss the old days hehe

[scmccarthy]

@c0de_v0id

I will posit a thought: If you use converttoyuy2() with xvid, you are forcing xvid to convert the YUY2 to YV12 in the encode. It will get stored in the same color space regardless of what you feed it.

Stephen

[sungey]

Originally posted by ^^-+I4004+-^^
don't know bout yv12,but i can say this:
mjpeg avi->avs205->ND(fastrepack)->ok...
so it's YUY2 input converted directly to yv12...
(off course,fastrepack means avs must do all the processing,and no
antishit...)
but that's no big news....

i think there was some talk on forum on this issue of yv12->yv12
ND.......(or just previewing it?)

oh .. it seems like anti-shit causes the colorspace to be converted into RGB32 ... damn anti-shit is important for me though .....

oh well, i seldom use divx3 nowadays though ... just xvid :)

@ c0de_v0id
just the same series as usual .... where have u been lol ..

[digitize]

@sung
hey there, used to talking to you on irc =P
anyway, use ffvfw if you're forced to use divx3, it can encode in divx3 compatible streams. Gives much better results, there is a thread for it in the xvid forum, take a look.

[sungey]

yaa i tried ffvfw 3 months ago IIRC (not sure if theres a newer build) but unfortunately the divx3 encode made with it doesnt work with Divx5, which like 80% of the leechers has installed on their pc.

Divx5 is prioritized higher than divx3 in Directshow (We can change that). Telling leechers to use Registry editor is too complicated and making a Regfile, sometimes work .. sometimes it doesnt. We can use ffdshow. Leechers only has DX50, DivX3 codec ... and Xvid (if they dont get playback jerkiness).. also ffdshow ( not many has this, they say "i cant install this blablabla", wth :( ) ..... anyway ffmpeg is cool ... :)

[digitize]

@sung
quick and easy solution to that, make the fourcc mpeg4 v3 or what not. It'll still be a divx encode basically, just with a different fourcc, and it will not get decoded by divx5, and most if not just about nearly all leechers/people in general should be able to decode mpeg4 v3.

[sungey]

yaa i actually did that once ..the encode was made publicly available and .. surprisingly there are people who
doesnt has mpg4-v3 installed on their pc ... no kidding :(
i had to deal with numerous priv msg on irc saying "waht codec is this? " .. oh man .... that was horrible ... :(

btw... divx3 encoding using ffvfw isnt 100% pixel-flip free .... :( (subtitle-block, whatever u call it)

[digitize]

It's probably better than using nandub... =P

Ermmmm, edit: it is better than using nandub

[sungey]

hell yeah ... :)

[aranaxon]

I feel bad for you fansubbers and the hordes of leechers in your chans :)

In any case, thanks for the codec Marc FD, its quite useful in terms of saving when i have heavy duty filters going.

[digitize]

Originally posted by aranaxon
I feel bad for you fansubbers and the hordes of leechers in your chans :)

and what exactly do you mean by that.....

[ohliuv]

Originally posted by -h
... and I want to release both this and the MSMPEG4V3->MPEG4 convertor ...

anything more for the MSMPEG4V3->MPEG4 ? anyone?

[sh0dan]

@ohliuv: Please stay on topic!

@others: I'm sure -h will release it when it's done! You are not doing anything but annoy him, by constantly demanding him to release it. If you want it so bad, pick up a C-compiler and do the modifications yourselves (as Marc did!)

[aranaxon]

Originally posted by digitize
and what exactly do you mean by that.....

It seems like a relatively thankless job *shrug*

but hey, its a good service for the masses, and its what you like to do.

[litz]

Tried marc's VBLE codec on a Nvidia GF200 capture input, get 1 frame, 1 dropped frame, repeated for the length of the capture.

Almost like it's capturing at 15fps and duping frames. :confused:

Any ideas?

Would like to be able to capture directly to YV12, but with Marc's codec the only one out so far (-h ?) I haven't had much luck so far.

thankx ...

- litz

[mf]

Any idea why this (http://lsoust.hypermart.net/artifact.png) would happen ? This is 1 of 3 setbacks to a super-high-definition version of The Matrix Reloaded trailer and I think I'm gonna give up. AVISynth access violates (hung system), I'm having problem with splitting/appending my broken encodes, and then there's this. These are all great tools but they really don't seem high-definition resolution ready. No offense to anyone.

[sh0dan]

@mf: Could tou describe your problems more in detail - someone might even be able to help you out then. Keeping them to yourself never solve any problems.

[mf]

Originally posted by sh0dan
@mf: Could tou describe your problems more in detail - someone might even be able to help you out then. Keeping them to yourself never solve any problems.
Well, this is in the VBLE thread, so I'm compressing to VBLE. As you can see from the image I posted, the frame size is 1600x864. That's about all the relevant info. The irrelevant info: this is a transcode of the Matrix Reloaded Trailer at 1000x540 from Quicktime to VBLE using TMPGEnc, processed in AVISynth (neither of these things mentioned show this problem), and then encoded to VBLE again. All processing (except maybe the TMPGEnc conversion) is done in YV12. The problem seems to come from the extreme resolution. I don't see a good workaround for it, since AVISynth filtering is already down to 1 frame every 5 seconds.
Oh, and some relevant info I just remembered: if I re-encode the part with the artifact seperately I don't get it, but while appending I run into the VDub problem, which I'm talking to Suiryc now about. Nuff detail now ? :)

Edit: I found the split/append problem, I was splitting at the wrong frame but couldn't see it because Vdub wasn't showing me frame 0. I will now try to fix all the artifacts by means of splitting/appending until I have a clean stream.

[Marc FD]

maybe a error in the codec, but i don't see at all why it would lead to that.

i good idea might be to use smaller and smaller resolutions to see when the bug occurs (how big the resolutions has to be - if it's resolution related)

hope it'ld help.

Marc

[mf]

Originally posted by Marc FD
maybe a error in the codec, but i don't see at all why it would lead to that.

i good idea might be to use smaller and smaller resolutions to see when the bug occurs (how big the resolutions has to be - if it's resolution related)

hope it'ld help.

Marc
Thanks for being around even though you're busy leeching anime :p. I think the problem is memory related, I think one of the 3 (VBLE, AVISynth(and/or plugins), or VdubMod) is having memory leaks or corruptions or whatever (dunno too much about that stuff) causing a total system hang after some time. I really don't know, the errors seem to appear randomly. I can't recreate the problem at a certain frame either. I'm now filtering again (I managed to append everything together properly), so we'll see what results I'll get this time.

[cweb]

Marc,

Tried your codec and it's the best lossless capture codec so far.
I get no dropped frames with it. With the others I always got a small amount (nothing to worry about), but with yours, no drops!

[Valky]

Well, I haven't never captured with Huffy cause it takes too much space from my opinion. But in a little test-sample (3min43sec) when I tried with already encoded file, this Marc's codec gave me 383mb file comparing to huffy's 564mb file so if this is really lossless codec then I am very happy if I can replace my good old huffy with this one.

It saves lot of space,cause I always edit my captured files in VD when cutting out commercials and then save this edited file to one clean avi-file with de-noised audio file.

Thank you for that.

[N_F]

Originally posted by Zarxrax
Hrmm, anyone know what happened to -h? It was weird cause he said he would try and have it finished in like a day, then hasn't posted again for 1-2 weeks. I hope nothing has happened to him :eek:
I've seen a lot of people just suddenly up and disappear off the face of the internet, and it never gets out of the back of my head that these people could be dead or something :scared:
Make that ~11 weeks :(