Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 5th April 2003, 22:25   #1  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
New experimental Lossless codec -- needs your help

Quote:
Newest version is 004, download below
Note to moderators: I'm posting this here because the code is in need of your help. Although the current code works, it's not ready for wide release. If you feel this thread should be moved, please do so.

I've spent a bunch of time writing a new codec. It's based loosely on the LOCO-I algorithm.

Summary:
- Lossless file size if about 70% of Huffyuv
- lossy can decrease file size another 10-20% percent
- about 1/4 the speed of huffyuv :-<
- Stores YV12 natively
- GPL: (C) 2003 Mohammad Rezaei

TODO:
- Performance.
-- calcLoco can be vectorized. Part of comp_one_pel can be vectorized.
-- riceEncode can not be easily vectorized.
-- unlikely to be made more than 50% faster
- YV12:
-- conversion to YUY2 and RGB after decompression
- interlacing
-- is there a flag indicating interlacing in the incoming images? if so, store it.
-- a quick test showed that unless most of the video is highly combed, it does not make sense to compress fields separetely.

Essentially, I've hit a wall with the code as far as performance goes. You guys are the closest thing I can find to video related performance experts. I need your help in tuning this codec so it can be more useful. I'm not sure how much more time I can spend on this, so if someone wants to take this over, that's fine too.

The project was inspired by the YV12 codec thread in this forum. Feel free to test the codec, but please don't store anything important using this codec: the internals may change.

The code is based on huffyuv and is GPL, so please don't release binary only versions.

Your help is much appreciated.

Edit note: removed loco001.zip, added loco002.zip (small bug fix) down below.
__________________
Thanks
Moh

Last edited by TheRealMoh; 29th April 2003 at 02:06.
TheRealMoh is offline   Reply With Quote
Old 5th April 2003, 23:49   #2  |  Link
Dali Lama
Registered User
 
Join Date: Jan 2002
Posts: 331
Hi Moh,

Edit: I guess you know about Marc FD's VBLE codec. Perhaps you can try to PM him or "-h"?

Take Care,

Dali
Dali Lama is offline   Reply With Quote
Old 6th April 2003, 04:37   #3  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
Here is a small bug fix.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Old 6th April 2003, 05:35   #4  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,922
Quote:
Originally posted by TheRealMoh
Here is a small bug fix.
Please describe the bug when posting bug fixes.
Guest is offline   Reply With Quote
Old 6th April 2003, 05:53   #5  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Your readme indicates this is installable without a compile, however you did not include the DLL. Was this your intention?
DDogg is offline   Reply With Quote
Old 6th April 2003, 06:32   #6  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
@neuron2:
sorry. Changes to 002:
- GUI was not accepting "0" for max loss.
- When in lossless mode, it's slightly faster.

@DDogg:

Here is a binary release of 002.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Old 6th April 2003, 09:23   #7  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
(Copy/Paste from an earlier posts I made regarding ffvfw)

Regarding colorspaces, it would of course be a good idea to support all output formats. Any input format would also be optimal. XviD provides most conversion routines.

YV12 -> YUY2 - use AviSynth routines (if possible). These routines have better chroma upsampling, and optional interlaced handling.
There _must_ be an option to select interlaced conversion. Sort of: Use interlaced upsampling, if more than 'n' lines, would be best.
There would have to be special adaption if width is not mod-16, since AviSynth has larger pitch in these cases. I would be glad to help out here.

YV12 -> RGB - use XviD conversion routines. Unfortunately they are not qualitywise optimal, but they are fast and has support down to mod-8 width.
Interlaced upsampling is also a problem here.
A solution could be to use AviSynth YV12 -> YUY2 and also use AviSynth YUY2 -> RGB. This will be slower, but on the other hand it would be possible to properly support interlacing and have optimal chroma upsampling.

It is also possible to modify the existing AviSynth assembler RGB24->YUY2 to support 24bits.

So to conclude, you can find most conversion routines needed in AviSynth - the only problem is interlaced YV12 -> RGB.

Link to the latest convert_yv12.cpp.
There are MMX and ISSE YUY2 -> YV12 (interlaced and noninterlaced) and YUY2 -> YV12 (interlaced/non-interlaced).


Lastest XviD conversion routines - adapted for MASM, if they are of any help. No interlaced handling, and sub-optimal chroma interpolation.

The best way for lococodec to handle this would probably to be able to pass through material, if possible, or use the simplest possible conversion - this does include much work though - but AviSynth has assembler routines for most of these.

|YUY2 -> RGB32| - |MMX RGB32 -> RGB24| - |MMX YUY2 -> RGB24/34|


IMO any VFW codec should be able to output in (interlaced/progresive) YV12, YUY2 and RGB24/32.
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 8th April 2003, 16:18   #8  |  Link
ominte
In training.
 
Join Date: Jan 2003
Location: Australia
Posts: 39
pardon my ignoranance TheRealMoh but i can't view any material encoded with loco002. Any help would be appreciated .

Otherwise nice job on the compression my tests show loco002 lossless reduces file size by 8% when compared to VBLE. Also I am getting encoding speeds of around 20 fps on an AMD +1600, nice.
ominte is offline   Reply With Quote
Old 8th April 2003, 19:05   #9  |  Link
terwin
Registered User
 
Join Date: Oct 2001
Location: Germany
Posts: 14
The same problem here, I can't decode the Loco-movies. But the encoder works fine for me.

cu terwin
terwin is offline   Reply With Quote
Old 9th April 2003, 05:04   #10  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
@terwin & ominte

How you view the output depends on the color space (right now). If you compress YUY2 or RGB, you should be able to play it (but you need a very fast computer). Better yet, you should open it in virtualdub.

If you compress YV12, the only way to view the video is to create a small avs script:

video=avisource("C:\temp\yv12test.avi")
video.ConvertToYUY2()

You can then load the avs as normal.

The automatic conversion of YV12 is on the todo list, but I'm concentrating on performance right now.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Old 9th April 2003, 07:57   #11  |  Link
terwin
Registered User
 
Join Date: Oct 2001
Location: Germany
Posts: 14
Ok, thanks for the quick answer. The problem was YV12.

cu terwin
terwin is offline   Reply With Quote
Old 9th April 2003, 10:22   #12  |  Link
ominte
In training.
 
Join Date: Jan 2003
Location: Australia
Posts: 39
Thanks for your help, everything is working fine now . Any planned support for YV12 in the future?
ominte is offline   Reply With Quote
Old 9th April 2003, 11:34   #13  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
@TheRealMoh: YV12 support is quite important, as all YV12 <-> YUY2 is lossy. Also YV12 should give additional compression, due to the chroma information.
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 10th April 2003, 00:09   #14  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
YV12

@omnite

I'll code YV12 -> YUY2 conversion during decoding right after I'm done with performance stuff. I'm leaving on vacation soon for about 10 days, so probably 3 weeks.

@sh0dan

I'm not sure YUY2->YV12 is necessary. This is a lossless codec afterall. It would be nice, but I don't think anyone would use it. The way I see it, a lossless codec has two uses:

1) Capture. This requires real time encoding. That's why I'm concentrating on performance right now. I'm not sure conversion to YV12 would slow things down or speed things up. All captures cards I know are YUY2 natively.
2) Intermediate step where quality is important. In this situation, I doubt people would choose to lose 1/2 of UV planes.

Again, I'm not saying it's a bad idea, just not one high on the priority list.

Maybe you know the answer to this question: is there anything in the avi api's that deals with interlacing? I need to know if the source is interlaced so I can do the YUY2->YV12 conversion correctly.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Old 10th April 2003, 00:26   #15  |  Link
ErMaC
Lurker in Training
 
ErMaC's Avatar
 
Join Date: Apr 2002
Location: Halfway Between The Gutter And The Stars
Posts: 160
Unfortunately there is nothing in the AVI standard to flag interlaced or non interlaced. The way Huffyuv deals with this is it assumes anything over a certain verical resolution is interlaced (288 vertical resolution by default). Then what it does is splits the frame into the two fields and encodes them separately for supposedly better compression.
Doing this in YV12 colorspace causes a problem because the chroma sampling varies between interlaced and progressive footage. The only way you could properly handle this, in my opinion, is to add a checkbox in your codec's configuration screen for interlaced encoding.
ErMaC is offline   Reply With Quote
Old 11th April 2003, 18:29   #16  |  Link
MrTibs
Registered User
 
Join Date: Jan 2003
Posts: 90
Personally, I appreciate the work. I'd like to get an addional 30% compression over Huffy during capture if possible. With the CPU's increasing at present rate, we could even use "slow" codecs for live capture. For now, however, sometimes I run my Huffy captures through Avisysnth with filtering and resizing and save back into huffy. This technique allows me to reduce my Huffy files by 80-90% before I convert to Mpeg1 (KVCD). I do this for three reasons:

1) Lossy (Mpeg] conversion requires much fiddling to get it perfect.
2) Lossy encoding is MUCH faster since Avisynth filters are not invovled.
3) Filtering through Avisynth is much faster then Mpeg conversion so I can make room on my HD for another TV capture very quickly.
MrTibs is offline   Reply With Quote
Old 11th April 2003, 23:29   #17  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
loco003

I spent some time optimizing (vectorizing) the code for P4's. This requires a P4, so there are now two binaries P4 and non-P4.

It seems the vector operations on a P4 are not alot faster than the interger ones (probably because of the dual, dual-speed ALUs). The P4 code is about 10% faster now.

I'm going on a vacation for about 10 days, so this the last release for a while. Next on the todo list is the color space conversions.

The attachment has the source plus 2 binaries.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Old 12th April 2003, 10:10   #18  |  Link
mf
 
mf's Avatar
 
Join Date: Jan 2002
Posts: 1,729
Re: YV12

Quote:
Originally posted by TheRealMoh
2) Intermediate step where quality is important. In this situation, I doubt people would choose to lose 1/2 of UV planes.
That intermediate step is most likely to be converted to MPEG1/2/4 or other YV12 codecs. So why not convert to YV12 directly (or, in the case of an MPEG source and AVS 2.5 processing, don't convert it at all) and save the space.
mf is offline   Reply With Quote
Old 22nd April 2003, 16:26   #19  |  Link
cweb
Registered User
 
cweb's Avatar
 
Join Date: Oct 2002
Location: The Pandorica
Posts: 527
Perhaps updating it to use YV12 would also speed it up.

I experimented with both Marc's new yv12 huffy codec and with yours.
Marc's seems faster and has no frame drops, unlike yours.

I found your codec interesting, I hope you will continue developing it.
__________________
PC specs for bug reports: Athlon 64-bit Phenom II X6 2.8Ghz Win7/Linux PCI express NVIDIA GTX 285 graphics card
http://twitter.com/cwebdesign
cweb is offline   Reply With Quote
Old 27th April 2003, 23:35   #20  |  Link
TheRealMoh
Registered User
 
Join Date: Mar 2003
Posts: 19
loco004

I'm back from vacation and here is a new version:

Changes:

- Lossy compression is improved (compression wise).
- YV12->YUY2->RGB is implemented during decompression.
- replaced the field threshold with a simple check box: check it if the source is interlaced. The setting is only used during compression.

If you have a YV12 interlaced avi that was encoded prior to this release, it won't be properly decoded to YUY2/RGB. You can re-encode the clip with the "source interlaced" checkbox checked.

@cweb & mf
Thanks for the comments. I still haven't decided if/how I'll implement YUY2->YV12 during compression.
__________________
Thanks
Moh
TheRealMoh is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.