WARP codec source code available.

[General Lee D. Mented]

I've decided I can't quite get this thing to work on my own. I've been stumped on some bugs for a long time now and have decided to ask for whatever help I can get. It's about 98% done, but not working yet.

The project page is at http://corecodec.org/projects/warp/ and if you look in the CVS you should find the current source for it. I'm told anonymous CVS downloads are working now.

So, anyone who wants to please feel free to try and help me track some of the bugs down. Last I remember (I got upset and didn't work on it for awhile) I was getting all sorts of weird errors with new and malloc in files I couldn't trace like kernel32. I was testing with AVS2AVI 1.13, which makes a nice little test app because I can compile/debug it and it's pretty small in number of lines.

I'll be happy to answer any design questions or provide comments for anything I forgot. Just please ask somewhere public like here or the project's forums, not by private emails. I don't want to explain the same things over and over to everyone.

[jcsston]

Originally posted by General Lee D. Mented
Last I remember (I got upset and didn't work on it for awhile) I was getting all sorts of weird errors with new and malloc in files I couldn't trace like kernel32.
You can download the symbol files for Windows
at http://www.microsoft.com/whdc/ddk/debugging/symbolpkg.mspx that will likely help in debugging.

[General Lee D. Mented]

Well that should definately help with debugging, thanks. =)

[Sirber]

Can you explain a little what WARP codec os good at ant it's specs? Thanks!

[General Lee D. Mented]

Originally posted by Sirber
Can you explain a little what WARP codec os good at ant it's specs? Thanks!

It would be better with specific questions but I'll try.

WARP is Wavelet Amplitude Reduction Processing. It's a new codec I designed based on biological systems like how the eye and brain work together and some concepts I thought up.

It's a supertemporal (think multi-bframe) lossy codec with a wavelet based transform, a fractal scan, a global heap data selector (instead of a simple quantizer), and a burrows-wheeler entropy coder for the final compress stage. I plan to get the last part from libbz2 but haven't added the library link to the current code yet. Even that may be temporary as I'm looking for a better BWT-arithmetic solution that's not as limited as bzip in terms of memory size. Help with that would be VERY appreciated.

Alot of time has been spent making the design as flexible as possible and reducing code size. Right now it's about 2000 lines of code and it should be easy to add support for pretty much any image format via the WARPframe struct defined in the common files. This means any resolution, colorspace, number of channels, bits per channel, framerate, aspect ratio, interlacing system, steroscopic system, etc can be supported. Holographic and higher order dimension modes are planned out but haven't been implemented yet since there's no source data right now and adding them is trivial anyway.

It's also been made extremely scalable and should be able to run on just about anything (all the math is pure integer right now), although I'm worried the bzip-like portion may slow things down too much for things like a palm. It'll probably have to be replaced with a less efficient system like lzh (zip) for that. Otherwise it should scale in memory use from about 200KB to 250PB(kilo->mega->giga->tera->peta = P), with efficiency improving as available memory and data size increases. I'm hoping to get compression competative with modern mpeg4 codecs in about a 200MB footprint on decode, and a reasonable multiplier of that on encode.

SMP support is written in a seperate test but not implmemented in the main code, to simplify things to get it to work normally first. Support for dual processor to several million cpus scaling with data size should not be a problem, other than OS limitations. Distributed processing has been thought out but may not be practical due to network speeds.

Multipass is planned for sometime later, probably after moving most of the main functions to a more flexible plugin format to customize for various hardware. Since the system inherantly does 2 phase VBR in one pass and 3 phase will likely not be difficult, multipass isn't a real high priority right now. It is on the list for the future though.

And umm... I'm running out of steam here, how about more specific questions?

[BetaBoy]

In other words... you need a 10ghz processor ;) ... GLDM ping me when you can. I wanna talk about WARP pls.

[General Lee D. Mented]

Originally posted by BetaBoy
In other words... you need a 10ghz processor ;) ... GLDM ping me when you can. I wanna talk about WARP pls.

No, you just need lots and LOTS of memory and memory bandwidth. It's not very cpu-bound at all. The transforms run about 7000fps on my Athlon XP 1800+, but that's without the entropy coder and the heap running doing alot of memory swapping. It's all about memory, and like I said it's scalable. :P

[phrentec]

any relation to warpenterprise?

also is your codec suitable for low bandwidths in the 56-350 kbit range? I can't find a binary of your codec and the pages still says "This Project Has Not Released Any Files"

regards.

[General Lee D. Mented]

Originally posted by phrentec
any relation to warpenterprise?

also is your codec suitable for low bandwidths in the 56-350 kbit range? I can't find a binary of your codec and the pages still says "This Project Has Not Released Any Files"

regards.

No relation, never heard of it. It should be good for low banwidth. There is no binary because if you read the first post you'll see that it doesn't work yet! Also hence why it's listed as alpha status, not beta.

[Hiro2k]

Originally posted by General Lee D. Mented
The transforms run about 7000fps on my Athlon XP 1800+, but that's without the entropy coder and the heap running doing alot of memory swapping. It's all about memory, and like I said it's scalable. :P

So how long does it take? I'm curious about this. Also how can you make tests if it's in alpha status. Also will your codec be opensourced GPL or closed source?

[General Lee D. Mented]

Originally posted by Hiro2k
So how long does it take? I'm curious about this. Also how can you make tests if it's in alpha status. Also will your codec be opensourced GPL or closed source?


I'm not sure how long it will take or what it will look like yet. I've tested the various pieces of it while building them to try and weed out bugs. Guess I didn't get all of them cause it's not working yet now that I've finally put most of it together.

You can find the license in the CVS at http://cvs.corecodec.org/cgi-bin/viewcvs.cgi/*checkout*/warp/warp/LICENSE.txt?rev=HEAD&content-type=text/plain directly. It's basicly QPL with a few clarifications I made.

[superdump]

This sounds like an inspirational codec... now if I only I knew how to code... *goes back to reading his book* another time maybe. :)

Good luck with the project general! *salutes* ;)

[ChristianHJW]

Hi GLDM,

good to see you put WARP code online. I am normally in Doom9 vacation, but your thread makes me post. Just please try to answer my following questions :

- Is it correct to say that the only real problem the existing WARP code has is the missing VfW/VCM interface ? We always knew it would be hard, if not impossible, to support your very specific codec type from VfW, right ?

- You had plans about a new, much improved codec API, and the basis of it sounded to be quite clean and straightforward, as well as easy and extensible. Any plans to start documenting your ideas, and maybe make some example code coming along with it ?

- With some luck, we could maybe try to win DaveEL to support your API from avs2avi or avs2mkv ( last one is better suited i guess, as its not VCM based ), and we could try to use WARP with the new API from avs2mkv ? In this case, it would be worth considering if its maybe better to modify XviD to support your API, instead of DaveEL allowing to link xvid.dll dynamically using Dev-api-4 ... just an idea.

What you think ?

[General Lee D. Mented]

Originally posted by ChristianHJW
Hi GLDM,

good to see you put WARP code online. I am normally in Doom9 vacation, but your thread makes me post. Just please try to answer my following questions :

- Is it correct to say that the only real problem the existing WARP code has is the missing VfW/VCM interface ? We always knew it would be hard, if not impossible, to support your very specific codec type from VfW, right ?

- You had plans about a new, much improved codec API, and the basis of it sounded to be quite clean and straightforward, as well as easy and extensible. Any plans to start documenting your ideas, and maybe make some example code coming along with it ?

- With some luck, we could maybe try to win DaveEL to support your API from avs2avi or avs2mkv ( last one is better suited i guess, as its not VCM based ), and we could try to use WARP with the new API from avs2mkv ? In this case, it would be worth considering if its maybe better to modify XviD to support your API, instead of DaveEL allowing to link xvid.dll dynamically using Dev-api-4 ... just an idea.

What you think ?

1. Well, first of all, VfW was hard but eventually I believe I did get it to work. I had the current interface that's on CVS working with a dummy core that had empty functions. It seemed to do all the init and config just fine and even pass frame data into the core. I don't know why when I tested the actual core it blew up. I'm not sure if the problem is in the core's code, which I tested in parts pretty extensively, or something in the interface that just doesn't show up with empty functions. So I don't know where the problem is yet. Spyder said he's gonna run some tests on it, and I'll probably take another crack at it after work if I'm not exhausted.

2. Yes I still have API plans, which were pretty much brushed off by everyone when UCI began discussing redoing things from scratch. Since there was no interest in my ideas, I assumed it would be impossible to get any programming help on them, and I really don't feel like starting an API from scratch to fix this. It'll take way too long to build the API and apps etc.

3. I'm all for DaveEL trying that, but I don't know much about xvid's API. My driverproc (the entry point for VfW) is directly based off a combination of xvid's and huffyuv's. If someone would like to try talking directly to the core's native API, then WARPcodec.h and the two files in the common dir, WARPconfig.cpp and WARPframe.cpp should be all you need. They're fully commented and should be straightforward enough to see how to use them to link to any API you want. One issue may be handling things like the extended config info and dummy frames via a modified xvid, I don't know how much work it'd take to hack that stuff in.

I'll be back after work in about 9 hours, so there'll be a little delay in replying to the thread today. :P

[spyder]

Gldm, I may not have time today to check it but I definitely will look at it tomorrow. Let me know if you find anything out.

[spyder]

I feel like I am just lying to everyone lately...

My parent's have me doing too much today. I will be here all day tomorrow, at least until late afternoon and will have nothing to do :) So I will work on making a small test for your code tomorrow morning. But I also have that MPEG2 frame splitter to work on. I can't make any promises...

Spyder

[BlackSun]

Originally posted by spyder
I feel like I am just lying to everyone lately...

My parent's have me doing too much today. I will be here all day tomorrow, at least until late afternoon and will have nothing to do :) So I will work on making a small test for your code tomorrow morning. But I also have that MPEG2 frame splitter to work on. I can't make any promises...

Spyder

We all know you can't stop watching Shakira :o

[spyder]

LMAO. I have made good progress on the MPEG2 video frame reader today though :P

[General Lee D. Mented]

Worked on it some today thanks to Spyder, and found some bugs. Right now we're stuck with a problem involving the delete operator. Could really use someone who knows C++ better than me (which wouldn't be hard) to help out about now. I have no idea what's causing the problem, the syntax is correct and everything should be fine but it's not.

[Nic]

Point me to the part thats causing the problem and ill have a look. :)
(ive grabbed the CVS source so just say the .cpp file and the rough line number)

-Nic

[spyder]

line 186 of chunk.cpp ;)

I don't knwo why the hell this fails :/

Spyder

[Nic]

186 is the close bracket of the Chunk::Sort function...do you mean the
delete [] pData; above it?

[superdump]

Nic: Yes, he does. :)

[spyder]

I forgot I added some code there to make sure which line was failing and to see what was going on in the code. I think Gldm put the other changes into CVS.

[Acaila]

Just some offtopic code questions, I hope you don't mind :):

I noticed this line of code right above where you guys are talking about where the error is:
qsort(pWavelets, datasize, sizeof(wavelet), Chunk::Compare);I've never seen a construction like Chunk::Compare used as a parameter before, what does it do exactly? To me it looks a bit like using the result of calling Compare() as a parameter to qsort, but since Compare requires two parameters itself it has me confused.
Also, where is the function qsort defined? I can't seem to find it anywhere in any file. Is it taken from a common library?

[jcsston]

Originally posted by Acaila
Just some offtopic code questions, I hope you don't mind :):

I noticed this line of code right above where you guys are talking about where the error is:
qsort(pWavelets, datasize, sizeof(wavelet), Chunk::Compare);I've never seen a construction like Chunk::Compare used as a parameter before, what does it do exactly? To me it looks a bit like using the result of calling Compare() as a parameter to qsort, but since Compare requires two parameters itself it has me confused.
Also, where is the function qsort defined? I can't seem to find it anywhere in any file. Is it taken from a common library?
Passing a Function to a function without the () will pass a pointer to the function to the function. ;)

Prehaps some code could better explain it.

void RunMe(const char *msg) {
printf(msg);
};

void Test(const char *msgToPass, void (*FunctionToRun)(const char *) ) {
FunctionToRun(msgToPass);
};

int main(int argc, char* argv[]) {
Test("test with printf", RunMe);
return 0;
};

qsort is part of the Standard Run-Time Library in VC++, GCC also has it.

[Nic]

I've only had 5 mins to look at this morning, but its a pretty obvious crash, but thats easy to see from fresh eyes ;)

Whats happening is you have a Chunk which inside it has a pointer to pData.

In Compress you have the function:
mytransform->Transform(pChunklist[i]);
Transform is defined as Transform(Chunk c)
This means pChunklist[i] gets copied to c ;)
(I can here you thinking, as the penny drops ;) )

ok...when Transform is finished it calls c's deconstructor and that deconstructor deletes c's pData memory....which is exactly the same as pChunklist[i]->pData (because the pointer addresse's were copied)

and this is why it then crashes in the sort when you try to delete the same bit of memory again. Your best bet I guess is to make Transform use a reference ? (i.e. Transform(Chunk &c) instead...but I haven't tried it)

Got to get back to work, behind on a deadline. Hope that helps :)

Cheers,
-Nic

[spyder]

Thanks. I am new to the whole C++ thing :)

[Nic]

You'll soon get the hang of it, shout me if you need any more help :)

[spyder]

There is a lot of work to be done on this code...

I have adjusted some tests conditions in if's etc to make the code not hang in infinite loops. But it seems what I was thinking was caused by infinite loops is actually the code it self taking so long to sort the chunks. The next problem is in Selector::Fixup. I am going to try without setting bitrate.

[spyder]

Ok, if you set bitrate to 0 it by passes the code that fails.

The next problem is that i get a null pointer of pData at line 126 of chunk.cpp. This is in the pack stage so...that means we are almost out of the process :)

Spyder

[General Lee D. Mented]

You guys are getting ahead of me, I'll have to try and catch up today.

Nic, thanks for pointing out that Transform() nees to be passed a reference to not destroy the chunk it gets passed, I totally missed that.

Also if anyone's on the old CVS there were some big changes such as I totally screwed up the header file for the codec. The VFW interface had a bad copy and it was completely messing with things. Jcsston and Cyrius pointed it out and I uploaded it a couple days ago. If you still have a WARPcodec.h in your vfw directory, delete it since it's been removed from CVS. The vfw project should link to the one in the core directory now.

Fixup() is going to be fun. I basicly had to try and figure out how that worked from a book. If you want I could probably scan the couple pages of it but it was a pain in the ass trying to read their code and figure out what it was doing.

Also people mentioned alot of deletes don't check for null pointers, I'm gonna go through the whole project looking for them today, so expect CVS changes tonight.

Spyder, I'd like to catch up with what you've fixed before comitting to CVS so if you're around later look for me on #matroska.

[Acaila]

@jcsston:

Thank you :)

[General Lee D. Mented]

Ok, decision time.

Based on Spyder's progress through the encode pipeline, things are going well on the debugging front.

However, he's said the sort stage is REALLY slow. I was expecting this cause sorting a couple hundred megs gets tedious.

One plan I had been toying with here was replacing Qsort() with a heapsort.

So, Chunk::Sort() would perform a heapsort that heaps pWavelets, so that index 0 is the most significant element. Then, when Chunk::Select() is called, the threshhold pointer moves past it, and the remainder of the array (now minus the first element) is re-heaped with a fixdown.

What this would do is replace an average NlogN sort with a logN heap at the start, and a fixdown that is performed K times, where K is determined by how many wavelets selector selects. Also the fixdown is roughly logN as well. So, it's NlogN base, vs logN + KlogN. Note these numbers are PER CHUNK. Not all chunks get the same number of values selected. And the average chunk will likely not have N wavelets selected, so this is always going to be faster. Potentially much faster because half the sort time is deferred until after selection, so it should be much much faster considering we don't select all the wavelets except in uncompressed mode, which should bypass this entire process as sorting and selection are unnecessary for that.

The question is, do you want me to work on this now, and let spyder do all the debugging, or should I do more debugging? Cause this is gonna take at least a couple days I'm betting.

[Nic]

BTW: In ::Compare it's going to be likely at times that test is going to loop all the way round the DWORD so that it equals 0 (and result won't != 0). This means a x % test will cause a divide by zero.

[General Lee D. Mented]

Originally posted by Nic
BTW: In ::Compare it's going to be likely at times that test is going to loop all the way round the DWORD so that it equals 0 (and result won't != 0). This means a x % test will cause a divide by zero.

Hmm, you're right, though it wouldn't be that common, it could still happen...

I'm trying to do a bit by bit compare of the index values of the two wavelets in Compare(). Any idea what test condition I should use to break the loop?

Basicly if the values of two wavelets are equal, this loop does a runoff based on index position. In the transform, data on a higher power of 2 index affects more values on decode, so has priority over data in another position. Thus the highest priorty is index 0. The next would be index datasize/2, then the two datasize/4, etc.

So, the loop divides each index by 2, and compares. An evenly divisible index is more valuable than one with a remainder 1, hence the %test. If they're both divisible by 2, test gets shifted so it divides by 4, and so on, until either one index is not evenly divisible by a power of 2, or we're out of bits in the number to test. Note it IS possible to have two identical index values because Compare() compares to other chunks' pData.

So umm... the problem is my test for "we're out of bits" doesn't work. A suggestion would help, I'm too tired from work right now to figure out how to do it.

[spyder]

No debugging for me for the next few days. I have my hands full with MPV2MKV. Maybe in a few days...

[General Lee D. Mented]

Originally posted by General Lee D. Mented
So umm... the problem is my test for "we're out of bits" doesn't work. A suggestion would help, I'm too tired from work right now to figure out how to do it.

Some sleep and talking to a friend has produced an answer I like. If you've tested all the bits and it fails the test condition on the last bit and goes to loop around, the two values must be equal. Therefore just add a if(a != b) condition before even bothering with the loop. I'll put it in later today. Gonna try and get some debugging done today and tomorrow.

[spyder]

Good, I would help but my grandfather is in the hospital and I can't keep the phone tied up very long. I probably won't even be back today. Hopefully you can fix some of the problems. I will be working on mpv2mkv while I am offline ;)

[General Lee D. Mented]

Problems have been fixed. The current CVS runs without crashing using the tester code spyder sent me.

So it's doing something. Probably not the right thing, but something. There were quite a few smack myself on the forehead bugs, the big one being mypacker->Pack() was being called inside the for loop that made the Chunks do Repopulate(). That was bad. It should be called after, and only once. Packer also wasn't storing the size results correctly, Scanmatrix had loop errors, etc.

I'm really burnt out now, and hungry. I'll try rebuilding the current code into the DLL and linking to VFW tomorrow to see if that works. If not, I'll either tinker with VFW stuff or try and finish up and comment the packer, I notcied I never got around to that.

P.S. Spyder, unless you wanna wait awhile for test runs, I'd turn down your fake frame resolution to something like 32x16 if I were you. :P

[spyder]

LOL. I will change that too ;) Glad to hear it's somewhat working ;)

[General Lee D. Mented]

Originally posted by spyder
LOL. I will change that too ;) Glad to hear it's somewhat working ;)


I tried setting bitrate to something other than 0 to see what's going on with selector. That's a whole mess. It's like trying to plug up a sieve. However, I think most of the problems are related to the fact that it doesn't know when to stop. I found chunks with thresholds being set way past the end of their data arrays, causing illegal access errors, and that weird block type error you had before. pWavelets[6723] when datasize = 4096 is bad. As soon as it got to Repopulate() it'd crash.

I need to figure out the best way to get selection to stop when it runs out of wavelets but still has bitpool available. Gonna think on that for awhile. I managed to do it with alot of kludges all over the project but that was just a stopgap. I won't even commit that version to CVS because going back and removing them all will take forever.

Oh and I think the long sort times are because the transform produces so many zeros that it's throwing quicksort into worst-case mode, which is very slow. Definately need another sort.

[General Lee D. Mented]

CVS updated with lots of fixes. Selector seems to be behaving now, no more out of bounds array access. I also added a function to Chunk so selection can test to see if there's anymore data worth grabbing or if it's just zeros, so that may speed things up a little.

I'm gonna put the sort replacement on hold for now, there's not much point until output is visible anyway. I'd rather have a sort I know works slow than one I'm not sure of that works fast, one less thing to debug once data is coming through VFW.

I think VFW needs its config functions overhauled. People are telling me I should append my config data to the BITMAPINFOHEADER instead of sending it back in IC_GETSTATE as a seperate struct. I'm having a hard time figuring out how to do the former with memcopy, cause it makes me feel very uneasy manually throwing bytes around like that.

[General Lee D. Mented]

I messed with VFW a bit.

For some reason it seems that when an AVI app calls ICCompressorChoose(), which is a MS function defined in Vfw.h, that calls my DriverProc with DRV_OPEN 3 times int a row. The first two times, it seems ok, but then the 3rd time when I try and create a new VFWhandler object, new gives me a null pointer and throws an exception that says something weird like "user breakpoint".

I checked MSDN for ICCompressorChoose but all the pages on it seem to be down, of course. :/

I could really use some help on this one, I'm stumped.

[[Toff]]

Arg, you are gone from IRC :D

It go up to ICM_COMPRESS here.
And it crash cause the configuration struct is (intentionaly) not filled it seems. To go in details it crash in WARPcodec::Compress cause timedepth is not initialized.

[General Lee D. Mented]

Originally posted by [Toff]
Arg, you are gone from IRC :D

It go up to ICM_COMPRESS here.
And it crash cause the configuration struct is (intentionaly) not filled it seems. To go in details it crash in WARPcodec::Compress cause timedepth is not initialized.

I didn't know you were actually working on it on IRC, I would have stayed. It seemed like everyone was busy or idle, as usual.

I keep getting this error before it even displays the codec selection box. If I skip past the first error I get the box and can pick the codec, but since the handler object is just a null pointer it won't work anyway.

[Tommy Carrot]

Any news?

[General Lee D. Mented]

Originally posted by Tommy Carrot
Any news?

Working on the decode pipeline now. I hope to get the core completed soon so Spyder and others can do another round of testing using that and test drivers.

The VFW interface is still being a pain so it's on hold for a moment, I need to rethink a couple functions there, concentrating on the core for now.

I could really use some kind of easy to read tutorial on Windows Dialogs and how to make one. Blacksun's WarpUI.dll is nice but I still wanna make my own eventually. That's even further down the list than VFW though.

[General Lee D. Mented]

Last night jcsston pointed out void WARPcodec::Compress(WARPframe frame, char* pResult, int &resultsize) should be void WARPcodec::Compress(WARPframe frame, char* &pResult, int &resultsize). I'll have to search for repeats of similar errors, as I'm probably doing it all over in other functions and it's probably causing alot of bugs.

[Joe Fenton]

Originally posted by General Lee D. Mented
Last night jcsston pointed out void WARPcodec::Compress(WARPframe frame, char* pResult, int &resultsize) should be void WARPcodec::Compress(WARPframe frame, char* &pResult, int &resultsize). I'll have to search for repeats of similar errors, as I'm probably doing it all over in other functions and it's probably causing alot of bugs.

Which is exactly why I prefer working in assembly langauge. ;)

It has it's own share of problems, but these stupid pass by reference vs. pass by value issues isn't one of them. You also don't get bitched at when you handle a pointer like an integer. If I want to add an offset to a pointer, I don't have to worry that that the offset will be scaled by the size of the pointer type. Blah!

And C++ is even worse! Creation and destruction operators? Sorry, but if I want to "create" an object, I'll allocate the memory and set it myself. I don't trust things like that to just anyone.

[MfA]

Reference variables are just a pretty form of obfuscation.

[General Lee D. Mented]

Originally posted by Joe Fenton
Which is exactly why I prefer working in assembly langauge. ;)

It has it's own share of problems, but these stupid pass by reference vs. pass by value issues isn't one of them. You also don't get bitched at when you handle a pointer like an integer. If I want to add an offset to a pointer, I don't have to worry that that the offset will be scaled by the size of the pointer type. Blah!

And C++ is even worse! Creation and destruction operators? Sorry, but if I want to "create" an object, I'll allocate the memory and set it myself. I don't trust things like that to just anyone.

Ok, you go and rewrite the entire thing in assembly. No really, feel free, the source is online.

I don't know assembly (well not x86 assembly at least) so I'm not going to be doing that myself. I'm not even comfortable using memcopy let alone doing my own constructors by hand.

I was going to write it in C# but people complained it would complicate porting it to other platforms, so I went with C++. I'll probably go and rewrite it later anyway, once I can just have all the critical sections in optimized plugins. Probably a few months after it's running.

Anyone wanna do the Directshow and Gstreamer interfaces for me? No? How bout Quicktime and a MacOS port of the core, any takers?

[General Lee D. Mented]

Originally posted by MfA
Reference variables are just a pretty form of obfuscation.

If you have a more elegant solution I'm definately listening.

Currently on the list of things to do is overhaul all the functions looking for errors and cleaning things up, like making Compress() take an input frame and an output frame, instead of a frame and a buffer, which is silly and not as clear.

Also, I'm probably going to rework all the config functions to be more standard, and all take WARPconfig parameters instead of the mishmash of passed variables there are now. It's too easy to forget to set the local value of one of the variables like numchunks or blocksize that everything constantly needs. I'm planning to pass them by value since it's a struct and I don't want to do "bad things" like allow access to what should be private data, i.e. the config variables for each object. Doing something like changing a global config struct everyone points to and forgetting to tell say, Selector to reallocate its heap would be quite bad and probably hard to debug since the values would look right.

Also I need to check to make sure dynamic arrays are deleting right (again) in places like the destructors or there'll be bad memory leaks.

All that means it's going to be awhile before CVS updates cause it's gonna take awhile to go through all that.

Oh BTW I already thoroughly commented Packer but haven't put it on CVS yet because I haven't finished the unpack function (sorry).

[Joe Fenton]

Originally posted by General Lee D. Mented
Ok, you go and rewrite the entire thing in assembly. No really, feel free, the source is online.

I don't know assembly (well not x86 assembly at least) so I'm not going to be doing that myself. I'm not even comfortable using memcopy let alone doing my own constructors by hand.

I was going to write it in C# but people complained it would complicate porting it to other platforms, so I went with C++. I'll probably go and rewrite it later anyway, once I can just have all the critical sections in optimized plugins. Probably a few months after it's running.

Anyone wanna do the Directshow and Gstreamer interfaces for me? No? How bout Quicktime and a MacOS port of the core, any takers?

Once it's a bit more - mature - I might just take a crack at it. It looks like a rather interesting codec, but it currently doesn't really do anything. No need to try doing assembly until then. That is one big advantage in C/C++, faster coding. You can whip something together to see if the idea has merit in considerly less time. You have to be committed to the idea to work in assembly.

I work on a PowerMac (among other things), so I may work on the QT side of it.

[General Lee D. Mented]

Originally posted by Joe Fenton
Once it's a bit more - mature - I might just take a crack at it. It looks like a rather interesting codec, but it currently doesn't really do anything. No need to try doing assembly until then. That is one big advantage in C/C++, faster coding. You can whip something together to see if the idea has merit in considerly less time. You have to be committed to the idea to work in assembly.

I work on a PowerMac (among other things), so I may work on the QT side of it.

Ok, that'd be very helpful. I plan to move things to a plugin format to help ease optimizations for different architechtures and such after the first version is stable.

It really doesn't compress on it's own, it has to feed to an entropy coder like lzh, lzo, bwth, ppmz, etc to achieve compression. I got the best simulated results with bzip2's burrows-wheeler transform + huffman. I'm anticipating the BWT portion of the algorithm will be the bottleneck in terms of speed. Right now the plan is to link to libbz2 and use existing code, but their maximum blocksize of 900k is severely limiting for video applications that can use gigabytes of data. It's just too narrow a window for really efficient compression. If you want, that would be a good place to start looking for optimizations. I think I have some source a friend of mine was working on for an improved BTW that both runs more efficiently and takes larger working blocksizes, but it was never fully completed.

[Joe Fenton]

Originally posted by General Lee D. Mented
Ok, that'd be very helpful. I plan to move things to a plugin format to help ease optimizations for different architechtures and such after the first version is stable.

It really doesn't compress on it's own, it has to feed to an entropy coder like lzh, lzo, bwth, ppmz, etc to achieve compression. I got the best simulated results with bzip2's burrows-wheeler transform + huffman. I'm anticipating the BWT portion of the algorithm will be the bottleneck in terms of speed. Right now the plan is to link to libbz2 and use existing code, but their maximum blocksize of 900k is severely limiting for video applications that can use gigabytes of data. It's just too narrow a window for really efficient compression. If you want, that would be a good place to start looking for optimizations. I think I have some source a friend of mine was working on for an improved BTW that both runs more efficiently and takes larger working blocksizes, but it was never fully completed.

As long as it's just the compressor/decompressor, you just need to make a separate component for them. It's when you are trying to also support a new container (AVI, OGM, MKV) that you run into all the FUN stuff in QT.

Since libbz2 only works up to 900K, maybe you should use it on a frame basis instead of on the whole stream. I can't imagine a frame being larger than 900K. An uncompressed 800x600 video frame is less than 900K, so even with minimal compression for an I-frame, it should still fit at larger resolutions pretty easily. As you generate a frame's worth of compressed data, you then pass it to the BWT/Huffman stage. Besides, trying to compress over the entire video stream isn't error-resilient. You'd never be able to making a version of this codec for streaming video. Compression MUST be over only a single frame, or a small number of frames at most. The largest block you'd want to use would be from one keyframe to the next, which tends to be no more than 10 secs worth of video.

[General Lee D. Mented]

Originally posted by Joe Fenton
As long as it's just the compressor/decompressor, you just need to make a separate component for them. It's when you are trying to also support a new container (AVI, OGM, MKV) that you run into all the FUN stuff in QT.

Since libbz2 only works up to 900K, maybe you should use it on a frame basis instead of on the whole stream. I can't imagine a frame being larger than 900K. An uncompressed 800x600 video frame is less than 900K, so even with minimal compression for an I-frame, it should still fit at larger resolutions pretty easily. As you generate a frame's worth of compressed data, you then pass it to the BWT/Huffman stage. Besides, trying to compress over the entire video stream isn't error-resilient. You'd never be able to making a version of this codec for streaming video. Compression MUST be over only a single frame, or a small number of frames at most. The largest block you'd want to use would be from one keyframe to the next, which tends to be no more than 10 secs worth of video.


Compressing only one frame at a time would defeat the major design feature of the project. That's like saying "You shouldn't do lossy compression because it's inconvenient." Besides, once the transform is run it's impossible to deconstruct the block into single frames until decode time. I'd like to use a nice 64-256MB window for the BWT, but unfortunately I haven't found anything that will allow that, which is silly. If the user wants to shoot themselves in the foot with slow compression you should let them, or at most warn, not forbid it. What happens when machines speed up by another factor of 1000? I'm sure 900K seemed very large back in the days of 1.44 floppies and such, but it's quite pitiful now. I had someone take a look at bzip2's source and try to hack in larger support. But after about 2.3MB it blew up. Apparently nobody ever taught the author that hardcoded limits are bad, m'kay?

And there's other ways to do streaming, but that's for another day. ;)

[Joe Fenton]

I meant the BWT/HUFF part of the compression, not the wavelet/quantization part. I also mentioned that you might also just go from keyframe to keyframe. That's kinda how MPEG does it - you get compression in the form of P and B frames from one I frame to the next, but they do huffman on almost everything.

I agree that doing compression on the whole stream gives you the ability to do some major compression. Just look at RARs made of AVIs. It will prevent you from using the codec as a streaming format though. Not that big a loss, but something to consider.

Many compression standards were originally limited in the size they could handle. The company I used to work for had a custom compression they used for the executable of the program, but it had an internal limit of 256K (not MY fault... that was before I joined). Me and the guy who wrote it went back and rewrote it to handle 32bit lengths. The limit usually comes from offsets in the dictionary used for reverse lookup/decoding.

[General Lee D. Mented]

Originally posted by Joe Fenton
I meant the BWT/HUFF part of the compression, not the wavelet/quantization part. I also mentioned that you might also just go from keyframe to keyframe. That's kinda how MPEG does it - you get compression in the form of P and B frames from one I frame to the next, but they do huffman on almost everything.


Ok, let's see, right now I'm taking a group of frames, say 64. I cut that into 8x8x64 pixel volumes (it's user definable but just as an example) which get scanned through a fractal order LUT and stored in the Chunk objects. Then each chunk is transformed and sorted. Then they're all put into a global heap, which selects which wavelets to keep based on a priority system. Then the results are copied back into the chunks in the original (pre-sort but post-transform) order, fed to a modified RLE that compresses out the long strings of 0s, and then packed into a single buffer.

Now, exactly how would I go about extracting single frames out of that to feed to the BWT or any other entropy coder, which is the final stage in this process?

Also there aren't any keyframes, since there's no motion estimation. You can consider the entire 64 frame block a group of B-frames, which I believe they're closest to in MPEG terms.

In theory I could apply the entropy coding to individual chunk volumes before combining them, and I've considered that. But I'm worried about how much I'll lose in compressability by only being able to observe small regions of the image at a time when searching for redundant data. I'd at least like to get say, the entire luma channel into a single compare window. I'm also considering a two pass process where I do each one individually and then the result of that as a whole, but it'll require real world testing to see what the impact is in terms of performance.

Also there was some good news from others doing testing today, did some work with the zero-RLE on my old test data and it compresses between 4:1 and 30:1 just by itself. That means there'll probably be less load on the entropy stage than I'd thought.

As for streaming, consider the case where I use a less efficient entropy system that encodes the data in various stages. Let's say each block of data is stored as a series of passes, and each pass represents a single wavelet update to each chunk in the video. We store these in descending priority order, and begin a buffered transmission. As playback is streaming we keep sending layers of lower and lower priority data, until it is determined a threshold based on the network performance has been reached, then go on to the next block. Obviously you'd want to buffer at least one block into the future but the non-streaming static codec needs that for my case anyway. With this system, quality can dynamicly adjust up or down to any given bitrate (within realistic threshold limits) with a very fine granularity (probably about 2kbit/sec) so that a single stream can service a decoder at almost any given bandwidth connection without the need for multiple encoded streams, a la microsoft or real's streaming system. It can also do this on the fly, so if a network connection suddenly becomes significantly slower the video quality can decrease dynamicly without pausing the video for rebuffering or having to change to a lower bandwidth stream.

This would take some work to implement and it's not a high priority for now so I don't have any plans to write the streaming mode right now. It would have some serious differences though, such as storing data in value/position pairs instead of having RLE zero counts to maintain wavelet positions.

[silver_cpu]

Short and sweet: I'm not a coder, I'm not a specialist, but I am an enthusiast. And I'm very enthused about this project. I think a lot of good can come from it, and I hope to watch it grow :) Maybe even participate, once it does enough to be tested by end users. It's always good to see that indie programmers are still working on new and exciting things. Keep up the great work!

[Joe Fenton]

Originally posted by General Lee D. Mented
Ok, let's see, right now I'm taking a group of frames, say 64. I cut that into 8x8x64 pixel volumes (it's user definable but just as an example) which get scanned through a fractal order LUT and stored in the Chunk objects. Then each chunk is transformed and sorted. Then they're all put into a global heap, which selects which wavelets to keep based on a priority system. Then the results are copied back into the chunks in the original (pre-sort but post-transform) order, fed to a modified RLE that compresses out the long strings of 0s, and then packed into a single buffer.

Now, exactly how would I go about extracting single frames out of that to feed to the BWT or any other entropy coder, which is the final stage in this process?

Also there aren't any keyframes, since there's no motion estimation. You can consider the entire 64 frame block a group of B-frames, which I believe they're closest to in MPEG terms.

I see your point. Probably best to do it as you are. I guess you really thought that out ahead of time. :)

Originally posted by General Lee D. Mented
In theory I could apply the entropy coding to individual chunk volumes before combining them, and I've considered that. But I'm worried about how much I'll lose in compressability by only being able to observe small regions of the image at a time when searching for redundant data. I'd at least like to get say, the entire luma channel into a single compare window. I'm also considering a two pass process where I do each one individually and then the result of that as a whole, but it'll require real world testing to see what the impact is in terms of performance.

Also there was some good news from others doing testing today, did some work with the zero-RLE on my old test data and it compresses between 4:1 and 30:1 just by itself. That means there'll probably be less load on the entropy stage than I'd thought.

That's good to hear. Although there are data sets that cannot theorectically be compressed, video data generally lends itself pretty well to short term compression. The raster nature of video clumps the information around certain frequencies. It's the main reason nearly all video compression is done in the frequency domain as opposed to the spacial domain.

Originally posted by General Lee D. Mented
As for streaming, consider the case where I use a less efficient entropy system that encodes the data in various stages. Let's say each block of data is stored as a series of passes, and each pass represents a single wavelet update to each chunk in the video. We store these in descending priority order, and begin a buffered transmission. As playback is streaming we keep sending layers of lower and lower priority data, until it is determined a threshold based on the network performance has been reached, then go on to the next block. Obviously you'd want to buffer at least one block into the future but the non-streaming static codec needs that for my case anyway. With this system, quality can dynamicly adjust up or down to any given bitrate (within realistic threshold limits) with a very fine granularity (probably about 2kbit/sec) so that a single stream can service a decoder at almost any given bandwidth connection without the need for multiple encoded streams, a la microsoft or real's streaming system. It can also do this on the fly, so if a network connection suddenly becomes significantly slower the video quality can decrease dynamicly without pausing the video for rebuffering or having to change to a lower bandwidth stream.

This would take some work to implement and it's not a high priority for now so I don't have any plans to write the streaming mode right now. It would have some serious differences though, such as storing data in value/position pairs instead of having RLE zero counts to maintain wavelet positions.

I have an article on "Graceful degradation of data streams" that covers a similar approach to problems with digital transmission in noisy environments. That is definitely something to worry about later.

[General Lee D. Mented]

Originally posted by Joe Fenton
I see your point. Probably best to do it as you are. I guess you really thought that out ahead of time. :)


Well, given I had to spend alot of time learning the skills to create this before even beginning to actually write it, I spent alot of time just thinking about it. Of the first 2 years, I'd estimate 98% of the time was spent in the design stage. Out of that came things like the universal frame structure and flexibility with colorspace and scalability with memory and cpu resources. There are definately some serious tradeoffs and things that will just have to be worked around, like having to buffer frames in blocks, which does not sit well with "one in one out" video APIs like VfW. Thus there'll have to be some rule-bending and compromises, but I think overall the results will be worth it.


That's good to hear. Although there are data sets that cannot theorectically be compressed, video data generally lends itself pretty well to short term compression. The raster nature of video clumps the information around certain frequencies. It's the main reason nearly all video compression is done in the frequency domain as opposed to the spacial domain.


Actually I completely ignore frequency attributes. I realize this is another one of those "shock" design features like no motion estimation that gives MPEG coders fits, but I think alot of the decisions around the MPEG/JPEG design were based on bad prior assumptions from dealing with older standards such as NTSC broadcasting technologies. I can see where someone 25 years ago might have been thinking of how to do things in terms of frequencies and sine waves and discrete 2D fields and 2D block translation vectors, as it parallels earlier analog designs and theory. They probably started with existing analog standards and said "How can we use digital technology in conjunction with this to achieve compression?"

My approach was that I believed biological systems have already evolved much more optimal algorithms, and how could I create a digital model of what it is believed the biological systems do. Obviously there is selective information loss in the eye/brain system or things like sleight of hand or subsampled chroma or interlacing would not be as effective as they are. So I tried to recreate a similar system with the hopes that by compressing information in a similar way to the eye and brain, the resulting loss of detail (aka artifacts) would be less visibly noticeable because the recompression on the biological side would ignore the same things that the compression on the digital stream did in the first place.


I have an article on "Graceful degradation of data streams" that covers a similar approach to problems with digital transmission in noisy environments. That is definitely something to worry about later.

That will probably be useful when the time comes, but streaming is way down the list, probably somewhere around the distributed processing system.

[Joe Fenton]

Originally posted by General Lee D. Mented
Well, given I had to spend alot of time learning the skills to create this before even beginning to actually write it, I spent alot of time just thinking about it. Of the first 2 years, I'd estimate 98% of the time was spent in the design stage. Out of that came things like the universal frame structure and flexibility with colorspace and scalability with memory and cpu resources. There are definately some serious tradeoffs and things that will just have to be worked around, like having to buffer frames in blocks, which does not sit well with "one in one out" video APIs like VfW. Thus there'll have to be some rule-bending and compromises, but I think overall the results will be worth it.

I was going to mention that. I can see this as having some trouble with audio sync. Nothing that can't be solved, but it won't fit some video containers very well at all. I don't see this as AVI for example.

Originally posted by General Lee D. Mented
Actually I completely ignore frequency attributes. I realize this is another one of those "shock" design features like no motion estimation that gives MPEG coders fits, but I think alot of the decisions around the MPEG/JPEG design were based on bad prior assumptions from dealing with older standards such as NTSC broadcasting technologies. I can see where someone 25 years ago might have been thinking of how to do things in terms of frequencies and sine waves and discrete 2D fields and 2D block translation vectors, as it parallels earlier analog designs and theory. They probably started with existing analog standards and said "How can we use digital technology in conjunction with this to achieve compression?"

My approach was that I believed biological systems have already evolved much more optimal algorithms, and how could I create a digital model of what it is believed the biological systems do. Obviously there is selective information loss in the eye/brain system or things like sleight of hand or subsampled chroma or interlacing would not be as effective as they are. So I tried to recreate a similar system with the hopes that by compressing information in a similar way to the eye and brain, the resulting loss of detail (aka artifacts) would be less visibly noticeable because the recompression on the biological side would ignore the same things that the compression on the digital stream did in the first place.

Actually, you don't ignore frequency attributes as the wavelet transform is a frequency based transform. Anything done on wavelets is in the frequency domain. But I do understand what you're getting at. Most video compression schemes in use today are based on the "understanding" NTSC spectrum interlacing introduced with color broadcasting. I've seen a few attempts at not using frequency info. For example, I have a circuit design article on a geometric based color filter for NTSC as opposed to the standard comb filter. The only geometric video compression scheme I'm aware of is the fractal compression scheme. RealVideo used that in one of their versions. It's pretty good for certain uses. Your compressor here seems to have a few ideas that come from that as well as from neural nets... compression based on the identification of structures rather than arbitary frequency histograms. Wavelets tend to be useful in structure recognition, so you seem to be doing wavelet based prioritized video feature decomposition.

[Lobuz]

I'm not sure if that's what you're looking for but 7z format is GPLed and have better comression ratios than rar. Dictionary size can be set to 256MB in commandline version. It uses LZMA "an algorithm based on Lempel-Ziv algorithm" and is here (http://www.7-zip.org/) .

Regards
Lobuz

[General Lee D. Mented]

Originally posted by Lobuz
I'm not sure if that's what you're looking for but 7z format is GPLed and have better comression ratios than rar. Dictionary size can be set to 256MB in commandline version. It uses LZMA "an algorithm based on Lempel-Ziv algorithm" and is here (http://www.7-zip.org/) .

Regards
Lobuz

Umm I looked at 7zip briefly and it seems like there's no documentation in existance for it, so I can't figure out what things like -m do. Also, I don't see any way to set dictionary size with it.

[General Lee D. Mented]

Originally posted by Joe Fenton
I was going to mention that. I can see this as having some trouble with audio sync. Nothing that can't be solved, but it won't fit some video containers very well at all. I don't see this as AVI for example.


Oh it'll work in AVI, just with a little help, such as the delay audio feature of virtualdub. The audio just needs to be skewed by one block to sync, and since the differential is constant it shouldn't be hard to do. Seeking presents some challenges though.

[i]
Actually, you don't ignore frequency attributes as the wavelet transform is a frequency based transform. Anything done on wavelets is in the frequency domain. But I do understand what you're getting at. Most video compression schemes in use today are based on the "understanding" NTSC spectrum interlacing introduced with color broadcasting. I've seen a few attempts at not using frequency info. For example, I have a circuit design article on a geometric based color filter for NTSC as opposed to the standard comb filter. The only geometric video compression scheme I'm aware of is the fractal compression scheme. RealVideo used that in one of their versions. It's pretty good for certain uses. Your compressor here seems to have a few ideas that come from that as well as from neural nets... compression based on the identification of structures rather than arbitary frequency histograms. Wavelets tend to be useful in structure recognition, so you seem to be doing wavelet based prioritized video feature decomposition. [/B]

The frequency data comes from the index position of the results in a transform such as a DCT or wavelet transform. I don't consider the index position at all in my selection system other than when it determines whether more values will be impacted by error due to how the transform is structured with dependancies. Rather I'm only interested in the AMPLITUDE of the result, hence that A that comes after the W in the codec name.

My main principle works on analyising the signal for large deltas in time, i.e. high amplitude wavelets. The idea being that big changes are more visible than small ones. The speed at which they happen is less important. The main reason I'm using a wavelet transform is it gives me a recursive set of relative change data, i.e. the change from frame 1 to 2, 2 to 3, etc, as well as the first 4 to the second 4, first half to the second half, etc. Therefore it's easy to pick out where the relative changes are and the frequency they occur at doesn't matter because the transform gets all of them.

The new transform I'm working on has somewhat different properties than the haar transform I currently use though, and is more ideally suited to this process, because I designed it myself for it. It's not a standard wavelet transform.

[Lobuz]

In binary downlod file there is a help file with all switches:

LZMA

d={Size}
Sets Dictionary size for LZMA. You must specify size in bytes, kilobytes, or megabytes. The maximum value for dictionary size is [B]256MB = 2^28 bytes. Default value for LZMA is 20 (1MB) in normal mode and 23 (8MB) in maiximum mode (-mx). If you do not specify any symbol from set [b|k|m], dictionary size will be calculated as DictionarySize = 2^Size bytes. For decompressing file compressed by LZMA method with dictionary size N you need about N bytes of memory (RAM) available.

mf={MF_ID}
Sets Match Finder for LZMA. Default method is bt4. Methods from bt* group require less memory than methods from pat* group. Usually bt4 works faster than any pat*, but for some types of files pat* can work faster. Algorithms from hc* group doesn't provide good compression ratio, but they often works pretty fast in combination with fast mode (a=0). Memory requirements depend from dictionary size (parameter "d" in table below).

MF_ID Memory Description
bt2 d*9.5 + 1MB Binary Tree with 2 bytes hashing.
bt3 d*9.5 + 65MB Binary Tree with 2-3(full) bytes hashing.
bt4 d*9.5 + 6MB Binary Tree with 2-3-4 bytes hashing.
bt4b d*9.5 + 34MB Binary Tree with 2-3-4(big) bytes hashing.
pat2r d*26 + 1MB Patricia Tree with 2-bits nodes, removing.
pat2 d*38 + 1MB Patricia Tree with 2-bits nodes.
pat2h d*38 + 77MB Patricia Tree with 2-bits nodes, 2-3 bytes hashing.
pat3h d*62 + 85MB Patricia Tree with 3-bits nodes, 2-3 bytes hashing.
pat4h d*110 + 101MB Patricia Tree with 4-bits nodes, 2-3 bytes hashing.
hc3 d*5.5 + 1MB Hash Chain with 2-3 bytes hashing.
hc4 d*5.5 + 6MB Hash Chain with 2-3-4 bytes hashing.


Regards
Lobuz

[General Lee D. Mented]

Thanks, I dug around and found it. I can't get 256M to run due to the high multiplier in the trees. However 128M is producing some very very impressive results.

I'll have to try linking to their lib in the beta as well as libbz2. I think LZMA is going to win out at this point, especially because of the faster and lower memory decode.

[Lobuz]

Maybe try to add additional swap files. ;-) It will be slow but ...
If system pf is not enough I think TweakXP allows to set 3 4GB pagefiles. I didn't tryed. Good luck.

Regards
Lobuz

[Lobuz]

@General Lee D. Mented
In the case You doesn't know that DataCompression.info (http://www.datacompression.info/). It's I think the biggest base of links to papers, programs, sites about any aspect of data compression. So it could be usefull.

Regards
Lobuz

[silver_cpu]

What kind of system resources are you predicting will be needed to effectively use this codec, once it's initial development stages are complete? Will people with a measley 512MB of RAM be able to work efficiently with it, or will we need several gigs of RAM to keep our hdd's from doing most of the data swapping?

[TheXung]

As exciting as this codec sounds, can you give us any indication of the level of compression you think is possible? Any preliminary test results. Perhaps some papers that you are basing the any research from.

Sad as this may seem, I think you could get even more assistance by building up hype and fud. Many open source developers need to get excited about a project before they jump in.

From the description of how this works, I think realtime decompression will definitely be possible. Sounds like just entropy decoding and inverse transform. The only limitation sounds like the amount of memory needed for buffering.

[General Lee D. Mented]

I'll answer silver_cpu's post in this one too, to save space.

Originally posted by TheXung
As exciting as this codec sounds, can you give us any indication of the level of compression you think is possible? Any preliminary test results. Perhaps some papers that you are basing the any research from.


I don't know, I'd like to see 1000:1 but I don't know how practical it will be to reach that level given memory constraints. In theory efficiency should scale with memory resources, but I don't know how well or what the baseline will be like.

As for research, there aren't any papers because I never wrote any on my theories.


Sad as this may seem, I think you could get even more assistance by building up hype and fud. Many open source developers need to get excited about a project before they jump in.


Possibly, but if the hype and fud so far hasn't attracted people, I dunno what will. Maybe once the project goes beta if doom9 or someone posts about it it will attract more developers.


From the description of how this works, I think realtime decompression will definitely be possible. Sounds like just entropy decoding and inverse transform. The only limitation sounds like the amount of memory needed for buffering.

Yes, memory will be a problem. I'm hoping that a machine with 256MB and a 1ghz cpu will provide a suitable decode platform for compression similar to current mpeg4 codecs. Encoding will most likely require several times the memory footprint, I'd estimtate 4-8 times, so swapping will probably be an issue on all but the most powerful machines. 64bit hardware will help somewhat by allowing larger memory sizes.

I've had excellent results with the suggestion of 7zip's LZMA algorithm and a large dictionary size. It's currently beating bzip2 in compression and its decompress performance and memory use are highly favorable. At this point it will probably win out in a head to head test with most of the other popular algorithms in use including bwth(bzip), bwt+arithmetic, lzh(zip), lzo, ppmz, and dmc.

Since the entropy coder will be linked seperately to the codec, I don't forsee much of a problem in swapping one for another and testing them all to see which works best.

[General Lee D. Mented]

Just threw a massive update into CVS. It's mostly work on the core, though I did go back and modify VfW so it at least compiles with it. Here's the changes:

Changed core interface functions to take 2 WARPframes instead of 1 frame and 1 buffer, for clarity.

Rewrote ALL object config functions to take WARPconfig params, and deleted local variables to reduce confusion and standardize.

Changed Selector::Select() to take a double argument to use as the bitpool each time and moved the bitpool calculation to WARPcodec. This will simplify things when it comes time to write bitrate control.

Wrote WARPpacker::Unpack() and the reverse of the RLE functions, so that should be complete.

Wrote WARPcodec::Decompress() now that packer is complete it should be workable.

Everyone interested in helping out, now would be a good time to grab the CVS and try some test drivers on core to see if it is producing correct data.

[General Lee D. Mented]

http://www.gldm.net/avitest.jpg

Well it's *almost* working....

It runs through an entire compress sequence in avs2avi, then dies and never outputs the file.

The problem seems to be the heap going bad again, because I get that whole debug assert failed _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) problem. The line is in avs2avi.cpp 's Cleanup() function, at ICCompressorFree(&CompVars); The line before that is GlobalFreePtr(SRCData); in avs2avi's Compress() function. Everything before that executes just fine, then it bombs. I don't know where the compressed data goes, it seems to get wiped when the app dies.

Any ideas? Anyone?

[jcsston]

Originally posted by General Lee D. Mented
The problem seems to be the heap going bad again, because I get that whole debug assert failed _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) problem.
I always use the following when trying to debug memory problems

// Header
#ifdef _DEBUG
#include <crtdbg.h>
#endif

// Inside Dllmain
#ifdef _DEBUG
// Setup the debug options
_CrtSetDbgFlag( _CRTDBG_ALLOC_MEM_DF
//| _CRTDBG_LEAK_CHECK_DF //Check for memory leaks on app exit
| _CRTDBG_CHECK_ALWAYS_DF); // Check the memory heap at every new/delete
#endif

With this it will break at the next new/delete after the memory heap has gone bad.

Once you start trying to track down memory leaks a memory alloc hook function comes in really handly. ;)

#ifdef _DEBUG
int YourAllocHook(int allocType, void *userData, size_t size, int blockType, long requestNumber, const unsigned char *filename, int lineNumber)
{
//I can use this to find exactly where a leak started
if (size == 8 && requestNumber == 1796)
{
printf("cool"); // set your breakpoint here
}
return TRUE;
};
#endif

// inside Dllmain
#ifdef _DEBUG
_CrtSetAllocHook(YourAllocHook);
#endif

Good luck :)

Edit: I got the debug assert '_BLOCK_TYPE_IS_VALID(pHead->nBlockUse)' before when I tried new[] a bunch of objects and then delete each one by itself instead of the entire array.

[General Lee D. Mented]

Fixed by removing a delete, it seems to run now. I doubt it's doing the right thing. Observe:

Input to avs2avi was version.avs, an avisynth script that just has "Version()" on one line. It should produce the version info as static text on a static background.

The raw file output is 420x80, 240 frames, RGB. That means 420*80*240*3 = 24,192,000 bytes uncompressed.

The transformed and result test.avi with blocksize 8, timedepth 8, precision 8, bitrate 0 (use lossless) is 437,760, which is about 55:1

The bzip2 -9 of THAT, is 1,136 bytes, which is another 385:1.

But that makes a final compress ratio of 21,295:1.

And that's gotta be impossible! Even with the roundoff errors using 8bit pixels would make.

[General Lee D. Mented]

The 32bit versions that should have no roundoff errors are even more impossible, 25,627:1. For some weird reason it's varying as expected with the parameter changes, but it's still way too small to be a correct result.

[General Lee D. Mented]

Well it's obviously doing something wrong. I've been backtracing through the system and it may be in the scan. But I don't know because nobody ever ran any core tests to see if it returned proper data, and I don't know how to write them easily.

Where is Spyder anyway?

[TheXung]

I haven't had time to look at your sources but if you wrote this entire thing without validating any layers of it, there could be lots of bugs and they could be very hard to track down because you aren't sure whats working and whats not.

When I work on any lossless compression, I almost always work on the compressor and decompressor in tandem because that is the only way I can be sure that improvements are actually improvements.

If you don't already, I suggest you have a decompressor. Debug this one layer at a time starting either with the top layer or bottom layer. Essentially follow the spiral model of development. For instance, if the first thing your encoder does is a transform, make it output the coefficents of that in just text and have the decoder do the inverse transform from the coefficients. Then whatever your next encoding process is, make it output the result of that and then build the decoder to reverse that. And so on. For me, bitstream stuff is always the last thing that I put into place.

[General Lee D. Mented]

Originally posted by TheXung
I haven't had time to look at your sources but if you wrote this entire thing without validating any layers of it, there could be lots of bugs and they could be very hard to track down because you aren't sure whats working and whats not.

When I work on any lossless compression, I almost always work on the compressor and decompressor in tandem because that is the only way I can be sure that improvements are actually improvements.

If you don't already, I suggest you have a decompressor. Debug this one layer at a time starting either with the top layer or bottom layer. Essentially follow the spiral model of development. For instance, if the first thing your encoder does is a transform, make it output the coefficents of that in just text and have the decoder do the inverse transform from the coefficients. Then whatever your next encoding process is, make it output the result of that and then build the decoder to reverse that. And so on. For me, bitstream stuff is always the last thing that I put into place.

Gee thanks for the programming 101, but I've already done that. Unfortunately it takes a ton of glue logic to stick the individual modules together so that's most likely where something's gone wrong.

I don't know how to get it to write out to text, I was hoping for some help with that but everyone's too busy. Yes there is a decompressor but I haven't even tried running it yet because I know the compress side has a serious problem. Oh and I can't figure out how to configure it.

[TheXung]

I'm sorry if it sounded like I was talking down to you because that wasn't my intention. It just sounds like you're haven't done some really needed testing on key parts that should have been tested way earlier in the production cycle. I'm sure you've put lots of time into this already.


If you want to output text to a file, have a file pointer declared globally; while poor practice, it easiest to just declare it globally since this is just for debugging purposes.

FILE *fp;

Where you instantiate the codec, open the file.

fp=fopen("C:\\FILENAME.TXT","w");

And then whereever in the code you're ready to output something, do a fprintf

fprintf(fp, "value=%3d anothervalue=%3f\n", y, x);


Or you could just output to debugview. Run this in the background http://downloads-zdnet.com.com/3000-2218-10213956.html

and use
OutputDebugString(char *);

[General Lee D. Mented]

Well, the main bug turned out to be that ScanFrame() was not traversing the frame properly. I forgot some variables in a calculation so it was just scanning the first block of the frame into each chunk, which would obviously compress very well but not be very useful. :P


I'm getting problems with VFW again now that that's fixed. I think it's something to do with the returned data and its size.

Everwicked talked me into trying to write a testfile that feeds raw rgb to the core. This is generated by using virtualdub(mod)'s image sequence to output bmps, then paintshop pro to strip the bmps to raw data, which is then read in from file pointers. However, I'm not very good with dealing with files yet and so I haven't actually got it to work so far. I think that it will help pinpoint the latest problem though. It does look like the data is actually being processed correctly and is there, I just don't know what's wrong with the return values yet.

[General Lee D. Mented]

Does anybody even care anymore?

[Atamido]

Yes, we are all waiting.

[sh0dan]

We just want to see the money. I need to see something useable before getting excited about it. For now this is just vaporware.

[silver_cpu]

I'm very interested in this codec, and wonder greatly what will come of it. I'm sure you're working hard on it, so keep up the good work!

[mf]

Originally posted by sh0dan
I need to see something useable before getting excited about it. For now this is just vaporware.
I've seen a buggy AVI clip from a very old test version, and it looks promising. :D

[Baalthazaar]

Many of us are eagerly awaiting a semi-stable release to test with, even if we can't help you with the code. Personally I'd love to take your codec on a spin just because it's so much different than today's standard codecs.

[ChristianHJW]

You came such a long way Gldm, and i admit i had doubts if the project would ever come that far, dont give up now, there seems to be light on the end of the tunnel ....

[General Lee D. Mented]

I just threw my latest working code into CVS. Added the rawtest project which builds an exe that can test the core dll by feeding it data from .raw files.

To make raw files, use vdubmod's save image sequence to make .bmps. Then use paintshop pro's batch process to convert the bmps to RGB raw files. Make sure you specify 4 digit numbered sequence (0000.raw-9999.raw) on your files, as the tester doesn't perform much sanity checking. Config and such are hard-coded in there.

Thanks to everwicked for most of the rawtest code.


As for project status, compress works in rawtest, but not in vfw, where it throws access violations. Decompress also throws access violations in rawtest. I've noticed it sometimes runs off the end of the compressed array, so I think the problem is likely in WARPpacker's RLE routines or buffer copying. I think the sizes being returned may be wrong somehow, but I can't track it down yet.

I'd really appreciate it if people would compile the latest core and rawtest and see if they can track down any bugs. Oh and yes I know it's leaking memory like hell but I'm not worried about that just yet, let's get it to execute properly first on a small encode and then solve the memory issues later.

[Latexxx]

What kind of hardware does the encoder currently require?

[General Lee D. Mented]

Originally posted by Latexxx
What kind of hardware does the encoder currently require?

Require? Not much, it'll run on virtually anything. It's still very slow though.

[crOOk]

Originally posted by General Lee D. Mented
Does anybody even care anymore?
@General Lee D. Mented
Keep up the good work and don't give up now! Your project sounds really promising.

[ChristianHJW]

Gldm,

any news ? I am still wondering if it was the right thing to do to mess with 2 things the same time, the codec itself and the VfW interface to be able to use it in Vdub.

A proggie like DaveEL's avs2matroska, that takes avisynth scripts on its input, links to the encoder directly and writes into MKV or OGM files would have been a better choice IMHO, but maybe i am wrong. For the playback part i am sure Toff would have helped you on a suitable decoder filter :) ...

[General Lee D. Mented]

Originally posted by ChristianHJW
Gldm,

any news ? I am still wondering if it was the right thing to do to mess with 2 things the same time, the codec itself and the VfW interface to be able to use it in Vdub.

A proggie like DaveEL's avs2matroska, that takes avisynth scripts on its input, links to the encoder directly and writes into MKV or OGM files would have been a better choice IMHO, but maybe i am wrong. For the playback part i am sure Toff would have helped you on a suitable decoder filter :) ...

No news, ever since I fixed the file i/o and got my hopes knocked down on the lossless results, I've been too depressed about it to do any work on it. I know there's still bugs around in the packer but what's the point if it's not going to do anything useful anyway?

Learning to write avi files directly would be about as much work as learning to write the VFW interface. Once the raw file read/write I'm working on now works, I'm pretty sure I can debug VFW in no time. The thing is getting it to work, and I can't get myself interested and nobody else who could do anything cares.

[ChristianHJW]

Originally posted by General Lee D. Mented ..but what's the point if it's not going to do anything useful anyway? ... :( ... are we sure about the results ?
Learning to write avi files directly would be about as much work as learning to write the VFW interface. ... i wasnt talking about AVI :P .... give libmatroska a frame and a timestamp, and tell it what type of frame it is .... then sit back and relax :D ...

[General Lee D. Mented]

Originally posted by ChristianHJW
... :( ... are we sure about the results ?
... i wasnt talking about AVI :P .... give libmatroska a frame and a timestamp, and tell it what type of frame it is .... then sit back and relax :D ...

Dunno about the results. I can't see it being very useful. Even if the compression was higher than say huffyuv or ffvfw's lossless codec, it runs so slow nobody would use it. As for the lossy stage, I don't see how it could ever compete with the kind of ratios other codecs get.

If someone wants to play with using libmatroska feel free, my current cvs has the raw file tester for them to play with.