Help tritical's interpolation research by contributing CPU cycles

[Terranigma]

:(
Hello? Does anyone know tritical's Status? Is he ok or...?
I've haven't seen him around lately, I hope he's ok. Please return, we miss you (well I do anyways). :scared:
You'll be surely missed around, that's for sure. You have inspired me in many ways, you and the lot of you here @ this community. If anyone know his status, please share it with me.

EDIT by neuron2: I took the liberty of changing your title to draw attention to tritical's research. Thanks for inspiring this thread.

[Leak]

If anyone know his status, please share it with me.
Last thing I know is that he's finishing his masters thesis, which should be some kind of filter. But of course this means he can't really get much help from other people...

(He told you so in May, incidentally... (http://forum.doom9.org/showpost.php?p=1005957&postcount=1048) and believe me, writing a masters thesis takes several months - I should know, mine still isn't finished after several years... :D)

Also - why don't you just send him a mail to ask him? ;)

np: Retina.IT - Per Assurdo (Semeion)

[tritical]

Actually, I've just been taking some time away from the forums and avisynth for the summer. However, I have spent a lot of time working on the project mentioned previously. Atm, it is still up in the air as to whether it will be my master's thesis, but the idea definitely works. It has the potential to completely blow away eedi2 in terms of edge directed interpolation and is faster to boot. I am posting here to ask for assistance from anyone who might be interested in helping... specifically, donating computer time to run my training algorithm.

Basically, the idea requires searching very high dimensional spaces using CMAES (http://www.bionik.tu-berlin.de/user/niko/cmaesintro.html). I have written heavily optimized code to do this (sse2, multithreaded, etc..), but even with 2 Q6600's running 24/7 it takes about 3 weeks to complete (that means find 'ok' solutions, not the best possible). The overall computation consists of 41 separate problems (meaning each of these 41 can be run on separate computers). To complete (i.e. find an 'ok' solution) one problem in a reasonable amount of time (say 1-3 days) requires something along the lines of an amd x2, amd fx, core 2 duo, core 2 quad, or similar processor. For reference, my Q6600@3.0Ghz can complete an average problem in ~.5-1 days (they vary in complexity). My 1.6 Pentium M laptop can complete an average problem in 8-12 days. The program requires ~150-400 MBs of memory so a minimum of 1GB of RAM would be necessary if you intend to use your computer while the program is running. Since the program uses avisynth, it requires windows as the os (I've only tested it in 32bit windows xp). The cpu must support sse2.

If anyone agrees to help, all I would need to send you is one dll, one exe, one binary file, and two avi files totaling ~40 MBs. The program writes out results and all information needed to restart (in case you need to stop it or the electricity goes out, etc...) as it goes (these files total about 20KB). It also spits out information via OutputDebugString() which is how you would know when it has reached the minimum training criteria. It is set to run at IDLE priority and can be started such that it wont show up in the taskbar or system tray if desired (though would still show up in task manager). Once training is complete you would just need to send me two files totaling 10KB.

If anyone is interested or has questions post here or send me a pm. If you are on irc join #editrain on irc.freenode.net (should be easier to answer questions and help with setup that way). I guarantee that any cpu time donated will be towards a good cause: better intra interpolation for deinterlacing :).

[halsboss]

I'm interested... a 1Gb AMD3500 is no good, then ? My kid's AMD 3000 (no SSE2) would not be a good either I guess.

[Terranigma]

Glad to see that you're ok. :cool:
I would love to help, but unfortunately my computer doesn't meet the criteria required for such a tedious task. :(

[mitsubishi]

Hi Tritical, I can spare you some CPU cycles, C2D & 2GB. I'm not into gaming right now, so when not encoding or watching HD I have plenty to spare.

[tritical]

For all those interested, I have put up some initial instructions (text from an irc conversation :)) on how to get things going here: http://bengal.missouri.edu/~kes25c/tclips/setup_instructions.txt. This will require some coordination among those participating so that work isn't duplicated. Joining the irc channel mentioned before would work best, but if you just want to communicate with me via pm's or email I can try to keep everything coordinated.

Atm, I am a little behind in generating the initial starting point files, but should have more than enough for everyone by the end of today.

Thanks to everyone helping out.

@halsboss
Unfortunately, neither of those computers would probably work, but thanks a lot for the interest :) (and Terranigma as well).

[tritical]

Most of the number crunching is done at this point. There are still a few parts being worked on, but no more left to be started. I just wanted to thank everyone who contributed (sorry if these aren't your doom9 nicks):

checkers
Flaarn
Mitsubishi
junkd
Kentar
Aqualung99
kaptah
ReufToc
ChiDragon
jinkslynx

What would have taken me about 4 weeks to do alone only took 1 :). I should be able to release an initial version of the filter here in another two weeks or so (there is still one large problem left whose computation can't be split among computers).

[Chainmax]

Sorry to be late, just found out about this. I am expecting to complete an X2 3800+ system with 1GB RAM within a couple of hours, can I still join the party? :)

[tritical]

Yes, actually. I've decided to break the larger problems up into smaller ones. Which will sacrifice some speed in the final filter operation, but means that the computations can be spread across multiple computers. So anyone who wants to continue helping or wants to join in feel free :).

[Revgen]

@tritical

Will this be a filter that the university keeps for profit? Or will you post it as open-source?

[tritical]

Definitely not for profit, and afaik there is no policy granting the university rights or ownership of the ip or the work itself if I used it as my master's thesis. Admittedly, I haven't looked deeply into it. However, since this is still just a hobby project of mine I can do whatever at this point. My plan is to release an initial, closed source version of the filter here in a few weeks. If it turns out that I am not going to use this for my master's thesis then I'll just make it open source at that point.

[psme]

I've a C2D 6600 OC to 3.2G and 2G ram. Let me know if I can be any help.

On the other hand, I know your filter is primary for offline encoding, but my interest is only in realtime DVD playback. Maybe the resulting filter will work as a general film/video deinterlacer. Feed it any (ok, normal 480i/576i) interlaced source and it will automatically detect for 3/2,2/2 or any other pulldown cadence as well as real 60i video content (for the new EDI video deinterlace processing), with a user perfected output framerate (60 for general output with mixed content, 24/48 for film based content etc), all in auto mode, like a real world video deinterlacer in a standalone video processor/scaler box. And it'd be great if all the process can be run in realtime on my C2D at 3.2G. Or I will get a 4 core in a heart beat if needed! :D

Thanks in advance.

regards,

Li On

[foxyshadis]

As long as there's more remaining in http://bengal.missouri.edu/~kes25c/tclips/who_has_what.txt help is needed. ;) I'm crossing my fingers on speed too.

[psme]

Ok, download all the files from the ftp...

regards,

Li On

[tritical]

I can give some rough numbers on the speed of the final filter. Running on 720x480 YV12 frames, my 1.6Ghz pentium M laptop gets ~2 frames per second. My Q6600 @ 2.8Ghz gets ~20 frames per second. The speed does depend on the image content or complexity, and I would say those numbers are towards the low end because the sequence I used to test is primarily made up of complex frames. Anyways, it definitely isn't going to set any speed records or run realtime on a typical system. However, it is highly parallelizable and should scale very well to 8 or 16 core systems.

As far as realtime playback and film/video detection, it doesn't interest me that much. So the likelihood that I would create such a filter is pretty small, though not zero :).

[cwk]

Just for reference purposes Tritical, how fast would eedi2 run in those environments for that file?

Is the quality as good as you were hoping for?

cwk

[Chainmax]

Ok, I took stage1-09_8_ffv1.avi/output-stage2-09_8-linear.txt. My AVS is now like this:

SetMemoryMax(16)

LoadPlugin("C:\tritical\nntrain.dll")

avisource("C:\tritical\tall_ffv1")

nnborder()

mskc = avisource("C:\tritical\stage1-09_8_ffv1.avi")

nntrain(maskc=mskc,restart="C:\tritical\output-stage2-09_8-linear.txt")

There is one thing I'm missing though: where do I set the PATH environment variable?

[juhok]

There is one thing I'm missing though: where do I set the PATH environment variable?
http://www.siluriformes.net/doom9/path.png

[Chainmax]

First try gave an error because I forgot to add the .AVI extension to the first AVISource call :o. It seems to be working now.

[edit]If I understood correctly, this output:

http://members.arstechnica.com/x/master_burrfoot/Dibujo.PNG

means it went from iteration 19316 to 19317, is that correct?

[tritical]

@Chainmax
Yep, that is correct.

@cwk
I did a more complete speed test. Results for bobbing the 657 frame clip I mentioned earlier:
1.6 Ghz Pentium M:
EEDI2: 3.74 fps
New Filter: 2.38 fps

Q6600 @ 2.8 Ghz:
EEDI2: 7.64 fps
EEDI2 (mt): 18.77 fps (multithreaded version by foxyshadis)
New Filter (1 threads): 6.99 fps
New Filter (2 threads): 13.27 fps
New Filter (4 threads): 23.89 fps
New Filter (8 threads): 23.46 fps

The quality is pretty good. On the 657 frame clip I used for speed testing, it achieves more than a 25% reduction in squared error compared to cubic interpolation. For reference, eedi2 achieves a 7.5% reduction, and all of the common ela methods (like those found in tdeint, tomsmocomp, yadif) are worse than cubic. This clip contains frames that should heavily favor an edge directed interpolater over cubic. However, there are some things it doesn't handle as well as it could. It should be relatively easy to improve its performance in those areas though.

EDIT: there was a little problem with the fps numbers the first time.

[Chainmax]

Approaching the 29k mark...
Sorry for the slowness, but I can't leave it on 24h a day :(.

[tritical]

If you just want to upload it now, I can finish the last 20k on one of my computers in probably 5 hours. It is just doing further training on already completed problems atm.

[Chainmax]

I'd like to see it reach at least the 40k mark before uploading it, but if doing it now will result in better data then I will upload it at once.

[Chainmax]

Files have been uploaded.

[Adub]

Is this still being done? Or has your research finished, Tritical?
I have a e6600 just sitting around, so I could set it up to run 24hrs a day, if you need it.

[foxyshadis]

Still lots left. Check who has what (http://bengal.missouri.edu/~kes25c/tclips/who_has_what.txt) to see.

[squid_80]

I grabbed 34, just to have a look see. It really manages to push the CPU - my Q6600 is sitting on 71 degrees right now, which has never happened before. Should I be worried?

[Dark Shikari]

I've got 36.

[burfadel]

I grabbed 34, just to have a look see. It really manages to push the CPU - my Q6600 is sitting on 71 degrees right now, which has never happened before. Should I be worried?

Depends on what programme you use to view the CPU temps...

[Dark Shikari]

Depends on what programme you use to view the CPU temps...71 C is pretty normal for a full-load Core 2 Quad to be honest. To get lower you need a seriously powerful air cooler or better. Four cores generate a hell of a lot of heat.

[squid_80]

Depends on what programme you use to view the CPU temps...
The program that came with the motherboard and speedfan 4.32 both show the same values, so I'd assume they're correct.

EDIT: I'm assuming other people have run this on a Q6600 and can tell me if it's normal or not.

[tritical]

I run it on 2 Q6600s overclocked to 2.8 and 3.0, but they aren't using the stock heatsink/fan so I'm not sure this info will help much. Both stay right around 55C for cores 0/1 and 50C for cores 2/3. That is according to core temp v0.95. They both have Thermalright Ultra 120A heatsinks with 120mm Scythe fans which move about 49 CFM.

Also, are you going to run 34 or just testing? ... I couldn't tell from your post.

[squid_80]

I'll run it completely. My Q6600 is a B0 stepping (engineering sample) so it's probably going to be hotter than the latest (G0?). It's got a Zalman CNPS7700 cooler (didn't come with a stock cooler), not overclocked, cores 0/1 are 80C and 2/3 are 71C. The only thing that worries me is that I think the bios is set to shutdown at 75C. Good thing the weather is still cold here.

[squid_80]

I took 35 too. May as well give the E6600 something to do.

[Demi9OD]

Coretemp reports my Q6600@3400mhz@1.375v 62/62/57/57 under load. Lapped U120 and Lapped Q6600.

[superuser]

is help still needed. I can try to allocate some of my computing for this purpose. let me know.

regards