Idea: Avisynth filtering/encoding contest?

[Dark Shikari]

An idea I had--why not have an Avisynth filtering contest to encourage people to make better, more versatile, and more interesting scripts for useful purposes?

The contest would consist of a posted clip and a target bitrate/resolution, and would be of one of two types:

1. Source restoration/postprocessing. The posted clip is a complete mess; perhaps a VCD rip, or even something uploaded to Youtube. The goal is to fix the source. The target bitrate is very high.

2. Encoding/preprocessing. The posted clip is picture-perfect, but for whatever reason extremely difficult to compress. The goal is to improve compression as much as possible without sacrificing quality--examples would include degrain scripts, etc. The target bitrate would be very low.

The winners would be decided through a blind vote after each contest, and the posted scripts could then be looked at--perhaps some would be adopted as the next-generation filter chains.

What does everyone think of the idea?

[neuron2]

I think we get enough challenges in the course of daily Doom9 life. What's lacking is the voting and crowning of a winner. Sounds fun but we don't have any infrastructure for blind voting, etc. And then, in effect, asking for the best script or encode is itself problematic in view of rule 12.

[Dark Shikari]

I think we get enough challenges in the course of daily Doom9 life. What's lacking is the voting and crowning of a winner. Sounds fun but we don't have any infrastructure for blind voting, etc. And then, in effect, asking for the best script or encode is itself problematic in view of rule 12.Doesn't Doom9's forum have a poll feature?

Blind voting is easy--just have all the videos uploaded in the same format.

In a sense I do agree on the "crowning of a winner" point you made--but on the other hand, it would be interesting to have some competition for specific types of scripts. We really don't have that many "magical" scripts, and even TemporalDegrain has some serious issues (http://shrani.si/f/27/Vh/2ZjVrtLf/bad-degrain-frame-0106.png).

[neuron2]

You're free to post threads and make polls. :)

I think at first post the challenge in one thread but not yet the poll, because you won't know how many entrants you will get. Then after some time limit, post a corresponding poll thread and link both ways to/from it. When you word things ask people to vote for what they prefer, not what they think is best. :)

[Dark Shikari]

You're free to post threads and make polls. :)

I think at first post the challenge in one thread but not yet the poll, because you won't know how many entrants you will get. Then after some time limit, post a corresponding poll thread and link both ways to/from it. When you word things ask people to vote for what they prefer, not what they think is best. :)That was my idea--once there are a good number of entrants, or a deadline is reached, then post the poll.

[Terranigma]

The thing I see wrong with this, is that there may be entrants who post their filtered/restored clips, but not their scripts. Some folks will never share their scripts in fear of someone eventually evolving it and becoming a better encoder. :P
I know Didée won't mind, because he can always come up with something better, as there's not really a challenger for him. :P

[Dark Shikari]

The thing I see wrong with this, is that there may be entrants who post their filtered/restored clips, but not their scripts. Some folks will never share their scripts in fear of someone eventually evolving it and becoming a better encoder. :P
I know Didée won't mind, because he can always come up with something better, as there's not really a challenger for him. :PThen we can make a rule that you're required to share your script when its done :devil:

[Adub]

Sounds awesome. I for one will be willing to participate in the vote. I can't script worth crap, but I have eyes.

[hkazemi]

This is a great idea. There are already collections of test video clips available, but for the noise cleaning tests I sure some really stressful sources could be added/found. Links to the test collections I know about are all here http://forum.doom9.org/showthread.php?t=135034

[Dark Shikari]

This is a great idea. There are already collections of test video clips available, but for the noise cleaning tests I sure some really stressful sources could be added/found. Links to the test collections I know about are all here http://forum.doom9.org/showthread.php?t=135034For grainy clips, I have some really really stressful ones I could find, including 300 for obvious reasons.

[Die*wrek*show]

The thing I see wrong with this, is that there may be entrants who post their filtered/restored clips, but not their scripts. Some folks will never share their scripts in fear of someone eventually evolving it and becoming a better encoder. :P

you've hit upon a nagging problem that the vast majority of "creative types" usually succumb to at some point. if you make something, it gets spread around, someone steals it, then you get left with nothing while someone else takes the credit,fame,ect.... from what was originally yours.

i've been giving it alot of thought, for a long time now. here is one solution that i think could work. it won't work on this messageboard, because there would need to be a "system" set up to support how it operates. which i will briefly explain now.

we could call all scripts "freely available to share, modify, and use" under one condition. you don't try to hide the fact that you used someone else's script, either whole or in part.

however, i don't know the best way of going about that. it could be something simple like putting a
#start"Dark Shikari"
at the beginning of the "borrowed" code or a complex system of tagging and such. in the end, if there is an .html page with the statistics on it of whose code got used where, and what script, ect.... i don't think people will feel cheated. they would probably be proud to see their code widely used on so many good scripts. we could vote, also, on which scripts are best or which portions are best.

anyway, on topic-

i'd be down for judging this contest, but you don't want my scripts they're buggy.

[mikeytown2]

My 3 Cents:

1. Source restoration/postprocessing
All entries should use the same lossless codec (like Lagarith) for their submission.

2. Encoding/preprocessing
Their should be a small list of codecs for this test. Example codecs i would like to see: h.264, flv sorenson h263, mpeg 2, & xvid. We might want to take this one step further, by saying you can only use freeware encoders. For each codec, have a target bitrate.

For Both 1 & 2
There should be a minimum FPS or timeline (24hrs of cpu time) rule. Tests should be scientific: aka repeatable (no cold fusion).

[Zarxrax]

I like the idea of a contest. It's something fun for people to participate in, and it can lead to people learning new techniques to deal with common problems. Win-Win situation :)

[*.mp4 guy]

I'll participate, I actually had a script I almost posted in the 300 denoising thread, but by the time I got something together there really wasn't a point anymore, as the problem was pretty much solved.

[edit] I don't think we should use lossless codecs for any stage of the contest, they are just too space hungry. Xvid, with fox home entertainment matrix at q2-3, no qpel, should be fine quality wise, and much faster decoding then lossless.

[Chainmax]

I'd like to participate, but the only thing I ever did as a way of a general purpoise chain was an update to something that Didée suggested me. I can still post the general idea behind the filterchain as a base for others to build upon, though.

I do, however, have an extremely interesting problem sample to contribute to this though. It's the first couple of minutes of the first Simpsons episode. I never was able to make a completely clean version of it, so it would be interesting to see what others can make out of it. Let me know if you're interested.

[Southstorm]

I'm in. Sounds fun.

[45tripp]

There should be a minimum FPS or timeline (24hrs of cpu time) rule. Tests should be scientific: aka repeatable (no cold fusion).

I agree.
Of course there are various setups about,
plenty of ancient ones still perfectly active.
24h vs 24h can hugely differ.

ideally there should be a master encoder as controller/tracker
of speed performance.
someone to encode all scripts on a portion of the sample,
and clock the speeds.

I also find speed performance very important to overall assesment of quality anyway.
Is the perceptual gain worthy of the time penalty incured?
Is there a fair trade of cycles to quality?


2. Encoding/preprocessing
Their should be a small list of codecs for this test. Example codecs i would like to see: h.264, flv sorenson h263, mpeg 2, & xvid. We might want to take this one step further, by saying you can only use freeware encoders. For each codec, have a target bitrate.

a target bitrate is not enough.
one should completely lock down encoders and encoding settings,
to keep encoder influence out of it.

again, ideally, one might say,
all encodes should be done by a single master encoder.


I think at first post the challenge in one thread but not yet the poll, because you won't know how many entrants you will get. Then after some time limit, post a corresponding poll thread and link both ways to/from it.

i also think there should be a limit to final elligble entries.
try and get a mass vote on a reasonably sized set,
being spoilt for choice does no good.

Sounds awesome. I for one will be willing to participate in the vote. I can't script worth crap, but I have eyes.

nicely said :)

[Blue_MiSfit]

Sounds fun!!!

I would love to volunteer to be the encoder... but I think there are others more up and able to the task.

I would be delighted to participate in the voting!

~MiSfit

[mikeytown2]

ideally there should be a master encoder as controller/tracker of speed performance. someone to encode all scripts on a portion of the sample, and clock the speeds.
I don't think we should use lossless codecs for any stage of the contest, they are just too space hungry. Xvid, with fox home entertainment matrix at q2-3, no qpel, should be fine quality wise, and much faster decoding then lossless.IMHO should have 4 "master encoders": Fast/Slow Intel, Fast/Slow AMD. It should also be limited to 1 core (no MT scripts). IMHO Because we are tracking the avs performance, a low cpu codec should be used, until the video is to be posted to the net. My vote is for h.264 as the final format since flash supports it now; using MeGui with the CQ-ASP_Q2_equiv preset. This would only be needed for the finalists. IMHO Screen Shots would be sufficient for entrants.



I also find speed performance very important to overall assessment of quality anyway. Is the perceptual gain worthy of the time penalty incurred? Is there a fair trade of cycles to quality?Excellent point! We need some sort of metric so we can create a ratio. Something like FPS/PSNR.
[http://en.wikipedia.org/wiki/Video_quality] [http://compression.ru/video/quality_measure/video_measurement_tool_en.html]


one should completely lock down encoders and encoding settings, to keep encoder influence out of it.Good Point, by doing this we are reducing feature/test creep. Let's keep it simple, then after this is done we could do different tests; but for right now we should stay focused on one goal.

[Dark Shikari]

Seriously, we don't need it to be over-organized. Master encoders/similar are silly.

[Adub]

Why not? If we want it to be a fair, legitimate contest, then why not make it organized. You say over organized. What are the specifications for your ideal implementation of the contest?

[Dark Shikari]

Why not? If we want it to be a fair, legitimate contest, then why not make it organized. You say over organized. What are the specifications for your ideal implementation of the contest?I was thinking something more along the lines of a "for fun" contest; no real prize, nothing that official. If someone manages to do better than anyone else, no matter how they did it, as long as its repeatable, we want to know about it.

[Adub]

Makes sense. Personally I never thought there was supposed to be a "real prize". Just the knowledge that you were crowned king! At least, temporarily.

[45tripp]

Seriously, we don't need it to be over-organized. Master encoders/similar are silly.

it's certainly not silly.
over-organized, maybe, i did say 'ideally'.

i also think the less organized it is,
the less likely it is to break out of the confines of the small group,
already at ease with tracking through such loose discussions
and deconstructing advanced scripts.

less likely to make it to a mass poll.

I was thinking something more along the lines of a "for fun" contest;

'organized' doesn't have to signal the death of 'fun'
and,
'too loose' can be just as disruptive to good fun as 'too organized'
but i'm sure you'll find a fun format for what you have in mind soon enough,
imo paramaters are needed though,
and you seem to be asking opinions about such in this preliminary thread. are you not?

IMHO Screen Shots would be sufficient for entrants.
imo not.
take as an example the 'dancing grain' removal objective.


Excellent point! We need some sort of metric so we can create a ratio. Something like FPS/PSNR.

I'm all for an evaluation of efficiency.
efficiency interests me.
it's doesn't seem to fit the bill here though.
still someone could always take the initiative and add it as a side order.

I would love to volunteer to be the encoder... but I think there are others more up and able to the task.

'encoder', would be the donkey work,
not much to do with any special abilities,
one would just need to provide processing power and time (lots)
I salute your eagerness to volunteer :)

Just the knowledge that you were crowned king! At least, temporarily.

so it's all down to ego? :devil:

[*.mp4 guy]

Dark Shikari is right, being overzealous in the pre-planning stage is a good way to kill this idea before anything happens. Furthermore the more restrictive the rules are, and the more dificult it is to follow them (master encoders, downloading huge lossless files, etc) the less people will bother participating. That said, it is very dificult to get neer lossless quality out of x264, and atleast fo sd content I think Xvid would be faster, easier, and just as effective as far as the contest is concerned (and yes, better looking then cq2_asp profile, along with being bigger).

Imo the following are all that need to be decided.

-output format, bitrate, resolution and encoding options restrictions.

-time restrictions, and how this will coexist with different speed systems.

-sources to be used, and the specific goals of the competition on each source.

-organizational and logistical issues (where to upload clips, who to notify, when/where/how to vote).

I beleive we are all aware of how amazingly time-consuming and inefficient committies are. Since this was Dark Shikari's idea, I beleive he should have the last word on the issues being decided, to insure that decisions are actually reached at all. There will be things that we will all probably think are non-ideal, but if no one allows themselves to make compromises, nothing will happen.

[45tripp]

Dark Shikari is right, being overzealous in the pre-planning stage is a good way to kill this idea before anything happens.

kill is a strong word,
i disagree.

Imo the following are all that need to be decided...

quite a few things in the end...

I think Xvid would be faster, easier, and just as effective

i'd agree
but i can see reason to use avc


I beleive we are all aware of how amazingly time-consuming and inefficient committies are. Since this was Dark Shikari's idea, I beleive he should have the last word on the issues being decided,

I agree.

[Adub]

I agree.
I can agree with that. Skip all of the possibly bureaucratic s**t and let the main man himself state the rules.



Quote:
Originally Posted by Merlin7777
Just the knowledge that you were crowned king! At least, temporarily.
so it's all down to ego?
Well, do you have a better idea?

[unskinnyboy]

Since we are talking about time restrictions, here is how I am reading that - everything else resulting in a tie regarding the output (visual quality, metrics etc), user A who does the encode in 2 hours wins and user B who finishes the encode in 2 hours and 1 minute loses. Is that right?

[Soulhunter]

I'd be in! :]

[mikeytown2]

@Immersion: I agree that this contest would have a lot of ego involved. There is no way around this. :devil:

@*.mp4 guy: Great Points. Xvid would be better. Also second that Dark Shikari has the final say.

@Merlin7777: Just posting ideas. I do agree with your thinking though.


We could create a google page [http://pages.google.com/-/about.html] to post results from the forum, or put it up on the avs wiki.

Limiting submissions would be a bad idea. The master encoder idea would be needed only for the last part. After all the scripts are in, we would test them for speed. To test every script version (1.0 vs 1.0.1) for speed would be nuts. The author of the script should estimate the speed of his script if possible. Failing a script because it takes too long isn't the brightest idea; MCBob + NNEDI is a prime example of a good but slow process/script.

I would be willing to run the speed tests for the finalists. I could cover the fast intel and the slow amd. the specs for my slow amd is a 850 oc-d to 933 (so i could run the bus at 133mhz), w/ a Radeon 64 DDR VIVO. I also have access to an amd 3000+.

[Die*wrek*show]

we need a kickstart. let's get something going guys, even if it's something small and simple. it's better to try and fail than never to try at all.

[Southstorm]

Has this idea fizzled out?

[MfA]

Regardless of the contest we could really use a common set of sequences and benchmark scripts for testing of deinterlacing, upscaling and denoising. There is simply nothing of the sort out there, everyone just does things completely ad-hoc.

For encoding I'm not so sure entirely new sequences are necessary though, since there are already a couple sequences out there which are used pretty universally by ISO/ITU and the academia. If you are missing something in the existing ones you could add some, but incorporating the old standby's (foreman, flower garden, football etc.) is desirable.

PS. not allowing discussing best scoring filters/encodings (by objective benchmarking or subjective voting) would be an overzealous application of rule 12 ... besides, it's been done many times already.

[cogman]

Just reading through, It did sound like a good Idea though.

I think that we should at least keep track of encoding time of posted scripts just to make sure we dont have a 3 week encode of a 3 minute video.

But heck, it is just for fun, so all that is needed is the origional, the script, and the encoder to post what he used to encode the file, IE x264 rev 800 with an Intel core 2 duo OC 3.2GHZ. After that the results can easily be disputed if someone feels so inclined. (sore losers)

I don't like the idea of putting any restrictions on what filters can or cant be used.

The only other thing I would suggest is some kind of time/quality winner, it would be nice to have a winner of both rediculusly long encodes and fast good looking encodes. The problem, of course, is how do you measure that?

[DarkT]

Too much talking, too little doing... I say, throw in a source mpeg which is like 5 minutes long, nominate 3 judges, make clear the rules, and go go go...

Ain't rocket science, ya know? And if it all goes down in flames, good, make v2 with the lessons learned.

[audyovydeo]

[QUOTE=DarkT;1120733]Too much talking, too little doing... I say, throw in a source mpeg which is like 5 minutes long, nominate 3 judges, make clear the rules, and go go go...
/QUOTE]


Agree, also count me into any contest.

I propose to add a rule : make the shortest possible script to achieve the given goals. After all, we all hate bloatware, no ?

cheers
audyovydeo