compression filter for speeding up denoisers?

[Nematocyst]

I'm assuming that if you had a sufficiently slow denoising filter (or any slow filter for that matter), that you can get quicker results by activating the slow filters only once. You could do this manually by encoding a single pass to something like huffyuv or picvideo, and then doing a 2-pass encoding on that intermediate file.

Would it be possible to achieve that same result during the first pass by using a filter? You'd also need a smart enough script to detect the 2nd pass so you could use the intermediate file. But it'd definately be neat if you could do this in just 2 passes.

[Mug Funky]

it depends on what kind of statistics the codec gathers in the first pass.

disabling a filter in the first pass will mean different stats are gathered, leading to either poor filesize prediction (under or oversized encodes), poor visual quality, or both.

i suppose if a codec gathers very simple stats, then you could get away with it. but i don't think anyone would recommend it.

[Nematocyst]

Ah. I thought the frames themselves were all that mattered. I didn't realize the filter chain used to get those frames made any difference to the codec.

[Leak]

Originally posted by Nematocyst
Ah. I thought the frames themselves were all that mattered. I didn't realize the filter chain used to get those frames made any difference to the codec.

Ummm... it doesn't. All a codec can work with is the frames you feed it - it doesn't know where they came from.

@Didee: unless I totally misunderstood him, he meant feeding the frames produced by AviSynth during the first pass into both a lossy and a lossless codec, and then using the lossless file as the input in the second pass.

@Nematocyst: if there were an AviSynth filter that wrote AVI files this could be easily done, but nobody has written one yet... :(

You can probably use ImageWriter to write out a PNG file for every frame, and use ImageReader on the 2nd pass to treat those like a lossless video.

np: AGF/Delay - Causing A Taifun (Explode)

[Nematocyst]

unless I totally misunderstood him, he meant feeding the frames produced by AviSynth during the first pass into both a lossy and a lossless codec, and then using the lossless file as the input in the second pass.

Yes, that's what I meant. Thanks for both understanding and clarifying.

Yeah, I figured there was no such filter. I was just sort of brainstorming out loud. And it occured to me I picked the wrong forum, and should have chosen "Avisynth Development."

[Didée]

@ Leak: Thanks for addressing me - but I didn't say a single word ... ;)

However that's what I understood, too: feed a script to the encoder, which during the first run writes both a lossless compressed file and the XviD stats file. There was a thread about a similar idea, a while back.
The new addition here is that the script shall be smart enough to automagically detect during the 2nd run that the lossless file is already there.

I like the idea of smart processing. Together with speech reckognition, I'd just quickly tell my rig "Please make a rip of my two Spiderman DVDs. Mpeg-4, max quality to fill a DVD!", leave out for a party, and after returning just label the perfecly burned disc.

[Leak]

Originally posted by Didée
@ Leak: Thanks for addressing me - but I didn't say a single word ... ;)

Whoops... sorry 'bout that - I thought that "Mug Funky" was spelled "Didée"... :p

np: µ-Ziq - Tango n' Vectif (Tango n' Vectif)

[tritical]

I hacked this up real quick using the avs2avi source code for reference... TWriteAvi (http://bengal.missouri.edu/~kes25c/TWriteAvi.zip). Basically it will pop up a dialog box on load so you can choose the compressor to use and configure it. So you could insert this filter at the very end of your avs script. Then open the script in vdub, set vdub up to encode the first pass for your two pass codec and TWriteAvi to encode the lossless version. Then on the second pass use the lossless version as input. Of course, its not automatic and you have to be there to choose the codec. However, if someone wants too, I think some sort of automated ability could be added easily enough.

[IanB]

Sounds like we want a disk based cache here.

Impl :- Each GetFrame()/GetAudio() check the cache diskfile to see if we have that video frame/audio samples, if so return it, if not generate it, store a copy and return it. It could even be be persistant across opens. I like it.:cool:

IanB

[Zep]

I hacked this up real quick using the avs2avi source code for reference... TWriteAvi (http://bengal.missouri.edu/~kes25c/TWriteAvi.zip). Basically it will pop up a dialog box on load so you can choose the compressor to use and configure it. So you could insert this filter at the very end of your avs script. Then open the script in vdub, set vdub up to encode the first pass for your two pass codec and TWriteAvi to encode the lossless version. Then on the second pass use the lossless version as input. Of course, its not automatic and you have to be there to choose the codec. However, if someone wants too, I think some sort of automated ability could be added easily enough.


just gave your filter a go and it works well when not using multithreads. it can not be used with tsp's MT SetMTMode() libs.

Example: super basic test script

SetMTMode(5)
mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0) #source input is 1080i
SetMTMode(2)
Lanczos4Resize(624,352)
SetMTMode(5)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)

I get a FILE ALREADY EXISTS error. My guess of course the first thread makes the file and then the second thread tries to and BOOM the file was already made by the first thread.

I thought TSP said SetMTMode(5) turned off MT but i guess it doesn't. I did a quick search and saw at one time there was a SetMTMode(0) to turn it off totally but that does not work anymore.

it is a shame because with MT on the Lanczos4Resize(624,352) and other filters i would add in, a much more CPU intensive script really benefits from MT but to get TWriteAvi to work i can't use MT and thus my CPU usage drop from 100% to 60 and the FPS rate drops accordingly as well. So the speed i was trying to gain by doing all the filters just 1 time in pass one is lost since the time it takes to do pass one goes way up.

I guess i need to talk to tsp and see if there is a way to totally shut off MT. My guess is he will say do not use setMTmode() but instead use MT() just on the filters you want directly I will plan to give that a try next chance i get to experiment with this stuff :)


Thanks!