compression filter for speeding up denoisers?

Nematocyst · 5th March 2005, 03:32

I'm assuming that if you had a sufficiently slow denoising filter (or any slow filter for that matter), that you can get quicker results by activating the slow filters only once. You could do this manually by encoding a single pass to something like huffyuv or picvideo, and then doing a 2-pass encoding on that intermediate file.

Would it be possible to achieve that same result during the first pass by using a filter? You'd also need a smart enough script to detect the 2nd pass so you could use the intermediate file. But it'd definately be neat if you could do this in just 2 passes.

Mug Funky · 5th March 2005, 07:43

it depends on what kind of statistics the codec gathers in the first pass.

disabling a filter in the first pass will mean different stats are gathered, leading to either poor filesize prediction (under or oversized encodes), poor visual quality, or both.

i suppose if a codec gathers very simple stats, then you could get away with it. but i don't think anyone would recommend it.

Nematocyst · 5th March 2005, 15:16

Ah. I thought the frames themselves were all that mattered. I didn't realize the filter chain used to get those frames made any difference to the codec.

Leak · 6th March 2005, 00:33

Quote:

Originally posted by Nematocyst
Ah. I thought the frames themselves were all that mattered. I didn't realize the filter chain used to get those frames made any difference to the codec.

Ummm... it doesn't. All a codec can work with is the frames you feed it - it doesn't know where they came from.

@Didee: unless I totally misunderstood him, he meant feeding the frames produced by AviSynth during the first pass into both a lossy and a lossless codec, and then using the lossless file as the input in the second pass.

@Nematocyst: if there were an AviSynth filter that wrote AVI files this could be easily done, but nobody has written one yet...

You can probably use ImageWriter to write out a PNG file for every frame, and use ImageReader on the 2nd pass to treat those like a lossless video.

np: AGF/Delay - Causing A Taifun (Explode)

Nematocyst · 6th March 2005, 01:35

Quote:

unless I totally misunderstood him, he meant feeding the frames produced by AviSynth during the first pass into both a lossy and a lossless codec, and then using the lossless file as the input in the second pass.

Yes, that's what I meant. Thanks for both understanding and clarifying.

Yeah, I figured there was no such filter. I was just sort of brainstorming out loud. And it occured to me I picked the wrong forum, and should have chosen "Avisynth Development."

Didée · 6th March 2005, 03:49

@ Leak: Thanks for addressing me - but I didn't say a single word ...

However that's what I understood, too: feed a script to the encoder, which during the first run writes both a lossless compressed file and the XviD stats file. There was a thread about a similar idea, a while back.
The new addition here is that the script shall be smart enough to automagically detect during the 2nd run that the lossless file is already there.

I like the idea of smart processing. Together with speech reckognition, I'd just quickly tell my rig "Please make a rip of my two Spiderman DVDs. Mpeg-4, max quality to fill a DVD!", leave out for a party, and after returning just label the perfecly burned disc.

Leak · 6th March 2005, 13:39

Quote:

Originally posted by Didée
@ Leak: Thanks for addressing me - but I didn't say a single word ...

Whoops... sorry 'bout that - I thought that "Mug Funky" was spelled "Didée"...

np: µ-Ziq - Tango n' Vectif (Tango n' Vectif)

tritical · 7th March 2005, 07:20

I hacked this up real quick using the avs2avi source code for reference... TWriteAvi. Basically it will pop up a dialog box on load so you can choose the compressor to use and configure it. So you could insert this filter at the very end of your avs script. Then open the script in vdub, set vdub up to encode the first pass for your two pass codec and TWriteAvi to encode the lossless version. Then on the second pass use the lossless version as input. Of course, its not automatic and you have to be there to choose the codec. However, if someone wants too, I think some sort of automated ability could be added easily enough.

IanB · 7th March 2005, 08:48

Sounds like we want a disk based cache here.

Impl :- Each GetFrame()/GetAudio() check the cache diskfile to see if we have that video frame/audio samples, if so return it, if not generate it, store a copy and return it. It could even be be persistant across opens. I like it.

IanB

Zep · 11th February 2007, 17:07

Quote:

Originally Posted by tritical

I hacked this up real quick using the avs2avi source code for reference... TWriteAvi. Basically it will pop up a dialog box on load so you can choose the compressor to use and configure it. So you could insert this filter at the very end of your avs script. Then open the script in vdub, set vdub up to encode the first pass for your two pass codec and TWriteAvi to encode the lossless version. Then on the second pass use the lossless version as input. Of course, its not automatic and you have to be there to choose the codec. However, if someone wants too, I think some sort of automated ability could be added easily enough.

just gave your filter a go and it works well when not using multithreads. it can not be used with tsp's MT SetMTMode() libs.

Example: super basic test script

SetMTMode(5)
mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0) #source input is 1080i
SetMTMode(2)
Lanczos4Resize(624,352)
SetMTMode(5)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)

I get a FILE ALREADY EXISTS error. My guess of course the first thread makes the file and then the second thread tries to and BOOM the file was already made by the first thread.

I thought TSP said SetMTMode(5) turned off MT but i guess it doesn't. I did a quick search and saw at one time there was a SetMTMode(0) to turn it off totally but that does not work anymore.

it is a shame because with MT on the Lanczos4Resize(624,352) and other filters i would add in, a much more CPU intensive script really benefits from MT but to get TWriteAvi to work i can't use MT and thus my CPU usage drop from 100% to 60 and the FPS rate drops accordingly as well. So the speed i was trying to gain by doing all the filters just 1 time in pass one is lost since the time it takes to do pass one goes way up.

I guess i need to talk to tsp and see if there is a way to totally shut off MT. My guess is he will say do not use setMTmode() but instead use MT() just on the filters you want directly I will plan to give that a try next chance i get to experiment with this stuff

Thanks!

5th March 2005, 03:32	#1 \| Link
Nematocyst Registered User Join Date: May 2004 Location: Albuquerque, NM Posts: 78	compression filter for speeding up denoisers? I'm assuming that if you had a sufficiently slow denoising filter (or any slow filter for that matter), that you can get quicker results by activating the slow filters only once. You could do this manually by encoding a single pass to something like huffyuv or picvideo, and then doing a 2-pass encoding on that intermediate file. Would it be possible to achieve that same result during the first pass by using a filter? You'd also need a smart enough script to detect the 2nd pass so you could use the intermediate file. But it'd definately be neat if you could do this in just 2 passes.

5th March 2005, 07:43	#2 \| Link
Mug Funky interlace this! Join Date: Jun 2003 Location: i'm in ur transfers, addin noise Posts: 4,555	it depends on what kind of statistics the codec gathers in the first pass. disabling a filter in the first pass will mean different stats are gathered, leading to either poor filesize prediction (under or oversized encodes), poor visual quality, or both. i suppose if a codec gathers very simple stats, then you could get away with it. but i don't think anyone would recommend it. __________________ sucking the life out of your videos since 2004

6th March 2005, 03:49	#6 \| Link
Didée Registered User Join Date: Apr 2002 Location: Germany Posts: 5,391	@ Leak: Thanks for addressing me - but I didn't say a single word ... However that's what I understood, too: feed a script to the encoder, which during the first run writes both a lossless compressed file and the XviD stats file. There was a thread about a similar idea, a while back. The new addition here is that the script shall be smart enough to automagically detect during the 2nd run that the lossless file is already there. I like the idea of smart processing. Together with speech reckognition, I'd just quickly tell my rig "Please make a rip of my two Spiderman DVDs. Mpeg-4, max quality to fill a DVD!", leave out for a party, and after returning just label the perfecly burned disc. __________________ - We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 6th March 2005 at 04:04.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

5th March 2005, 15:16	#3 \| Link
Nematocyst Registered User Join Date: May 2004 Location: Albuquerque, NM Posts: 78	Ah. I thought the frames themselves were all that mattered. I didn't realize the filter chain used to get those frames made any difference to the codec.

7th March 2005, 07:20	#8 \| Link
tritical Registered User Join Date: Dec 2003 Location: MO, US Posts: 999	I hacked this up real quick using the avs2avi source code for reference... TWriteAvi. Basically it will pop up a dialog box on load so you can choose the compressor to use and configure it. So you could insert this filter at the very end of your avs script. Then open the script in vdub, set vdub up to encode the first pass for your two pass codec and TWriteAvi to encode the lossless version. Then on the second pass use the lossless version as input. Of course, its not automatic and you have to be there to choose the codec. However, if someone wants too, I think some sort of automated ability could be added easily enough.

7th March 2005, 08:48	#9 \| Link
IanB Avisynth Developer Join Date: Jan 2003 Location: Melbourne, Australia Posts: 3,167	Sounds like we want a disk based cache here. Impl :- Each GetFrame()/GetAudio() check the cache diskfile to see if we have that video frame/audio samples, if so return it, if not generate it, store a copy and return it. It could even be be persistant across opens. I like it. IanB