Doom9's Forum - View Single Post

Mini-Me · 13th June 2011, 06:06

Are you still having problems running the scripts you want with multithreading, without crashing?

I don't have an elegant solution, but there's a brute force workaround that will allow you to use all of your cores and potentially all of your memory. Short version: Split your result into pieces, concurrently render the different pieces to lossless intermediate files, then concatenate them afterwards and encode.

Long version:

First, do not return the whole result at the end of your script. Instead, split it into N sections with Trim (or stickboy's excellent Trim2), where N is the number of cores on your system (or hardware threads, if you want to try hyperthreading). Write a line for returning each section, e.g. (for four sections in a clip ~200000 frames long):

Code:

input = Avisource("TheFellowshipOfTheLanczosRings.avi")

result = MCFilterOfObsceneSystemPunishment(input)

return result.Trim2(0, 50000)
#return result.Trim2(50000, 100000)
#return result.Trim2(100000, 150000)
#return result.Trim2(150000)

Second, notice that I commented all but the first section. Load the first section in Virtualdub, and render the section to a lossless file (Fast Recompress, Huffyuv). While that's running, comment out the first return line, uncomment the second one, and resave the .avs script. (Changing the script will not affect the render that's currently running, because Avisynth parses the script text when it loads.) Load the modified script in Virtualdub again, and render the second part to another lossless file.

Lather, rinse, and repeat, and you can have as many videos rendering simultaneously as your hardware can handle (mainly your memory and disk performance). Once all of the sections are done rendering a few hours (days?

) later, concatenate them in another lightweight Avisynth script (use a single '+' to avoid audio glitches, instead of "++"), and directly encode the result to your output format. The downside of this method is that it requires a few minutes more work than it would take to load/render a single script with an actual working MT build, and if your final encoder is really fast, writing to lossless intermediates could seem like wasted time. The upside is, depending on your format, encoding to lossless intermediates before encoding the final result might be faster in the long run anyway.

Note that Windows will give each Virtualdub/Avisynth instance as much memory as it can give to a single process, which means that you can ultimately use a whole lot more than 4 gigs at once. The different processes might duplicate a little bit of work at the seams, but this should be insignificant unless you're using temporal filters with a radius of thousands and thousands of frames.

I'm only aware of one "gotcha," and it applies to a pretty uncommon use case: If you're using a ScriptClip function that assumes linear access and carries over some result from previous frames, each section will start "clean" without any information carried over from frames before the start of the section.

I've been using this trick a lot recently to get multiprocessing with the official Avisynth 2.6 alpha releases, and it's nice to combine the stability and "correctness" of single-threaded Avisynth with the speed of MT builds.

13th June 2011, 06:06	#17 \| Link
Mini-Me Registered User Join Date: Jan 2011 Posts: 121	Are you still having problems running the scripts you want with multithreading, without crashing? I don't have an elegant solution, but there's a brute force workaround that will allow you to use all of your cores and potentially all of your memory. Short version: Split your result into pieces, concurrently render the different pieces to lossless intermediate files, then concatenate them afterwards and encode. Long version: First, do not return the whole result at the end of your script. Instead, split it into N sections with Trim (or stickboy's excellent Trim2), where N is the number of cores on your system (or hardware threads, if you want to try hyperthreading). Write a line for returning each section, e.g. (for four sections in a clip ~200000 frames long): Code: input = Avisource("TheFellowshipOfTheLanczosRings.avi") result = MCFilterOfObsceneSystemPunishment(input) return result.Trim2(0, 50000) #return result.Trim2(50000, 100000) #return result.Trim2(100000, 150000) #return result.Trim2(150000) Second, notice that I commented all but the first section. Load the first section in Virtualdub, and render the section to a lossless file (Fast Recompress, Huffyuv). While that's running, comment out the first return line, uncomment the second one, and resave the .avs script. (Changing the script will not affect the render that's currently running, because Avisynth parses the script text when it loads.) Load the modified script in Virtualdub again, and render the second part to another lossless file. Lather, rinse, and repeat, and you can have as many videos rendering simultaneously as your hardware can handle (mainly your memory and disk performance). Once all of the sections are done rendering a few hours (days? ) later, concatenate them in another lightweight Avisynth script (use a single '+' to avoid audio glitches, instead of "++"), and directly encode the result to your output format. The downside of this method is that it requires a few minutes more work than it would take to load/render a single script with an actual working MT build, and if your final encoder is really fast, writing to lossless intermediates could seem like wasted time. The upside is, depending on your format, encoding to lossless intermediates before encoding the final result might be faster in the long run anyway. Note that Windows will give each Virtualdub/Avisynth instance as much memory as it can give to a single process, which means that you can ultimately use a whole lot more than 4 gigs at once. The different processes might duplicate a little bit of work at the seams, but this should be insignificant unless you're using temporal filters with a radius of thousands and thousands of frames. I'm only aware of one "gotcha," and it applies to a pretty uncommon use case: If you're using a ScriptClip function that assumes linear access and carries over some result from previous frames, each section will start "clean" without any information carried over from frames before the start of the section. I've been using this trick a lot recently to get multiprocessing with the official Avisynth 2.6 alpha releases, and it's nice to combine the stability and "correctness" of single-threaded Avisynth with the speed of MT builds. Last edited by Mini-Me; 13th June 2011 at 08:08.