Getting the most out of Avisynth with my new "overkill" machine

[Gargalash]

Hello,
I have bought a shiny new machine a couple weeks ago. The other one was getting painful to use. I use Avisynth to process video to lossless before using this first output to encode to various final formats (wmv, flv, x264 avi, mpeg).

The new computer has way more power than the other and I am asking for help to learn what I can do to harvest the full power.

Here are some specs:
- 12gig of ram
- i7 930 clocked at 4ghz, hyper threading off (to reduce temperature, still 4 threads)
- GPU: Gigabyte Super Over Clock gv-n470so-13l with Nvidia GeForce GTX 470.
- System on 2x SSD Raid 0

Here are some questions on "correct" avisynth usage with this:
- 64 bit world: is there a 64 bits avisynth version and filters that make good use of the the 64bits environment?
- RAM: in the avs script, how should I set the memory usage and can it make a difference in performance?
- Threading filters: in the world of high quality deinterlacer and denoiser, is there anything that can be MT'ed to use the 4 threads? The goal here is to have a script that puts the CPU on 100% use all the time (no way...). Other suggestions on filters that make full use of the CPU?
- Using the GPU: I work with interlaced content sometimes and I plan on using DGDecNV to deinterlace. Otherwise, I would like to know what other processing can be done with the GPU (denoising is one of them).
- Any other idea?

To give some hints on what filters I like to use, here is a list of stuff I use regularly (feel free to suggest others):
- TGMC for deinterlacing (multithreadable?)
- Limited sharpen faster (ok with MT, 4 threads)
- awarpsharp2 (ok with MT, 4 threads)
- Dehalo_alpha (ok with MT, 4 threads)
- antialising (ok with MT, 4 threads)

A particular case is TemporalDegrain_beta(). Here is the line I use:
MT("TemporalDegrain_beta(gpu=true, degrain=1, ov=8, hq=0, SAD1=100, SAD2=100)",4,2)
I get 19fps with progressive PAL 720x576 content. Is this a number that I should expect? I actually get weird splitting in the image with large bogus areas and as I am typing this, I notice the filter just doesn't produce any effect. So, I guess I'm looking for a Denoiser ;).

Any suggestion will be greatly welcomed and appreciated!
Thanks!

[Didée]

- 64 bit world: is there a 64 bits avisynth version and filters that make good use of the the 64bits environment?
Sure, here (http://forum.doom9.org/showthread.php?t=152800). It works like a charm, as long as doesn't break. (The april version is said to have borked resizer code [though it usually works for me], the October version has the resizers fixed, but other things are broken (some have reported plugin problems, and I'm pretty sure that TemporalSoften is broken, which in turn will break many other scripts.)

A good bunch of the "major" plugins has been ported, but there's still plenty plugins not yet available for x64.


- RAM: in the avs script, how should I set the memory usage and can it make a difference in performance?
For normal scripts, no need to do anything. For very ressource-hungry scripts, use SetMemoryMax().
Keep in mind that 32bit-Avisynth is limited by the 2GB process limit.


- Threading filters: in the world of high quality deinterlacer and denoiser, is there anything that can be MT'ed to use the 4 threads? The goal here is to have a script that puts the CPU on 100% use all the time (no way...). Other suggestions on filters that make full use of the CPU?
Basically, almost anything can be multithreaded ... except for a few filters that are just not compatible. Any GPU filter, outdated filters like e.g. Dust. There might be other filters not compatible with multithreading, but offhand I don't know any.


- Using the GPU: I work with interlaced content sometimes and I plan on using DGDecNV to deinterlace.
That's an either-or story. You can either use NV deinterlacing, or you can have quality deinterlacing. Both are mutually exclusive.


- TGMC for deinterlacing (multithreadable?)
Yes, it's multithreadable.


A particular case is TemporalDegrain_beta(). Here is the line I use:
MT("TemporalDegrain_beta(gpu=true, degrain=1, ov=8, hq=0, SAD1=100, SAD2=100)",4,2)
I get 19fps with progressive PAL 720x576 content. Is this a number that I should expect? I actually get weird splitting in the image with large bogus areas and as I am typing this, I notice the filter just doesn't produce any effect.
You cannot use multithreading with GPU-filters. When going to multithread TemporalDegrain, you have to use GPU=false.

(Just for fun, try a plain MT("""FFT3dGPU(sigma=20,bt=4)""",4,4) to see what happens. Bad surprise...)

Also, when MT()-ing TemporalDegrain, use bigger overlap. My rule-of-thumb is that MT's overlap should be at least as big as the blocksize used by MVTools.

-----

Unrelated, but still ...
- System on 2x SSD Raid 0
I'm not a fan of SSD-Raids. One single SSD is already playing "in another universe" compared to a traditional HDD. The drawback of SSD-Raid is that TRIM is not possible in a RAID. You start with a fast RAID, but it will become slower & slower over time, and the SSDs will wear-out much more fast.

[Dogway]

Try out tnlmeans, is MT'able but I bet you wont see again your system as fast as before, try this:

MT("tnlmeans(ax=8,ay=8,az=6,sx=2,sy=2,bx=0,by=0,h=1.0,a=1000.0,sse=true)",4,4)

[Gargalash]

Thanks for your answers!

Didée:
I have downloaded and installed the x64 version and I gathered all the filters. I will go through them and see what could be used in my workflow. Thanks for pointing out!

You can either use NV deinterlacing, or you can have quality deinterlacing. Both are mutually exclusive.
I use TGMC to deinterlace, and I have tested with QTGMC. Is there anything superior (ideally in the 64 bit world)? My sources are usually NTSC or PAL DVDs.

About the SSD, I have made a few research and there is a firmware for the drives that makes TRIM work with a RAID0 on the OCZ vertex 2 I have. Same for Intel's, but I don't know which one.

Dogway:
I have tested your line, but I don't see the point of trying out stuff that obviously hang the script for 30 seconds.
But I have discovered a sweet denoiser when using the default parameter, i.e.: MT("tnlmeans"),4,4). Thanks!

Gargalsh

[mastrboy]

not to crash your party, but in my experience related to SSD trim, it does not matter what firmware the disks have, but if the raid controller exposes the availabilty of this option to the os or not. There are some third party tools to work around this though, like wiper: http://www.ocztechnologyforum.com/forum/showthread.php?57434-windows-with-Wiper-0525. So you should look into firmware and drivers updates to your raid controller also...

[Didée]

Dogway's suggestion probably was just to show how easily your new overkill machine can be brought to its knees, and worse. The funny thing with TNLMeans is that by minor adjustment of 2 or three numbers, you can switch between seconds-per-frame, minutes-per-frame, hours-per-frame, ... no matter what system.

You're sure about TRIM available through RAID? Some link for reference? For all I know it's still a no-go ...
(If it would've become possible, after all the time without, there would've been BIG hype about it.)

[burfadel]

You can make use of your system by encoding two things at once :) you can set them up through staxrip. If doing only two encodes, just start one encode, set up the next one and instead of sending to current instance, just add it to the list, remove the original entry, and start that encode too.

If doing say, 10 encodes for example (can be any number, doesn't matter). Set everything up in the encode list, close, then open two instances of staxrip. In the first one, untick the second half of the boxes in the job queue, and in the second instance of staxrip, untick the first half. You can then start both encodes, and each list will stay valid for each instance of staxrip. Just leave a few minutes between starting each instance, as depending on how things are set up etc, the start of the encode where audio tracks are extracted, the file is indexed etc is the heaviest on the disk.

This method works and the lists do stay valid for both the instances of staxrip. I know this because I've encoded like this quite often recently. I get a good 100 percent cpu use when its actually encoding the video, and since the audio part of the encode etc occurs (most likely) when the other instance is still encoding the video, you still get 100 percent or close to cpu use. Since these are set to low priority by default, you don't really notice it affecting the performance of the computer, and of course you can still play media etc no issues. Last night I was recording two shows off tv (I have a dual digital tv tuner card + 1 single digital tv tuner card), encoding using two instances of staxrip as just mentioned, downloading something, and had no problems playing Mafia II's new DLC Joe's Adventures... which is actually unsurprising as thats the purpose of prioritisation in Windows.

[ianken]

Unrelated, but still ...

I'm not a fan of SSD-Raids. One single SSD is already playing "in another universe" compared to a traditional HDD. The drawback of SSD-Raid is that TRIM is not possible in a RAID. You start with a fast RAID, but it will become slower & slower over time, and the SSDs will wear-out much more fast.

Even with full read-write cycles 24/7 you're looking at years for an SSD cell to fail.

If the SSD is an indilinx based unit like those from OCZ and others they will garbage collect while idle.

RAIDing SSD for perf doesn't seem to gain much. I did it for capacity.

Back on topic: I'm also in the camp of trying to get the most perf out of AVS as I can. On my I7 box even with a big script, using MT, it rarely loads the system more than 20%. It's not disk bound either.

I assume it's blocking at various filters as the pipeline stalls waiting for sample or waiting for one filter in the chain to get done. Even so I'd expect at least one core to be loaded down.

Anyway, is there a guide/wiki/best practices document on how to optimize your scripts?

[Gargalash]

You're sure about TRIM available through RAID? Some link for reference? For all I know it's still a no-go ...
(If it would've become possible, after all the time without, there would've been BIG hype about it.)

Not sure anymore, I dug further, and the info that came was wrong. Websites not verifying their sources... . still no trim supporting raid...

[tormento]

You can make use of your system by encoding two things at once :)
I tried to make multiple encodes with MEGUI_X64 and it did not speeded up things a lot. I suppose there must be some obscure x264 parameter not to make it piss off other x264 threads.

[ajp_anton]

Even with full read-write cycles 24/7 you're looking at years for an SSD cell to fail.
I don't think it's so much about the drive failing as it's about it getting slower, making your Raid0 no faster than a single drive.

[trevaaar]

I've basically given up on MT in x64 Avisynth, it always crashes at a seemingly random point. If I need extra performance I just run a few different encodes at once.

[TheRyuu]

You shouldn't disable HT. You can see performance gains of up to 25% (especially in things like x264 which actually thread well).