Help needed with optimising ImageSource

[colinhunt]

System: Windows 7 Ultimate 64-bit, Core i7 860 @ 2.80GHz, 8GB RAM
Installed: Avisynth 2.58 32-bit, latest 32-bit x264.exe

Input: A sequence of 1920x1080x24b TIFF images, approx. 12MB each
Output: 1920x1080x24b video at 23.976fps

test.avs:
ImageSource("W:\TEMP\b194out\yard.%7.7d.tif",0, 760).AssumeFPS("ntsc_film")

encode.bat:
set input=D:\test.avs
set output=D:\imgseqtest_v.mkv
set ref=5
set bframes=5
set x264Path=D:\x264.exe
%x264Path% %input% --preset fast --ref %ref% --bframes %bframes% --output %output%

c/p from CLI during encoding:
avs [warning]: converting input clip to YV12
avs [info]: 1920x1080p 0:0 @ 24000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 5.0
[83.6%] 636/761 frames, 0.98 fps, 3343.68 kb/s, eta 0:02:07

Problem: low encoding speed! Less than 1fps while x264.exe's CPU load hovers around 5-9%.

I'd really appreciate any and all help with this. What can I do to speed up the encoding?

[wonkey_monkey]

Input: A sequence of 1920x1080x24b TIFF images, approx. 12MB each

What speed do you get from a Video Analysis pass in VirtualDub?

David

[colinhunt]

What speed do you get from a Video Analysis pass in VirtualDub?
It's been years since I last used VirtualDub. You have to help me out here; how do I trigger a Video Analysis pass? I'm guessing I open the .avs in VirtualDub first.

edit: Whoop, never mind, found it. I get 1.16-1.17fps from the analysis pass.

[Didée]

Input: A sequence of 1920x1080x24b TIFF images, approx. 12MB each
Can't tell if it's related to that "speed problem", but ... a 1920x1080@24b TIFF should have a size around 6MB. When your TIFFs are 12MB, then they have a colordepth of 16bit instead of 8bit. Check the actual colordepth, and try if it goes faster with 8bit ones. Also, try using "Converttoyv12()" directly in the avs script, to see if that makes a speed difference.

[colinhunt]

Can't tell if it's related to that "speed problem", but ... a 1920x1080@24b TIFF should have a size around 6MB. When your TIFFs are 12MB, then they have a colordepth of 16bit instead of 8bit. Check the actual colordepth, and try if it goes faster with 8bit ones.
Well spotted! I opened a few of the files in IrfanView and looked at the status bar, which said the files are 1920x1080x24b. But I took a closer look now, and the image information screen says the files are actually 48bpp. Unfortunately the software I use to output these TIFFs has no settings for colordepth, so I can only get 48bpp TIFFs out of it. What would be the fastest way of converting these files from 48bpp to 24bpp?

Also, try using "Converttoyv12()" directly in the avs script, to see if that makes a speed difference.
Will do, thanks!

[Mounir]

Have you ever tried .bmp ?

[colinhunt]

Have you ever tried .bmp ?
The software outputs only TIFF and DPX files.

[colinhunt]

Can I use, for example, ImageMagick as part of the Avisynth script to convert from 48bpp to 24bpp on-the-fly before feeding the image for processing?

[wonkey_monkey]

I can only think that disc access is your bottleneck. You might be able to speed things up slightly by using some other input function, but if your CPU is idle for 95% of the time, I'd guess this must just be because the images are taking so long to load.

Are D: and W: separate physical disks from C:?

David

[Wilbert]

Can I use, for example, ImageMagick as part of the Avisynth script to convert from 48bpp to 24bpp on-the-fly before feeding the image for processing?
Yes, but that's what ImageSource (well, the DevIL library actually) is doing too. So, i don't know if there is any speed gain. You will have to try it yourself.

Btw, could you upload some of these 16bit tiff images somewhere?

[colinhunt]

Did some additional testing. First I batch converted those 16bit TIFFs to 8bit files with IrfanView, which took wayyyy too long. I then fed those 8bit TIFFs to ImageSource and got no speed increase at all. Adding ConverttoYV12() to the .avs script didn't help either.

I then tried 'immareadseq'. With 8bit TIFFs I got a whopping (it's relative, isn't it?) speed increase to 2.8fps. LAN transfers reached 160mbps and x264's CPU load went all the way up to 14% at best. That's still way too slow; I should be looking at more than 15fps for x264 to be feasible in larger projects.

What do you guys think, should I start messing about with MT and 64-bit stuff? How about speeding things up with piping, like RipBot? (No, I don't know what I'm talking about here, just throwing ideas around.)


I can only think that disc access is your bottleneck. You might be able to speed things up slightly by using some other input function, but if your CPU is idle for 95% of the time, I'd guess this must just be because the images are taking so long to load. Are D: and W: separate physical disks from C:?
C: and D: are on the same SSD drive, W: is a NAS that has reached read speeds of 90MB/sec. I have a much faster box which I intend to bring on-line next week, so I'll be able to see if disc access really is the bottleneck here.

Yes, but that's what ImageSource (well, the DevIL library actually) is doing too. So, i don't know if there is any speed gain. You will have to try it yourself.
Hmm, sounds a bit pointless in that case.

Btw, could you upload some of these 16bit tiff images somewhere?
At first I thought "sure"... but this test was shot on my front yard and I realised releasing pics of my house on the 'net might not be the most sensible thing to do. I'll venture further out to shoot more test footage next week; I can upload 16bit TIFFs from that shoot later on.

[wonkey_monkey]

C: and D: are on the same SSD drive, W: is a NAS that has reached read speeds of 90MB/sec.

I'd try putting the input files on C:/D: and saving the output to W:, just to see if it makes any difference, or put a few of the files in a RAM disk and run them through from there. I may be missing something, but if the CPU is doing nothing for 95% of the time, it must be because it's waiting for something, and those are some huge input frames you've got there.

David

[colinhunt]

I'd try putting the input files on C:/D: and saving the output to W:, just to see if it makes any difference, or put a few of the files in a RAM disk and run them through from there. I may be missing something, but if the CPU is doing nothing for 95% of the time, it must be because it's waiting for something, and those are some huge input frames you've got there.
RAMdisk is a good idea, thanks! I can also try OCZ's Throttle 32GB eSATA drive, it should be pretty fast too.

[colinhunt]

Whoa! David, you were absolutely correct: there was indeed a file access bottleneck slowing the whole process down.

I copied the 8bit TIFFs to a OCZ Throttle eSATA drive, then edited the scripts for eSATA drive as source and NAS drive as destination - and boom, 15.03fps!
Outputting to the internal SSD drive was slightly faster at 15.36fps. x264 CPU load wobbled between 65 and 80 percent.

I guess this proves that reading small-ish files off a NAS over a Gb LAN is not a good solution, even when you can read large files off the NAS at 90MB/s. (Hmm, and yet copying the TIFF files from the NAS to internal SSD happens at 65MB/s. I wonder what in the encoding process chokes down the NAS read ops...) It will be interesting to see if I can squeeze more fps out of the system when the SAS comes on-line next week.

Thank you very much to everyone who took time out to reply; I'm still interested in any tips you might have for speeding up this process further. After all, when there's 120 000+ TIFF files to encode, every fps counts :)

[colinhunt]

Urgh, I meant to retire for the night, but instead I couldn't help running some tests with those 16bit TIFFs.

Source: SSD / Destination: NAS / ConverttoYV12() in .avs: no / fps: 5.48 / CPU load 20-25%
Source: SSD / Destination: NAS / ConverttoYV12() in .avs: yes / fps: 5.55 / CPU load 19-28%

Source: eSATA Flash / Destination: NAS / ConverttoYV12() in .avs: no / fps: 3.38 / CPU load 11-18%

Almost 3 o'clock... sleepytime.

[7ekno]

The NAS is the bottleneck, throughput != individual file access ...

Never a good idea to do any video work over a network, no matter the "apparent" speed ...

7ek

[IanB]

Well you seemed to have worked it out.

Summary :- With images of size 12MB each, you can expect this to be extremely limited by the read speed of the storage medium.

E.g. With a very fast disk i.e. 120MB/sec, with no subsequent processing, all you could ever expect is something less than 10 fps. Much less when you factor in processing time as well.

The DevIL libraries used by ImageSource() don't help much either. They do standard OS buffered, record level reads from the source file.

[colinhunt]

The DevIL libraries used by ImageSource() don't help much either. They do standard OS buffered, record level reads from the source file.
Hello Ian, and thanks for chiming in. Can you recommend a faster way of converting from 48bpp to 24bpp on-the-fly? Currently it seems to slow the process down a lot.

[IanB]

I take it you already have the 12MB files. :(

If so you are stuck with 12MB*1000/[disk rate MB/s] milliseconds per file plus all the processing time per frame.

Accessing the data from a really really fast disk system is your only hope.

Maybe copying to a Ram disk will help the script run faster, but you need to still add the time it take to copy the files to the Ram disk as part of the whole process. If you can batch the jobs then you could overlap copying the next batch with processing the current batch.

If you have a choice at the source use a compressed format. Good lossless should see 3 or 4 to 1 depending on the image data. Slightly lossy maybe 10 to 1. If the images are 95% uniform white then very high levels of compression can be achieved. If the images contain 100% irregular very fine detail then compression will not be high.