Ganged vs. Unganged

[Darien1629]

I'm not new to video encoding but I do need some help on this one. I have the option to use my ram ganged and or unganged.
I understand the differences between the options what i need help with is do I get faster encoding with using the ram ganged or unganged. I'm using an AMD quadcore with 4gb of ram 4 x 1gb ram sticks. I'm using megui encoding blu-ray back ups to x264.

Thank you.

[LoRd_MuldeR]

http://pics.computerbase.de/1/9/7/4/6/8.jpg

[yetanotherid]

I'd be tempted to run the same encode twice, while running the RAM both ways to see if it effects the encoding time by any significant amount.

I don't know the answer, but I'd suspect it'd make no difference.

[Ghitulescu]

One needs to run the tests at least twice, as the second run (and subsequent ones) would always profit from caching of data of the first run. That is in particular important for small data that may fit entirely in the RAM.

[yetanotherid]

One needs to run the tests at least twice, as the second run (and subsequent ones) would always profit from caching of data of the first run. That is in particular important for small data that may fit entirely in the RAM.

Sounds fairly unlikely.
I'd be thinking that in order to switch the mode in which your RAM runs you'd have to do it in the BIOS, which I'd be thinking necessitates a reboot, which would have me suspecting you wouldn't be benefiting from any caching of data in RAM.

Which data gets cached to be used during a second encode while encoding?
As I was actually encoding a video when I read your theory (XviD, but I assume the same theory applies?), and as it was only a 20 minute video, I ran it a second time as soon as it finished the first run. As it turned out the second encode took basically the same time, but was 0.06fps slower, so then I had to know.... so I picked another short file to encode and this time ran it twice without using the PC while it was encoding.

[12/02/2011 8:32:16 PM] Custom resolution settings: fixed width of 704 pixels
[12/02/2011 8:32:16 PM] Standalone support enabled: ESS
[12/02/2011 8:32:16 PM] Started encoding.
[12/02/2011 8:32:16 PM] Source resolution: 1280x720
[12/02/2011 8:32:16 PM] Source fps: 23.976
[12/02/2011 8:32:16 PM] Output will contain 30681 frames
[12/02/2011 8:32:16 PM] Using VAQ in XviD
[12/02/2011 8:32:16 PM] Running single pass encoding.
[12/02/2011 8:40:48 PM] Duration was: 8 minutes 32 seconds
[12/02/2011 8:40:48 PM] Speed was: 59.83 fps.
[12/02/2011 8:40:48 PM] Job finished. Total time: 8 minutes 32 seconds

[12/02/2011 8:42:41 PM] Custom resolution settings: fixed width of 704 pixels
[12/02/2011 8:42:41 PM] Standalone support enabled: ESS
[12/02/2011 8:42:41 PM] Started encoding.
[12/02/2011 8:42:41 PM] Source resolution: 1280x720
[12/02/2011 8:42:41 PM] Source fps: 23.976
[12/02/2011 8:42:41 PM] Output will contain 30681 frames
[12/02/2011 8:42:41 PM] Using VAQ in XviD
[12/02/2011 8:42:41 PM] Running single pass encoding.
[12/02/2011 8:51:13 PM] Duration was: 8 minutes 32 seconds
[12/02/2011 8:51:13 PM] Speed was: 59.83 fps.
[12/02/2011 8:51:13 PM] Job finished. Total time: 8 minutes 32 seconds

[LoRd_MuldeR]

Which data gets cached to be used during a second encode while encoding?

HDD data gets cached in the HDD's own cache as well as in the RAM. RAM data get's cached in the CPU's cache.

I think we can assume that the latter has no effect on two consecutive encodes, even without a reboot. With a reboot between the encodes both won't matter.

As it turned out the second encode took basically the same time, but was 0.06fps slower, so then I had to know....

Such minor difference is probably the result of random fluctuations.

You should run the the same test many times for each configuration and then take the average (or median) time/fps for each configuration.

It's likely that the differences will disappear then...

[yetanotherid]

HDD data gets cached in the HDD's own cache as well as in the RAM. RAM data get's cached in the CPU's cache.

True but when you run an encode a second time would the encoder (or the OS etc) realise it's just encoded the same file and look for cached data, or would it simply start from scratch?


Such minor difference is probably the result of random fluctuations.

Most likely they would be, I guess you can't control every background process either, and myself... I'd have considered a 0.06fps variation as no real variation at all, but when some people are determined to disagree with you in this forum it pays to be more precise. ;) Which is why I then ran a different encode twice.

You should run the the same test many times for each configuration and then take the average (or median) time/fps for each configuration.
It's likely that the differences will disappear then...

The first two times I ran the encodes I was using the PC for browsing as well, so I figured that'd be what made the difference. When the new encode and it's second run both resulted in exactly the same frame rate while I'd not been using the PC, I figured I'd disappeared those differences and further runs were unnecessary. :)

It'll be interesting to see if the OP comes back to report whether ganged v unganged actually effects the encoding time, and by how much.

[LoRd_MuldeR]

True but when you run an encode a second time would the encoder (or the OS etc) realise it's just encoded the same file and look for cached data, or would it simply start from scratch?

Well, the encoder requests the data from the filesystem. The filesystem will only request the data from the HDD, if the data isn't in cached in the filesystem's cache (in the RAM) anymore. And, if the data is actually requested from the HDD, the HDD will only re-read the data from the disc, if the requested data isn't in the HDD's cache anymore. Having said that: If we assume that the encoder reads a "big" video file and does that in a linear fashion, then it's very unlikely that the beginning of the video file (which the second encode would request first) is still cached. Anyway, other files that the second encode needs to read right at the beginning, such as the encoder binary itself an the FFMS index file, might still be cached! And this can very well make a difference, especially when the difference between both runs is very small anyway...

[yetanotherid]

Anyway, other files that the second encode needs to read right at the beginning, such as the encoder binary itself an the FFMS index file, might still be cached! And this can very well make a difference, especially when the difference between both runs is very small anyway...

So I'd assume you could run the same encode 27 times and not have any way of knowing what was read from the cache each time and what wasn't? In that case I can see running several encodes might give you an average, although I'd tend to reboot, setup the encode, run it, reboot and then do it all from scratch again, so hopefully two encodes would be all you'd need to run.

Caching of the FFMS index file is one thing I'd never considered. I kind of assumed the index file would be created from scratch each time, probably through years of encoding with AutoGK.

[LoRd_MuldeR]

Well, if you want to be sure that caching behavior doesn't effect your test, you can either do a clean reboot right before each run -or- you run the test n+1 times and ignore the result of the very first run.

[Sharktooth]

the "unganged" option is usually faster. i've run some tests last year but the difference is below 1 FPS.

[Darien1629]

I agree with sharktooth, i tried an ecode with ganged turned on, and turned off. No real significance in fps. Though what i need to figure out is why i'm only incoding blu-rays is at 10fps. it's only about 9hours per back up. I have a quadcore and 4gb of ram. i set to run 4 threads of x264.

[LoRd_MuldeR]

Though what i need to figure out is why i'm only incoding blu-rays is at 10fps.

Doesn't sound too unusual. Unless you are bottlenecked by a slow decoder or by slow pre-procssing filters (e.g. Avisynth), there isn't much you can do about it, except for switching to a "faster" x264 preset or getting a better CPU ^^

it's only about 9hours per back up. I have a quadcore and 4gb of ram. i set to run 4 threads of x264.

The rule of thumb to calculate the "optimum" number of threads for x264 is 3/2 * <number of cores>.

That's also the formula x264 uses by default. And usually there should be no need to overwrite the number of threads manually.

Why did you explicitly choose 4? For a Quadcore CPU you normally would use 6 threads...

[Darien1629]

I tried 6 and it crashed but that was before i switched to windows 7. I'll try it out on one of my next encodes

[Sharktooth]

if it crashed your system is unstable. i regularly use x264 with 24 threads and it runs rock stable.

[LoRd_MuldeR]

if it crashed your system is unstable. i regularly use x264 with 24 threads and it runs rock stable.

...might also be a borked build or an out-of-memory issue (more threads allocate more memory).

I'd definitely recommend trying a different/up-to-date build and watching the memory usage. If memory usage goes up to ~2 GB, you have problem, at least when using 32-Bit x264 with 32-Bit Avisynth. It can easily happen that Avisynth occupies a lot of memory. Switching to 64-Bit resolves the limitation...

[Blue_MiSfit]

If you have a lot of cores, and usually encode with small-ish VBV buffers (around 1 second or less), you should definitely limit the number of threads.

I do a lot of encoding with systems that have 24 logical cores (12 + HT), and when I let x264 provision 36 threads per defaults in this case, and encode with 1 second VBV buffer sizes, rate control will regularly screw up - totally destroying image quality for a second or so. Limiting the number of threads to a sane value seems to fix this. I use 12, but that's because I do a lot of encodes in parallel.

That may be off topic, but I figured it was worth mentioning - just in case there was somebody trying to do the same thing I am.

Cheers,
Derek

[Sharktooth]

i had some problems in that regard, but x264 never crashed.