x264 2-pass size variation?

[magic144]

I originally opened this thread in the MeGUI forum, but it seems it is better suited here...

----

wonder if anyone can help with this one

I'm brand new to x264 encoding and MeGUI, but not to encoding in general.

I have been trying to encode the same movie file now 3 times. Each time I use the same MeGUI 'jobs' and each time I get a very slight variation in output file size, as well as small variations in some of the logged info from x264 (see below at end of this message for a comparison of 2 such 2nd-pass iterations - the variations seem to appear in the 2nd pass of each run - the 1st pass logging output is identical so I haven't included it here for now).

Can anybody explain such variations? The profile and source files used were exactly the same - in effect, I just turned the handle a 2nd time and got a different result. Is h.264 encoding not strictly determinstic, or should I be looking at something in particular for a problem? The reason I'm asking is because the first time I tried this, I noticed a tiny glitch in playback (the screen goes fully white for a very few frames - am using ZP, Haali, CoreAVC) on both of my Core 2 Duo PCs. I did NOT see this glitch (at least not in the same place) on the 2nd and 3rd encoding attempts.

BTW, I created the 3 MeGUI jobs (pass1, pass2, mux) in Auto-Encode mode with a target size of DVD9.
MeGUI version is 0.3.0.1020
x264 version used here was 936 (though since these tests, I have progressed to 937)

================
1st Run
================
--[Information] [2008-08-16 22:31:39] Started handling job
--[Information] [2008-08-16 22:31:39] Preprocessing
--[NoImage] Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 11158 --stats "G:\_compressed\my_movie.stats" --keyint 240 --min-keyint 24 --ref 5 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --b-rdo --bime --weightb --direct auto --nf --subme 6 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --sar 99:100 --progress --no-dct-decimate --no-psnr --no-ssim --output "G:\_compressed\my_movie.mkv" "G:\_compressed\my_movie.avs"
--[Information] [2008-08-16 22:31:41] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 1280x528 @ 23.98 fps (136449 frames)
---[NoImage] x264 [info]: using SAR=99/100
---[NoImage] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4 Cache64
---[NoImage] x264 [info]: slice I:1406 Avg QP:15.92 size:112917
---[NoImage] x264 [info]: slice P:68248 Avg QP:16.55 size: 79484
---[NoImage] x264 [info]: slice B:66795 Avg QP:18.75 size: 35251
---[NoImage] x264 [info]: consecutive B-frames: 16.6% 36.9% 46.5%
---[NoImage] x264 [info]: mb I I16..4: 5.6% 78.3% 16.1%
---[NoImage] x264 [info]: mb P I16..4: 1.1% 15.7% 2.0% P16..4: 30.8% 26.1% 21.5% 0.0% 0.0% skip: 2.8%
---[NoImage] x264 [info]: mb B I16..4: 0.1% 3.1% 0.3% B16..8: 36.9% 2.6% 4.2% direct: 7.7% skip:45.1% L0:40.6% L1:27.4% BI:32.0%
---[NoImage] x264 [info]: 8x8 transform intra:83.8% inter:53.3%
---[NoImage] x264 [info]: direct mvs spatial:92.7% temporal:7.3%
---[NoImage] x264 [info]: ref P L0 40.7% 24.5% 13.7% 11.2% 9.9%
---[NoImage] x264 [info]: ref B L0 47.7% 30.2% 13.8% 8.4%
---[NoImage] x264 [info]: ref B L1 91.2% 8.8%
---[NoImage] x264 [info]: kb/s:11158.5


================
2nd Run
================
--[Information] [2008-08-17 06:20:18] Started handling job
--[Information] [2008-08-17 06:20:18] Preprocessing
--[NoImage] Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 11158 --stats "G:\_compressed\my_movie.stats" --keyint 240 --min-keyint 24 --ref 5 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --b-rdo --bime --weightb --direct auto --nf --subme 6 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --sar 99:100 --progress --no-dct-decimate --no-psnr --no-ssim --output "G:\_compressed\my_movie.mkv" "G:\_compressed\my_movie.avs"
--[Information] [2008-08-17 06:20:20] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 1280x528 @ 23.98 fps (136449 frames)
---[NoImage] x264 [info]: using SAR=99/100
---[NoImage] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4 Cache64
---[NoImage] x264 [info]: slice I:1406 Avg QP:15.92 size:112947
---[NoImage] x264 [info]: slice P:68248 Avg QP:16.55 size: 79485
---[NoImage] x264 [info]: slice B:66795 Avg QP:18.75 size: 35249
---[NoImage] x264 [info]: consecutive B-frames: 16.6% 36.9% 46.5%
---[NoImage] x264 [info]: mb I I16..4: 5.6% 78.2% 16.2%
---[NoImage] x264 [info]: mb P I16..4: 1.1% 15.8% 2.0% P16..4: 30.8% 26.1% 21.5% 0.0% 0.0% skip: 2.8%
---[NoImage] x264 [info]: mb B I16..4: 0.1% 3.1% 0.3% B16..8: 36.9% 2.6% 4.2% direct: 7.7% skip:45.1% L0:40.6% L1:27.4% BI:32.0%
---[NoImage] x264 [info]: 8x8 transform intra:83.8% inter:53.3%
---[NoImage] x264 [info]: direct mvs spatial:92.7% temporal:7.3%
---[NoImage] x264 [info]: ref P L0 40.7% 24.5% 13.7% 11.2% 9.9%
---[NoImage] x264 [info]: ref B L0 47.6% 30.2% 13.8% 8.3%
---[NoImage] x264 [info]: ref B L1 91.2% 8.8%
---[NoImage] x264 [info]: kb/s:11158.5

[Sharktooth]

FYI: the x264 build r936 and r937 included in megui have psy-rdo/psy-trellis (subme 6 and trellis 1 in your commandline), the new b-frame decision (not used in your commandline) and nal-hrd patches included (not used in your commandline). i dont know if the new psy stuff is deterministic, however it shouldnt cause any white frames.

[magic144]

thanks Sharktooth
yeah, I'm holding out hope that that white-frame business was a one-off accident or operator error somehow, even if my testing with encodes proves non-deterministic, I do at least hope that the process is consistently error-free!

perhaps other x264 developers/gurus could offer their opinion(s) as to why subsequent same-parameter encodes would vary

meanwhile, I am going to see if redoing pass-2 encodes with identical STATS files from a single/common pass-1 outcome will yield predictable/identical results...

[Dark Shikari]

The problem likely lies in your source; if the decoder craps out for a frame or two, that might be the result. Its certainly not x264's fault.

[magic144]

well it could be all manner of things I guess (even memory/RAM glitches perhaps) - I can't think to blame the source though, since subsequent encodes have handled the same scene without the error - but then again, like I said, subsequent encodes have been different sizes for some reason!! (which is one of my biggest concerns, founded or unfounded!)

can anybody say categorically that subsequent encodes with the same pass1/pass2 params should/would generate a bitwise equivalent output/target? (perhaps I am very naive in understanding the process - I'm new to x264 - but it's just a bit surprising to have a different outcome on each turn of the handle)

I'm trying some more things still - obviously a lot of patience is required in this number crunching!

[Sharktooth]

x264 is deterministic by default. there's an option to make it non-deterministic, the problem here is what you used is not a vanilla x264 build but it contains some patches and i dont know if those patches add some non determinism.
dark shikari will know that for sure though.

[Dark Shikari]

x264 is deterministic by default. there's an option to make it non-deterministic, the problem here is what you used is not a vanilla x264 build but it contains some patches and i dont know if those patches add some non determinism.
dark shikari will know that for sure though.In multithreaded mode, x264 is not necessarily deterministic. In singlethreaded mode, it is guaranteed to be so.

[magic144]

hmmm, ok - is there a way to force single-threaded mode then - just for another test scenario
I'd like to give that a try too

-EDIT - I guess threads=1 should do it, no?!
(no doubt it will slow down my encode by 50% tho)

thanks a lot for the feedback from you both by the way, much appreciated

in the same vein, is there a 'vanilla' build I should consider using instead?? - just for testing - all I know right now is the x264 that comes from MeGUI's own update process

[ajp_anton]

Remove "--threads auto" (= --threads [CPU cores*1.5]) or set "--threads 1" which is the default. I get identical encodes with the latest patches (different on different machines but I think that's not x264's fault).

[magic144]

thanks anton - will try that

btw, does anyone know if these params violate the DXVA compatibility goal
--no-dct-decimate
--no-fast-pskip

only I've seen them recommended elsewhere

thanks

m

ps - yes, sorry this is all very rudderless and rambling - it's just that I keep waiting for 8-hour encodes to finish! - and all this on the so-called latest-and-greatest E-8500

[Snowknight26]

btw, does anyone know if these params violate the DXVA compatibility goal
--no-dct-decimate
--no-fast-pskip

No, they don't.

[Sharktooth]

In multithreaded mode, x264 is not necessarily deterministic. In singlethreaded mode, it is guaranteed to be so.
--threads default value is 1. i meant with default settings, it is deterministic. however, some settings break the determinism.

[foxyshadis]

well it could be all manner of things I guess (even memory/RAM glitches perhaps) - I can't think to blame the source though, since subsequent encodes have handled the same scene without the error - but then again, like I said, subsequent encodes have been different sizes for some reason!! (which is one of my biggest concerns, founded or unfounded!)

If your avs script uses DirectShowSource, you can never guarantee determinism in the source. That's where the white frames come from, and the other encodes may well have white or grey frames in other areas that you haven't checked.

[magic144]

sorry foxyshadis, just been away for 2 days

I believe MeGui's default behaviour is to use DirectShowSource in the script

what's the alternative to avoid white frames!!??
the current script looks like this:-

--------

# Set DAR in encoder to 12 : 5. The following line is for automatic signalling
global MeGUI_darx = 12
global MeGUI_dary = 5
DirectShowSource("somepath\00000.m2ts",fps=23.9759856527702,audio=false)
#deinterlace
crop( 0, 144, 0, -136)

LanczosResize(1280,528) # Lanczos (Sharp)
#denoise

--------

and what is the root of the problem here? - is it inherently DirectShowSource, or simply the lack of a well-known graph specification?
is it better to use a graph (.grf) in DirectShowSource??
many thanks in advance for your expertise here!

[kemuri-_9]

if your .m2ts is MPEG-2 then use neuron2's DGIndex/DGMPEGDec,
if it's AVC/H.264 then use his DGAVCIndec/DGAVCDec.

those are frameserver filters that will decode the .ts and serve the frames to avisynth in a more deterministic way (granted his AVCDecoder is still alpha)

[magic144]

thanks very muchly kemuri-_9 - I'll give that a try

still wondering what aspect of using DirectShowSource is non-deterministic - the filters that will get used, or the actual frames that will get served...??


...One thing that I notice using AVCSource is that you MUST use 1 thread on my Core 2 Duo box for this encode, else the resultant .mkv or .mp4 file is full of blocky corruption...

-EDIT - hold that thought - apparently x264 r940 broke threading...

[kemuri-_9]

still wondering what aspect of using DirectShowSource is non-deterministic - the filters that will get used, or the actual frames that will get served...??

The frames that DirectShowSource serves to avisynth are non-deterministic: they vary from load to load.

[Avenger007]

The frames that DirectShowSource serves to avisynth are non-deterministic: they vary from load to load.
Is there an alternative to DirectShowSource that can do all DSS does (like serve wmv files) and still be deterministic?

[kemuri-_9]

http://forum.doom9.org/showthread.php?t=127037

[Avenger007]

I tried FFmpegSource but it has a problem with recognizing the correct framerate for wmv files.

23.976 becomes 23.810
29.970 becomes 30.303

[kemuri-_9]

use AssumeFPS() in avisynth

i.e.
AssumeFPS(30000,1001) #approx 29.97
or
AssumeFPS(24000,1001) #approx 23.976

[Avenger007]

It works. :thanks:

Unfortunately audio is another problem. FFmpegSource gives "System exception - Access Violation" when using atrack=-1. (FFmpegSource 1.21)
It seems DSS still has its use.

[magic144]

and as far as using AVCSource goes, for a BluRay source will it be best to "ignore pulldown flags" - I ask as I presume if the input is 23.976, such a setting would preserve this in the output (and if not set as such, the output could/would? be frame-served at 29.97??)

problem also is how to know/verify how "constant" the source fps is??!

[Sagekilla]

Yes, it would be best to ignore pulldown since blu-ray discs come as a 23.976 source -- They don't have any (hard or soft) telecine present on them, afaik. So you should just use Directshowsource("file.grf",fps=24/1.001) or Directshowsource("file.grf").AssumeFPS(24/1.001). Either will produce the correct frame rate.


Edit: woops, just noticed you said AVCSource, but the advice remains the same: Blu-ray sources come as 23.976 fps :)

[kemuri-_9]

the Blu-ray disc accepts 24p in addition to the 30i format that was originally accepted on DVDs.

when you ran the stream through DGAVCIndex, it should have told you data on the stream as it processed, notably the FPS of the video.

use that information to help you with what needs to be on it.

and Sagekilla, pay attention to previous posts, we're not using DirectShowSource because of its fundamental lack of determinism.

[poisondeathray]

Can someone please explain a few things; I don't fully understand and I want to learn. I've searched and read through the FFMpegSource() thread, but there was no satisfactory explanations:

1) What is meant by "lack of determinism" in an encoding sense? Can you define it? I gather it's "bad", and you can get white/grey frames. I've tried to duplicate the poor results with DirectShowSource(), searching encodes for bad frames but I am unable to find any.

2) Under what scenarios does this "non-determinism" occur? More threads has been mentioned, but it's not a causative factor, seems like it might be associated with it

3) What makes FFMpegSource() deterministic? What makes DirectShowSource() non-deterministic?

[magic144]

thanks kemuri - just wondering if you'd know if any source was 'totally' constant fps by such indicators - I suppose in theory it is possible for some weirdo source to have 3 frames buried like needles in haystacks with repeat flags :-(( !!!

anyway, the end of this .dga file says FPS 48000 / 2002 which sounds pretty darn regular I guess

surely DVDs had progressive (24p) encoding too didn't they? - or at least soft-pulldown progressive to allow 3:2 30fps playback?

so between ffmpegsource and avcsource, would you still recommend avcsource at this point - I'm not so sure about this timecode business with ffmpegsource and avenger seems to have had a few issues thus far too...

[magic144]

@poisondeathray

not sure why DSS is non-deterministic vs e.g. AVCSource yet, but that certainly is the opinion here - the non-determinism in this case referring to the actual frames that get served to the encoder actually being unpredictable in the case of DSS (again, I don't know the details)

x264 with multiple threads most certainly is from my own experiments
if you restrict it to a single thread, you get consistently sized output files time after time
whereas, multiple threads (or the default 'auto' setting for dual core's, etc) will give you an observable degree of size variability

I do not know if the x264 non-determinism contributes to the white-frame glitch I've seen (once), but foxyshadis clearly pointed the finger at DSS for such a phenomenon

as for looking for it, I would think it hard to find - I might have just been unlucky enough to see it - but it was 30 mins into a movie, so it is apparently by no means a common occurrence...

[Sagekilla]

Non-deterministic behavior is when the outcome can't be reproduced reliably -- I.e. if you encode the file five times, you'll get slight variations in each one. So naturally it's something that's unwanted in video encoding.

Also, IIRC x264 SHOULD be determnistic with threads. I think one of the newer builds (which was fixed) actually broke the determinism in threads. I don't know about that for 100% though.

[kemuri-_9]

x264 is not guaranteed to be deterministic with multiple threads, only in single thread cases with a deterministic source is it guaranteed to be so.

[Sagekilla]

Hm, never knew that. I've personally never seen any variations with x264 when doing the same encode over and over -- but this was with CRF (1-pass)