x264 revision 610 - sliceless threading - what's new

[Sharktooth]

look at the post above...
megui wont set the optimal number of threads for x264... so the threads = 0 option in the config window is a better choice (coz will force --threads auto)

http://mewiki.project357.com/wiki/MeGUI_FAQ#How_does_the_number_of_threads_affect_quality
http://mewiki.project357.com/wiki/Settings_window
and http://mewiki.project357.com/wiki/X264_configuration

[Thunderbolt8]

did it with all that closing & reopening, with =0 (threads auto) and =4. in both cases ~10,5 fps.

i want 606 back :S

[bob0r]

@Thunderbolt8

Can you download a 700MB yuv file?
Then we can do a speed test on:

x264.606.exe --threads 2 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720
x264.606.exe --threads 3 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720

x264.614.exe --threads 2 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720
x264.614.exe --threads 3 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720

You probably already answered one of these but:
What CPU do you have (name - # cores - HT yes/no - Ghz?)
What Operation System are you using?

If these results show positive towards 614, and you have problems with an .avs encode.
I have learned that people using MultiThreaded avisynth at first also had the slowdown, but then changed settings from setmode 5 to setmode 2 for example, and all of a sudden their encode went a lot faster.

Let's test this and show once again avisynth can be a huge bottleneck.

[experttech]

@Thunderbolt8

Can you download a 700MB yuv file?
Then we can do a speed test on:

x264.606.exe --threads 2 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720
x264.606.exe --threads 3 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720

x264.614.exe --threads 2 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720
x264.614.exe --threads 3 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720

You probably already answered one of these but:
What CPU do you have (name - # cores - HT yes/no - Ghz?)
What Operation System are you using?

If these results show positive towards 614, and you have problems with an .avs encode.
I have learned that people using MultiThreaded avisynth at first also had the slowdown, but then changed settings from setmode 5 to setmode 2 for example, and all of a sudden their encode went a lot faster.

Let's test this and show once again avisynth can be a huge bottleneck.

I can do this too. Where can I download the file?

[vanger]

This is my avs file:


DirectShowSource("I:\Tape No 1\Wine_Country.avi",fps=29.97000002997,audio=false)
edeintted = last.AssumeBFF().SeparateFields().SelectEven().EEDI2(field=-1)
TDeint(order=0,full=false,edeint=edeintted)
#crop
#resize
#denoise

ConvertToYV12()
TDeint+EEDI2 is too slow... With my x2 3800+ (oced to 2500Mhz) the CPU load is near 50% (same as yours)...
Try using other deinterlacer, e.g. TomsMoComp or LeakKernelDeinterlace...

I have another problem with x264 rev.611/612/614 - the encoding stops or slows down fps to 0 without any errors after 13800-14600 frames of encoding...
I tried different sources such as mpeg-2 (.d2v) or Avi (huffyuv) with filters and without...

I tried such avs scripts:
DGDecode_mpeg2source("D:\capture\video\edw.d2v",info=3)
ColorMatrix(hints=true)

crop(16,0,0,0)

or
DGDecode_mpeg2source("D:\capture\video\edw.d2v",info=3)
ColorMatrix(hints=true)

crop(16,0,0,0)

SelectRangeEvery(140, 14)
or
AVISource("rtr.avi")
FFT3DFilter(sigma=6)
or just
AVISource("rtr.avi")
With the last script the encoding stops near 14500 frame, with adding FFT3DFilter(sigma=6) it stops near 13800 frame.

The MeGui log says nothing:
Starting job job2-1 at 8:41:32
encoder commandline:
--pass 1 --bitrate 1950 --stats "D:\capture\video\zxxxz2.stats" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads auto --thread-input --progress --no-psnr --output NUL "D:\capture\video\zxxxz2.avs"
successfully started encoding
Processing ended at 8:48:16
----------------------------------------------------------------------------------------------------------

Log for job job2-1

avis [info]: 768x576 @ 25.00 fps (16143 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [warning]: VBV maxrate specified, but no bufsize.

----------------------------------------------------------------------------------------------------------
The current job contains errors. Skipping chained jobs
Starting x264 in cmd - the result is the same...

The encoding of video fragment (13000frames) works fine...

[Thunderbolt8]

I have learned that people using MultiThreaded avisynth at first also had the slowdown, but then changed settings from setmode 5 to setmode 2 for example, and all of a sudden their encode went a lot faster.Let's test this and show once again avisynth can be a huge bottleneck.
im not using MT avisynth at all.

[bob0r]

Hence the slowdown, avisynth can't feed the frames fast enough for x264's sliceless threading.

[experttech]

TDeint+EEDI2 is too slow... With my x2 3800+ (oced to 2500Mhz) the CPU load is near 50% (same as yours)...
Try using other deinterlacer, e.g. TomsMoComp or LeakKernelDeinterlace...

You are the man! I used TomsMoComp and now get CPU usage of ~100 :) and ~60 fps!

Thanks a lot!

[Thunderbolt8]

so I definately have to use MT avisynth now in order to get the speed increase ?
what script lines should I enter ?

[experttech]

Using AVISynth will ony increase your encoding time. I just replaced an AVISynth filter. The other deinterlacing filter was not optimized for multithreading.

You said that you were getting slowness even from the command line and skipping avisynth. You also said that a progressive source was slowing down. So using Avisynth wont help you.

Thats all I can say.

[nm]

Using AVISynth will ony increase your encoding time. I just replaced an AVISynth filter.With another AviSynth filter.

You said that you were getting slowness even from the command line and skipping avisynth.He still used AviSynth through x264.

So using Avisynth wont help you.
Using the right version in the right way might help. But don't ask me what that way is; I only use MEncoder ;)

@Thunderbolt8:
If switching to multithreaded AviSynth won't help and you still want to find out what is going on, take look at the experiment that bob0r suggested. Get both 606 and the latest x264 build and run test encodes on the command line: first with YUV input and then with the simplest possible AviSynth script and default x264 parameters. If that works fine, add filters and parameters until the problem surfaces.

[Thunderbolt8]

@Thunderbolt8:
If switching to multithreaded AviSynth won't help and you still want to find out what is going on, take look at the experiment that bob0r suggested. Get both 606 and the latest x264 build and run test encodes on the command line: first with YUV input and then with the simplest possible AviSynth script and default x264 parameters. If that works fine, add filters and parameters until the problem surfaces.
where can I find the old builds (cli) of x264 ?

[Sharktooth]

x264.nl or http://megui.org/auto

[Thunderbolt8]

btw. which avisynt version is recommended to be used atm, 256 or 257 rc1 ?

[Blue_MiSfit]

Awesome results.

Working with 1080p24 HuffYUV (filtered result of fft3dfilter etc on MPEG2 HDV 25Mbps source), and setmtmode(3,0) with automatic threading on a MacPro 2.66 2x dual core, I get ~6.5 - 7 fps on both first and second passes. Utalization is constantly over 85%, and generally in the high 90s. Well done guys, this is a tremendous improvement!

@Thunderbolt8
I have learned that people using MultiThreaded avisynth at first also had the slowdown, but then changed settings from setmode 5 to setmode 2 for example, and all of a sudden their encode went a lot faster.

Let's test this and show once again avisynth can be a huge bottleneck.

MT Avisynth's mode 5 can be slower than single threading :)

I usually use 3, because I've found it works well with almost everything. 2 is a bit faster but is less compatible.

~MiSfit

~MiSfit

[Thunderbolt8]

aerf, just dled and installed megui from x264.nl and let it autoupdate, but it messed up all the profiles, because they are not there, although the auto update feature says all are up-to-date. in the update selection window, there were also no profiles to be seen I could have selected to import :S

[experttech]

I notice that Pass 2 is slower than Pass1 but I think its supposed to be that way. With the HQ-Slow profile i get ~50 fps in Pass1 and ~20fps in Pass2 and I am happy. CPU utilization is ~100% always. Nice.

Oh yeah the input is NTSC DV AVI 29.97 fps 720x480i

[Thunderbolt8]

its definately supposed that way.
the 20fps for 2nd pass is definately what it should be. i just reinstalled windows, lets see if something has changed for me

edit: nothing has changed -.-

[emmel]

Thunderbolt8, you could try simplifying your avs-script and x264 command line for test purposes. The avs-script does not have to include all the fancy stuff, it suffices to load the source and for this you only need one line, e.g. MPEG2source("clip.d2v"). Then, try x264 with default parameters:
x264.exe --progress --threads 1 -o "outfile.mp4" "script.avs"
x264.exe --progress --threads 2 -o "outfile.mp4" "script.avs"
If this works, try something a little more complex. In this way, it is much easier to figure out wheres the problem.

[Thunderbolt8]

took out colour correction in avisynth script. 1st pass got even increased by 13 fps, 2nd pass still stays ~the same..


edit: ok, this is funny now. colour correction still out in avs, I shortened the command line down to that:

C:\Programme\VIDEOBEARBEITUNG\MeGUI\tools\x264\x264.exe --pass 1 --bitrate 2008 --stats "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.stats" --threads 16 --thread-input --progress --output NUL "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.avs"

C:\Programme\VIDEOBEARBEITUNG\MeGUI\tools\x264\x264.exe --pass 2 --bitrate 2008 --stats "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.stats" --threads 4 --thread-input --progress --output "D:\DVD\Projects\HAPPY_TOGETHER_VTS_01_PGC1\testtrim.mp4" "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.avs"


1st pass got slowed down now to ~38 fps (why the hell is that possible, when I actually shortened the options), 2nd pass therefore got increased to ~40 fps. i dont have any clue whats going on :S

[emmel]

Did you try the default x264 options? Did you try other video clips? Try at least once something else.

[Thunderbolt8]

well what are the "default" options ?

edit: found the button -.-

same clip with default:

threads auto, 2, 4, 8, 16: all within ~2 fps difference


ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

[experttech]

Thunderbolt can you post your avs?

[Thunderbolt8]

# Set DAR in encoder to 37 : 20. The following line is for automatic signalling
global MeGUI_darx = 37
global MeGUI_dary = 20
DGDecode_mpeg2source("D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\OLD_BOY.d2v",info=3)
trim(84000,89000)
#blank deinterlace line
#crop
#resize
#denoise


avisynth 256, dgindex & decode 148

[emmel]

well what are the "default" options ?
I mean no options, except the input and output files.


threads auto, 2, 4, 8, 16: all within ~2 fps difference
Now how does that compare to --threads 1?

[nm]

Well, Thunderbolt8 already tried with (almost) default parameters and that seems to work as it should (to know for sure, he would need to do the same with r606).


edit: ok, this is funny now. colour correction still out in avs, I shortened the command line down to that:

C:\Programme\VIDEOBEARBEITUNG\MeGUI\tools\x264\x264.exe --pass 1 --bitrate 2008 --stats "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.stats" --threads 16 --thread-input --progress --output NUL "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.avs"

C:\Programme\VIDEOBEARBEITUNG\MeGUI\tools\x264\x264.exe --pass 2 --bitrate 2008 --stats "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.stats" --threads 4 --thread-input --progress --output "D:\DVD\Projects\HAPPY_TOGETHER_VTS_01_PGC1\testtrim.mp4" "D:\DVD\Projects\OLD_BOY_VTS_01_PGC1\testtrim.avs"


1st pass got slowed down now to ~38 fps (why the hell is that possible, when I actually shortened the options), 2nd pass therefore got increased to ~40 fps. i dont have any clue whats going on :S
That's how it works with default parameters -- both passes run at the same speed. To get a speed difference between the passes, you would need to lower the parameters for the first pass (to get so-called turbo or fast first pass), and you would probably want to use better quality (and slower) settings for the second pass. Just like MeGUI does for you.

Now just get the 606 revision and try the same thing with that.

[Thunderbolt8]

1 tread is ~20 fps less

[huang_ch]

Bring another report on a P4 3.0G w/ HT machine:
--threads 1: ~17fps
--threads auto: ~20fps
Pretty good performance gain. :)

[Sirber]

auto works on mplayer! :D

[experttech]

I tried the HQ Insane profile modified with 3000 kbps bitrate and thats giving me 3 fps or so :(


DirectShowSource("I:\Tape No 3\C.avi",fps=29.97000002997,audio=false)
ConvertToYV12()
TomsMoComp(0,5,1)
crop( 0, 0, -2, -2)
Lanczos4Resize(640,480) # Lanczo4 (Sharp)
Convolution3D("movielq") # Heavy Noise



Starting job job1-2 at 7:13:31 AM
Starting preprocessing of job...
Preprocessing finished!
encoder commandline:
--pass 1 --bitrate 3000 --stats "I:\Tape No 3\C.stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "I:\Tape No 3\C.avs"
successfully started encoding

[Thunderbolt8]

there you go, I also tried it all the time with the modified HQ profile (-5 ref frames, bitrate to reach 1/3 dvd size with audio), and I also only got 8-10 fps in 2nd pass, while I got ~15-17 with 606 -.-

[experttech]

there you go, I also tried it all the time with the modified HQ profile (-5 ref frames, bitrate to reach 1/3 dvd size with audio), and I also only got 8-10 fps in 2nd pass, while I got ~15-17 with 606 -.-

So it could depend on the profile settings maybe?

Try using CE-Mainprofile or HQ-Slow - what fps do you get with these?

Surprisingly CPU utilization is ~100% with HQ-Insane so maybe some setting is taking too much time.

Any ideas folks?

[Thunderbolt8]

CE main profile:

606 (2 threads): 1st pass: ~52 fps; 2nd pass: 19,5 fps
614 (0 = auto): 1st pass: ~60 fpd; 2nd pass: 23 fps

[foxyshadis]

Yes, fps depends on profile! After all, there's a reason they're named HQ-Slow, HQ-Slower, HQ-Slowest, and HQ-Insane! Please pick the one that best matches your speed requirements, and before assuming that you just must have the absolute top quality, ensure that you can actually see a difference with a few test-encodes. The top few quality options can enormously degrade speed while providing a nearly invisible quality difference on most video.

[Thunderbolt8]

my problem is more that with 606 I got like ~15-17 frames on HQ slowest with 5 ref frames and with 614 I only get like 8-11 fps, although the sliceless thing is meant to speed things up.

[foxyshadis]

If you suspect references of causing this, which is quite possible since reference threading is how all this works, try sticking with a single profile and changing only the reference count. Try 1, 6, and 16, for instance. That way you're controlling the variable, intead of changing a whole grab bag of options at once, and you can either confirm or rule it out and go on to the next.

I don't have time now or that's just what I'd be doing.

[vanger]

Trying rev614A and still have such problem:
http://forum.doom9.org/showthread.php?p=919862#post919862
any ideas?

[Thunderbolt8]

no, i dont suspect ref. frames causing this, I only meant I generally use the HQ slowest profile with 5 instead of 10 reference frames, because imho thats enough (I used same number for both versions of course). I have just no idea whats causing my problem.


edit: oh great, now my megui doesnt even start any more. I sometimes had an error when trying to delete some file in the qeue, i got an error screen each time I tried to delete that job or even when trying to close megui. closing via taskmanager was the only solution. but as said, now megui doesnt start any more, even after having already restarted windows. i click on the button, but nothing comes :/

edit: allright, after reinstall stupid megui has gone so far to mess up its own exe file after engaging the auto update so that I cant even start it any more ffs

[Sharktooth]

Trying rev614A and still have such problem:
http://forum.doom9.org/showthread.php?p=919862#post919862
any ideas?
overclocked CPU?

[Thunderbolt8]

could it be possbile that with overclocked cpus the overclocking is somehow not correctly identified and therefore mb kinds of performance distributions to different threads are based on an assuming lower cpu speed or something like that ?

[Dreassica]

Unlikely, sicne mine is overclocked and i managed to finish encodings longer then 34k frames no prob.

[Manao]

No. Overclocking may render your system unstable, leading to some encoding errors such as the one shown on this page, hence the question. It may be the cause of errors whether you're using threads or not.

[vanger]

overclocked CPU?
Yes, but it is stable! With 1 thread it works fine.
Without overclocking I have this error too...
Can I get any debug info from x264, when it stops?

P.S.: MC encoder works fine too (cpu load 100% without any interrupt for 3-7 hours of encoding)

[Romario]

I agree with vanger. Overclocked CPU IS stable if user know how to overclock CPU.

For example, mine Athlon 64 3500+ singlecore from default 2.2 GHz works WITHOUT any problem on 2.7 GHz, and I heavily use x264 encoding without any issue.

[Thunderbolt8]

I agree with vanger. Overclocked CPU IS stable if user know how to overclock CPU.

For example, mine Athlon 64 3500+ singlecore from default 2.2 GHz works WITHOUT any problem on 2.7 GHz, and I heavily use x264 encoding without any issue.
same for me, encoding processes work without any problems, apart from that >606 speed thing that its all about.

[Atak_Snajpera]

I sugest to use this tool if you are not sure if your cpu is stable

http://s24.quicksharing.com/d/6528656/1166992024/orthos_exe_20060420.cab

Go-Ortos is even better than SuperPi !!!

[Adub]

Yes, overclocking can be successful. Like my rig that I am typing this on is overclocked to 3ghz from 2.4ghz. Rock solid. And I am no where near being done.

[woah!]

i am still finding elder is giving me more frames at 100% cpu power ?

even tho both ways are showing 100% cpu power being used, elder gives me a 10fps gain using the same avs script and x264 settings on the 2nd Pass

still much better than before with the older x264 rev606

[Manao]

The concept is that debugging a program is complicated, so a programmer doesn't want to debug uselessly. When using an overclocked CPU, the probability for your system to be unstable and to be the cause of the apparent bug is quite higher than with a non overcloked one, so I find normal to discard bug report on possibly unstable system.

Moreover, an unstable system doesn't mean that it randomly crashes every now and then. It may very crash for one program only, with a specific configuration only. So saying "but only x264 multithreaded has the bug" isn't a justification at all.

Finally, just as a reminder. I haven't encountered yet a perfect test for checking hardware. Even if your hardware passes every known hardware checker, it doesn't mean it is stable. I once had some RAM that was passing every test, except that XviD was failing to encode some specific file with it. Replacing the RAM fixed the issue, so it wasn't XviD fault. Yet, only XviD was crashing.

[dvd_maniac]

How would I get this version to work with Staxrip?