x264 development

[temporance]

That looks like a MMX SAD algo, and very similar to the code where I was seeing my predictor-candidate related crash (now fixed, thanks). Could there be other sources of rogue MV candidates?

[akupenguin]

Originally posted by Yong
i tried with x264vfw and mencoder, still same result, 2pass encoded video file size are bit-identical to 3pass encoded What can I say? Works For Me.

Originally posted by Sharktooth
That means the ratecontrol just did it's job in the second pass.
You should be happy with that :) No, it isn't that precise. Even if the N+1th pass can't improve the distribution, it still won't be bit-identical.

[Sharktooth]

btw, nth pass is evil. 2 passes should be enaugh (as in xvid).

[akupenguin]

2 passes is enough. But that's no reason not to offer Nth pass, when all it takes is one checkbox (no extra code in the core). If nothing else, it provides a means to test the accuracy of the 2pass rc. And it will become more useful when multipass is used for more than just ratecontrol.

[digidragon]

I'm using rev 115 and when trying to encode an avisynth file in vdubmod I get "Cannot start video compression - maybe corrupt data (error code -100)".

After a bit of playing, it seems to be the crop and resize functions (either of them) in the script that cause the error.

These are the two relevant lines in the avs file:

crop(6,16,712,548)
LanczosResize(712,376)

Apart from not cropping or resizing, is there a workaround?

[stephanV]

keep your res mod16 (712,376 is mod8)

[digidragon]

Thanks - the error seems to have gone now.

I did check in the codec properties first, and the only format restrictions mentioned were about colour depth...

[Yong]

@akupenguin:
It's work for you?

Here's my encoding batch script:
start /b /low /w mencoder %1 -noskip -nosound -o nul: -ovc x264 -x264encopts keyint=300:nocabac:nodeblock:no4x4mv:nob8x8mv:no8x8mv:subq=1:bframes=3:pass=1:qp_constant=18 -passlogfile "x264.log"

start /b /low /w mencoder %1 -audio-preload 2.0 -noskip -nosound -o "%temp%\temp.avi" -ovc x264 -x264encopts keyint=300:frameref=15:nodeblock:deblockalpha=-6:deblockbeta=-6:ip_factor=1.4:pb_factor=1.3:cabacidc=2:scenecut=40:cabac:8x8mv:4x4mv:b8x8mv:direct_pred=2:subq=5:qp_step=3:rc_sens=1:qp_min=10:qp_max=51:pass=2:bframes=3:bitrate=800 -passlogfile "x264.log" -ffourcc X264

start /b /low /w mencoder %1 -audio-preload 2.0 -noskip -oac copy -o "%temp%\temp.avi" -ovc x264 -x264encopts keyint=300:frameref=15:nodeblock:deblockalpha=-6:deblockbeta=-6:ip_factor=1.4:pb_factor=1.3:cabacidc=2:scenecut=40:cabac:8x8mv:4x4mv:b8x8mv:direct_pred=2:subq=5:qp_step=3:rc_sens=1:qp_min=10:qp_max=51:pass=3:bframes=3:bitrate=800 -passlogfile "x264.log" -ffourcc X264

Output info by mencoder, Second Pass:

Video stream: 800.819 kbit/s (100102 bps) size: 15856213 bytes 158.400 secs
4755 frames
x264 [info]: slice I:37 Avg QP:17.95 Avg size: 8967
x264 [info]: slice P:1359 Avg QP:19.63 Avg size: 6157
x264 [info]: slice B:3354 Avg QP:21.52 Avg size: 2127
x264 [info]: slice I Avg I4x4:58.8% I16x16:41.2%
x264 [info]: slice P Avg I4x4:13.7% I16x16:7.1% P:47.4% P8x8:16.7% PSKIP:1
5.0%
x264 [info]: slice B Avg I4x4:1.3% I16x16:0.8% P:24.8% B:17.6% B8x8:10.0%
DIRECT:9.0% BSKIP:36.5%
x264 [info]: kb/s:799.9

Third Pass:

Video stream: 800.819 kbit/s (100102 bps) size: 15856213 bytes 158.400 secs
4755 frames
x264 [info]: slice I:37 Avg QP:17.95 Avg size: 8967
x264 [info]: slice P:1359 Avg QP:19.63 Avg size: 6157
x264 [info]: slice B:3354 Avg QP:21.52 Avg size: 2127
x264 [info]: slice I Avg I4x4:58.8% I16x16:41.2%
x264 [info]: slice P Avg I4x4:13.7% I16x16:7.1% P:47.4% P8x8:16.7% PSKIP:1
5.0%
x264 [info]: slice B Avg I4x4:1.3% I16x16:0.8% P:24.8% B:17.6% B8x8:10.0%
DIRECT:9.0% BSKIP:36.5%
x264 [info]: kb/s:799.9

The Secondpass and Thirdpass x264 [info] were same.
I tried many time, with mencoder and x264vfw, the result were same, may be something wrong in my batch encoding script...
I can't believe why it couldn't wotk for me...

[akupenguin]

Originally posted by Yong
pass=2
That's exactly what I told you not to do:
Originally posted by akupenguin
In that case, it's simple: "pass=2" means 2 pass encoding. "pass=3" means N pass encoding.
A 3 pass encode consists of one encode with fast settings and pass=1, followed by two encodes with identical slow settings and pass=3.

[Yong]

@akupenguin:
Ok, actually my batch script will have rem(remark) at the start of batch script.

Here's my encoding work:
1. Only the first line of batch script are active,
Second and third pass is disabled, after that will backup the stats file.

2. Only the second pass are active.

3. Only the the third pass are active.

4. Then i will end up a 2pass encoded and 3pass encoded video.(for comparing)

I have three questions,
Is second pass encoding read the stats file only?
Then thrid pass will read and overwtrite the stats file?
So the 3pass encoding batch script is look like this?
Or one 3thpass encoding command only?

start /b /low /w mencoder %1 -audio-preload 2.0 -noskip -nosound -o "%temp%\temp.avi" -ovc x264 -x264encopts keyint=300:frameref=15:nodeblock:deblockalpha=-6:deblockbeta=-6:ip_factor=1.4:pb_factor=1. 3:cabacidc=2:scenecut=40:cabac:8x8mv:4x4mv:b8x8mv:direct_pred=2:subq=5:qp_step=3:rc_sens=1:qp_min=10:qp_max=51:pass=3:bframes=3:bitrate=800 -passlogfile "x264.log" -ffourcc X264

start /b /low /w mencoder %1 -audio-preload 2.0 -noskip -oac copy -o "%temp%\temp.avi" -ovc x264 -x264encopts keyint=300:frameref=15:nodeblock:deblockalpha=-6:deblockbeta=-6:ip_factor=1.4:pb_factor=1. 3:cabacidc=2:scenecut=40:cabac:8x8mv:4x4mv:b8x8mv:direct_pred=2:subq=5:qp_step=3:rc_sens=1:qp_min=10:qp_max=51:pass=3:bframes=3:bitrate=800 -passlogfile "x264.log" -ffourcc X264

You say it's work for you(you are on linux, right?),
but now i can confirm the vfw "update statsfile" doesnt work,
Anyone try it before?

[virus]

Originally posted by Yong
but now i can confirm the vfw "update statsfile" doesnt work,
Anyone try it before?
Works For Me(TM)

500 frames, *very* difficult scene, target 800 kbit/s:

2-pass: 697 kbit/s
3-pass: 804 kbit/s
4-pass: 801 kbit/s

(that also shows the point of multipass: correcting the target size on short clips, when they come out heavily oversized/undersized. On long encodes it's usually not necessary)

virus

[Sharktooth]

ok, at this point i should ask if nth pass (even if "it's evil(TM)":p) is going to be implemented in vfw interface... :D

[virus]

Originally posted by Sharktooth
ok, at this point i should ask if nth pass (even if "it's evil(TM)":p) is going to be implemented in vfw interface... :D
it has been there since revision 105... :rolleyes:

(that's exactly the purpose of the "update statsfile" checkbox, it allows to rerun the 2nd pass as many times as you want reusing the stats from the previous pass, making it effectively a multipass encoding. The test above has been done using the VfW frontend)

[Sharktooth]

good! maybe renaming "2nd pass" with "Nth pass" in the vfw frontend... ;)

[Yong]

Originally posted by virus
Works For Me(TM)

500 frames, *very* difficult scene, target 800 kbit/s:

2-pass: 697 kbit/s
3-pass: 804 kbit/s
4-pass: 801 kbit/s

Wierd, i tried many times, with VDmod,
It must be something worng in my computer.(or my brain):mad:

I also uncoverd a bug, the option: frameref, always keeping crash form me(TM):D, when the value is 15, for some MPEG1 clips at certain percentage.
i tried lowering the value, also crash but more faster.(frameref=15 will crash at 75%, 5 will crash at 30%),
same as mencoder.

Sorry if i'm bring more troboule to this forum, or the x264 devs...

EDIT:
1. the frameref crash problem only occured in the mencoder compiled by celtic druid(but not the celtic druid fault).:p
2.(that also shows the point of multipass: correcting the target size on short clips, when they come out heavily oversized/undersized. On long encodes it's usually not necessary)
I think virus already answered my question,
i tried, it works, but only for short clips(500 frames)...
i tried full lenght(around 4000 frames only), with frameref=15,
all macroblock partition options were selected, still same result, may be not so effective if frameref=15 or something else...

[rushin_911]

To the developers of the codec, thx alot :) I really like how its going, and since I only encode for one cd it is becomming my codec of choise for many of those hard-to-encode anime...

I would like to, however, request the implamantation of Aspect Ratio support... It would be a really good feature :)

[Sharktooth]

Build 117 is up in the main celtic_druid's mirror.

- Changes:
VfW: use separate stats files for each pass of an N-pass encode.

- Revision 116 changes:
VfW: Enable multipass by default, increase the configurable range of I and B quant ratios.
core: Tweak error messages.

[hpn]

I just posted some screenshots using the new revision 117 (plus the latest Nero and XviD counterparts). It's a free hosting, so I guess the links will be down very soon (I'll remove them from the thread if needed)

http://www.freewebs.com/hdar/index.html
http://www.freewebs.com/hdar/index2.html
http://www.freewebs.com/hdar/index3.html

Edit: Feb.13 - the speed table is updated with revision 123 (~25% speed boost, compared to rev 117, both 117 and 123 builds deliver identical picture quality)

[ermannob]

Very nice test, hpn!

What about those parameters about max reference frames and max b-frames? Are they good for standard use, or are they test-specific?

[baer999]

I just saw the few screenshots and it's amazing how near the Codec pictures are on the original! I couldn't see big differences.
I've got a little Question :
Is that right "Nero H.264 Codec is much better then the x264" ?

[Ark]

@ermannob

You can safely use as many reference frames as you want up to the max (15...for now??), but b-frames are a little more tricky.

For now they lack adaptiveness and aren't deblocked, so (from some tests done by many users here), the suggested value is 1 max or nothing at all.

Btw, i think that x264 wins over Nero (by a small amount though) in low motion and stays behind it in fast motion (more noticeable here)... but there's room for improvement for both of them...

[len0x]

Originally posted by baer999
Is that right "Nero H.264 Codec is much better then the x264" ?

where did you get this from?

[baer999]

I've read it in some Threads that Nero is 2 / 3 quicker or something like this ! And Quality should be better too

[len0x]

You shouldn't trust what you read - you have to see the difference with your own eyes... (and I've heard the opposite about speed btw, but I never used Nero myself).

P.S. you might also be confusing Nero ASP with Nero AVC, coz Nero AVC simply cannot be 3 times faster than x264.

[Sharktooth]

x264 is faster than nero (sometimes even 2 times...) with the same settings (i always use 5 reference frames though...) but as ark said it still has not "adaptive b-frame" feature.
So dont use multiple b-frames or do not use b-frames at all.
For what concerns the final quality it cannot be compared for the above reason.
Try to encode both clips without b-frames for both the encoders and then you can draw some conclusions.

[virus]

watching the comparison posted by hpn, I saw something that catched my attention: revision 117 with the "update statsfile" disabled. It's now on by default (hit Load Defaults and see) but I think a brief explanation of why it has been enabled is in order.

Starting with revision 117, VfW automatically handles the statsfile name for you (that's why you aren't allowed to change its name during the Nth pass anymore, but only before the 1stpass). From now on x264 will never overwrite a statsfile previously created during previous passes of the same encode, so it won't be damaged if a crash happens. Let's see how the naming scheme works, using the default name (x264.stats) as an example:

1st pass:
input: none
output: x264-1.stats

2nd pass:
input: x264-1.stats
output: x264-2.stats

3rd pass:
input: x264-2.stats
output: x264-3.stats

...at this point you should start to see a pattern :D

That means that if you leave "update statsfile" enabled, you will still be able to see/use your original 1stpass statsfile if you need it. But that will also give you an important opportunity: if after 2 passes your target size has not been hit with enough precision, you can easily run another pass and improve it. In Vdub, that means simply pressing F7 and then OK, without the need for changing anything.

If you leave the checkbox disabled, you'll be forced to open the dialog, enable the checkbox, and then run both the 2nd pass (again) as well as the 3rd. That's why it's good to leave it always enabled. You have nothing to lose, and sometimes it will be very handy ;)

Oh btw... when you do a new encode using again the same filename in the same directory, x264 will delete automatically all previously created statfiles for each pass with the same name (in our example all three x264-*.stats would be deleted and a new file x264-1.stats would be created). So make sure to use a different name or rename them after the encoding if you want to keep them for future use ;)

hope this helps :)
virus

[Doom9]

but b-frames are a little more tricky.
For now they lack adaptiveness and aren't deblockedI guess we should have a x264 roadmap sticky so we can cross things off the list as they are implemented..
RDO would also have to be placed on that list, and I guess optimization for speed (using comparable x264 settings as for ateme AVC in the codec comparison, the latter is definitely faster on my box.. though by how much I cannot say at this point as I never leave the PC unattended during encoding except for codec comparisons).
And then there's the decoder part though I figure that's a low priority item.
What about weighted prediction?

@virus: good explanation :)

[Sharktooth]

@Doom9: on my machine (athlon xp) x264 is faster than Recode expecially in the second pass when sometimes x264 is twice as fast...

[Manao]

x264 is faster than nero (sometimes even 2 times...) with the same settings I seriously doubt that ( even with 5 references ). And anyway, the only thing that matters is speed comparison at same quality settings, not same settings ( different encoders -> different handling of the settings which can lead to different final quality ).

Doom9 : imho, RDO should be the priority, followed by adaptive bvop. Weighted prediction is cool stuff, but not that easy to use, and only really usefull in fadein / out which tends to occur only in trailers. I'd add FRext on the todo list with a higher priority than weighted prediction. The decoder is more advanced than the encoder, since ffmpeg decodes weighted prediction already. It still lacks bvop deblocking, though.

[baer999]

Ok now it seems that x264 is the new codec for me, but as I can see it is at the beginning and their are many improvements to do. When I want to make well looking Rips how long should I wait, or how much time does the bvop and RDO need (only a prediction). I think that will make the codec much better and because of this I don't want to make my Backups now ...

[Sharktooth]

Ok, im going to measure encoding time for both the codecs.

[Sharktooth]

Test is finished. X.264 is FASTER.


Source: Spiderman 2 trailer (DVD source)

Settings:
Ref Frames=5
B-Frames=1
Deblocking Strenght=0
All options enabled

Avisynth script:
# VIDEO SOURCE
Mpeg2Source("sp2tr.d2v", idct=2)

# CROPPING
Crop(0,14,720,548)

# RESIZING
LanczosResize(720,384)
The same script was used for both the encodes to ensure they were not influenced by a different code for mpeg2 decoding and resizing.


Encoding speed details for Nero Recode:
1st pass 2nd pass
Time(m,s) 11,08 11,04
Av. FPS 4.52 4.55

Encoding speed details for x.264:
1st pass 2nd pass
Time(m,s) 5,43 7,06
Av. FPS 8,82 7,10

[Doom9]

you guys obviously don't have the same ateme AVC encoder as I do ;) I guess you still don't get 31 fps for a full lenght movie (AVS input with no resizing). I'm not even sure if you can make Recode not do any preprocessing prior to encoding. And as you might know, I have an Athlon 64 3500+ (ateme's encoder should perform better on Intel processors afaik).

When I want to make well looking Rips how long should I wait,is a really uncool statement as it implicitly tries to push developers to give you what YOU want right now. You need to learn to live with what you got. And just as much as there'll always be newer codec builds, there'll always be new hardware, so you should never encode your movies and never buy new hardware ;)

[Doom9]

@Sharktooth:I'm afraid you'll have to do full movie encodings. As stated in the codec comparison, ateme's AVC encoder uses a to be patented technique that makes the first pass gradually speed up (look at my encoding speed numbers in the comparison. you'll see there's a significant difference between a 10'000 clip and the full movie).

[Sharktooth]

Ok, but what about the second pass?
The difference is still there and btw i didnt use "fast first pass" for x.264.
I dont want to be harsh but... patented or not nero seems slower than x.264 to me.

EDIT: btw we're talking of RELEASED software.

[Manao]

Sharktooth : what encoding settings ( especially, what quality settings ) did you choose with nero ? Because i can say that Divx is faster than XviD, if, let's say, i compare vhq4 + bvhq + trellis on one side, and fastest mode on the other side.

[Sharktooth]

I enabled and maxed out everything in both x264 and recode (except the settings below).
I used the same reference frames (5), same bframes (1), and same deblocking strength (0) for both.

[Manao]

Good, Best and Extra mode in Nero uses RD optimizations, which aren't yet implemented in x264, and which are slow. If you want to compare Nero and x264, you should use normal mode for nero.

[Doom9]

Ok, but what about the second pass?If you have > 100 fps in the first pass, that makes a sizeable difference ;) I get about real-time encoding using fast first pass in x264, and the 2nd pass is obviously quite a bit slower. Anyway, prior to the next codec comparison it's too early to compare speed ;)

[Sharktooth]

... so if i want to compare xvid to divx i should disable vhq for b-frames, trellis quant, custom matrices... etc etc.
Every codec has its features and since the x.264 encode looks better (even without the fancy RD optimizations) i consider enabling all options in nero the only way to be close in quality to x.264.
So, no thanks, i wont disable anything or the comparison will not be fair.
Originally posted by Doom9
If you have > 100 fps in the first pass, that makes a sizeable difference ;) I get about real-time encoding using fast first pass in x264, and the 2nd pass is obviously quite a bit slower. Anyway, prior to the next codec comparison it's too early to compare speed ;)
Well you dont get realtime encoding... you get 30fps at the end of first pass.
Take note of the number of frames, the time needed for the first pass and manually calculate the average FPS (that's what i've done in my comparison) then draw your conclusions.
For example, nero recode reported 9fps at the end of first pass and "wanted" me to believe that was the encoding speed... If math it's not an opinion, that was not true.

[Doom9]

@Sharktooth: I think you are just misunderstanding me here > 100 fps is encavc's first pass fps (encoding time in seconds/frames), the 31 or so fps is encoding time for both passes divided by twice the number of frames. Now, once again, in order to compare FPS you need to do full movie encodings.. everything else is just plain useless.

And in my vocabulary, real-time encoding = average fps > fps of the source. Your definition may vary, but at the end of the day, > 2x realtime = encoding a movie in two pass mode takes less long than watching the movie imho makes a lot of sense.

[Sharktooth]

whole movie encoding test coming soon...

[virus]

Originally posted by Manao
imho, RDO should be the priority, followed by adaptive bvop. (...) I'd add FRext on the todo list with a higher priority than weighted prediction. The decoder is more advanced than the encoder, since ffmpeg decodes weighted prediction already. It still lacks bvop deblocking, though.
oh, it's feature request time, let's join the party :D

My PSNR tests told me that: the most important thing right now is bvop adaptivity. It will bring a considerable quality boost with a (hopefully) limited increase in complexity. After that, I rate as second the use of bvops as references (but that requires bvop deblocking in lavc or the decoding will b0rk). Both of these are tasks where sysKin can throw his experience in ;)

After those 2, imho it's debatable. When I complained with akupenguin because he set subpel ref. to 5 as default (the slowest setting), he told me that it's not really slow... "wait and see when I add R/D optimization". Lesson learned: R/D will improve things, but with a considerable slowdown, so I don't hold my breath for it :)

That makes me think that a bit of general speed optimization may be in order before R/D becomes usable (speed-wise). But to do speed optimization, maybe it's needed to fully stabilize bvop implementation first (by adding all modes down to 4x4 and maybe other minor things I don't know). Also, currently bvops don't have a proper ratecontrol (just an offset from pvops), so maybe a bit of tweaking of the RC may be needed to fully exploit them, too. New MB modes, mode decision improvements, R/D and RC tweakings are well-suited to aku ;)

As for weighted prediction, I think it will be slow (on the encoding side) and provide very marginal gains - much like XviD's GMC - so I totally agree with Manao on that. As you can see, there's already quite a bit of work for the core devs, and I haven't even mentioned High Profile... :D

(disclaimer: of course aku and sysKin are totally free to ignore all this and work on anything they want... so guys just have fun hacking x264)

EDIT: typos fixed...

[Sirber]

Keep the great work :D

[akupenguin]

Actually, no, weighted prediction should be fast. In B-frames it doesn't add any analysis complexity, just an alternate averaging function (only ~4% of the decoding CPU-time in lavc even without MMX). In P-frames it takes only one least-squares fit to determine whether a frame can use weighting, so that should be about the same speed as the PSNR calculation (~3% CPU-time at the fastest settings, comparitively less at slow settings). (Yes, I have implemented that part in a deflicker filter, just not in x264.) And since weighted P-frames are probably rare, it's not a big deal if they're slow to encode. On the other hand, XviD has to do a full motion estimation pass to even decide whether to use GMC.

As for B-frame ratecontrol: I used to give them the same overflow treatment as P-frames. That ended up slightly closer to the target bitrate, but much worse quality. A fixed quant ratio is the best quality per bitrate, and there are better ways to improve prediction.

[hpn]

Originally posted by virus
watching the comparison posted by hpn, I saw something that catched my attention: revision 117 with the "update statsfile" disabled. It's now on by default (hit Load Defaults and see) but I think a brief explanation of why it has been enabled is in order.
virus

Yes, virus is absolutely right. It's a good idea to always keep the "update statsfile" checkbox enabled during second pass. I was aware of that and unchecked it only because I was absolutely sure I'd run a 2-pass encoding only and simply didn't need any extra files (in this case x264-2.stats). Note that in case of 2-pass encoding only, it doesn't matter whether you have 'update statsfile' checkbox enabled or disabled during the second pass. In both cases the resulting encodes after the second pass will be bit-by-bit identical.

[Sharktooth]

Revision 118 win32 vfw with installer is here: outdated

Changes:
VfW: tweak option names

Notes:
Compiled with gcc and nasm with default flags and NDEBUG=1 option.
This revision will produce bit-to-bit identical encodes to rev. 117. The only differences are in some option names in the vfw config dialog.

[temporance]

Originally posted by Doom9
@Sharktooth:I'm afraid you'll have to do full movie encodings. As stated in the codec comparison, ateme's AVC encoder uses a to be patented technique that makes the first pass gradually speed up (look at my encoding speed numbers in the comparison. you'll see there's a significant difference between a 10'000 clip and the full movie). If they haven't filed the patent yet, then it's too late to do so, because they've already released code containing their invention. Anyone got a link to the patent application, or a description of this technique? Sounds interesting, maybe xvid/x264 could use it. Yes I know it might be patented, but so is all of MPEG-4 ;)

[nicco]

Originally posted by Doom9
I guess we should have a x264 roadmap sticky so we can cross things off the list as they are implemented..


I agree!

[Doom9]

as far as the roadmap goes, here it is: http://students.washington.edu/lorenm/src/x264/todo.txt