x264 development

[baer999]

Oh well there are many improvements left :)
how many coders are active in this project and what language (C++???) is it ?

[iapir]

Originally posted by Doom9
as far as the roadmap goes, here it is: http://students.washington.edu/lorenm/src/x264/todo.txt


Interresting, there's not even a mention of VfW or AVI limitations, compared to the possibilities of AVC...

[Sharktooth]

Those "limitations" can be worked around.
AVI shouldnt accept b-frames and vbr audio too... but someone managed how to put it in...

[Doom9]

Interresting, there's not even a mention of VfW or AVI limitations, compared to the possibilities of AVC...x264 is container agnostic. In fact, the recommended way of using it is in mencoder, which can deliver raw streams that can be muxed into MP4 or another suitable container. In the future, there might be mpg output as well. As soon as a suitable spec compliant AVC capable MP4 muxer is released, I'll add MP4 output to my mencoder GUI (it's quite powerful so perhaps MP4 output will make you want to forget about VDub, who knows).

[JoeBG]

Originally posted by Doom9
As soon as a suitable spec compliant AVC capable MP4 muxer is released, I'll add MP4 output to my mencoder GUI (it's quite powerful so perhaps MP4 output will make you want to forget about VDub, who knows).

Yes...and many people are waiting for this ;)

[bond]

mp4creator should mux x264 bitstreams into .mp4 perfectly (tested it a lot recently)

it muxes from raw, but that shouldnt be a problem as mencoder can also output raw video streams

[Doom9]

there's more: an mp4muxer needs to be able to split the output at a predefined size... else.. no > 1 CD rips.

[TheBashar]

Originally posted by Doom9
as far as the roadmap goes, here it is: http://students.washington.edu/lorenm/src/x264/todo.txt

I was wondering if there was any hope of getting zones support on the roadmap.

[BoNz1]

Originally posted by TheBashar
I was wondering if there was any hope of getting zones support on the roadmap.

While I definitely don't speak for any of the x264 developers IMO it is more important to concentrate on making improvements to the core before adding things like zones.

[defunkt]

I think it's possible to implement a rudimentary zones capability without distracting the x264 dev(s) from the really important stuff. avs2yuv (for instance) has command line range parameters (or one could use Trim() in the script itself), so it can be used to pipe only portions of an avs source to mencoder. its then fairly simple arithmetic to apply weights (i.e. 0.25, 1.25 ala xvid) to the average bitrate over the whole length of the encode. If output as a raw x264 stream, each portion (encoded with its own bitrate) is then joined with a binary copy (i.e. copy /b enc_01.264 + enc_02.264 + enc_03.264 = encode.264) before muxing. Messy I know, but its all possible with the tools available now. In an ideal world, a bitrate calculator with this feature could be incorporated into a frontend like Doom9's.

What's really needed is some solid data about mp4 overheads.

[akupenguin]

Multiplying the target total bitrate by some number isn't at all the same as a weight (though it is a valid zoning option). If you're going to write a frontend to do this funky splitting and separate encoding, that probably involves more coding that just implementing zones the right way. And you don't have to know anything about H.264 internals to add zones: the ratecontrol functions are separate.

[defunkt]

I'd love to help out with something that doesn't involve deciphering an MPEG RFC & I don't mind using C (from the safety of a M$ IDE) but getting complex multi-platform projects to compile does my head in.

One other advantage of encoding in portions is that it breaks down the total encode time into something manageable on my 1.3 Duron.

[Sharktooth]

rev 120 vfw with installer is here: outdated

Changes:
(r119) bs_align_1() didn't actually write all ones. (so encoded streams with cabac were technically invalid, though no decoder cares.)
(r120) cosmetics

Notes: usual build made with gcc + nasm and default flags + NDEBUG=1.

[akupenguin]

@defunkt:
If you want to try adding zones, consider the beginning of x264_ratecontrol_start() (ratecontrol.c:348) for CQP 1st pass, and the end of get_qscale() for the 2nd pass. (Don't bother adding zones to 1-pass CBR.) Actually weighting or forcing quants should be easy; most of the work would be in the graphical interface if you want to control zones via VfW.

As for your IDE: At some points in the past, x264 has been compiled with MSVC. I don't know if the project files are up to date, since all the published x264vfw builds are now done with gcc.

[defunkt]

@akupenguin

Thanks for the pointer. Actually I've been working on getting it to compile (as a standalone windows dll with exported API style encoding function and status callback - not vfw) and think I'm close though I still need to make a simple client to test it. This done I thought I'd try & incorporate the avs2yuv source (you wrote this?) as a way of doing away with the command line piping and mencoder. Then maybe a zones GUI. Hopefully I'll be in touch soon. ;)

[Doom9]

@defunkt: I thought there's a mencoder patch that already adds avs input. Just so that you don't reinvent the wheel.

[hpn]

Can somebody please explain why I get different first pass stats files (and of course second pass encodes later) when using different versions of VirtualDub: 1.5.10.1 build 2439 and the experimental version 1.6.4 build 23174 (both available at http://www.virtualdub.org/ ). I run a very short avs scrypt (100 frames) for these tests and of course exactly the same x264 settings for both VDub versions, while encoding. On top of that I can't get identical results when running a third test, MANUALLY encoding the same script in VirtualDub build 2439, and when I run the script (and exactly the same bitrate and x264 options) through GKnot that uses the same! VDub 2439 build. Encoding 100 frames takes only a few seconds, so I'll appreciate if someone wants to run these 3 different encodes and confirm (or deny) my observations and explain to me what I'm doing wrong or missing. You could use any frame number or bitrate for these tests.

In the third test when encoding 100 frames in GKnot, aiming at a specific bitrate, in Bitrate tab I check "calculate Avi file size" instead of the default "Calculate average bitrate", then put 1000kBit/s in the "Average Bitrate" field. Then you'll notice that the x264 second pass target bitrate is set by GKnot to 1024kbps (1024b vs. 1000b, depending on the conversion system developers prefer). Then I manualy replace the KGnot generated avs scrypt with my 100 frames scrypt and then click "Start encoding". The GKnot setting files "06_1_firstpass.settings" and "06_1_secondpass.settings" both contain the "passbitrate"=dword:00000400" line (400 HEX = 1024 Dec), so I suppose GKnot really encodes using VDub and x264 at exactly 1024kpbs (but I maybe wrong here, only len0x can confirm). However when I try to MANUALLY encode the same avs scrypt in the same VDub build at 1024kpbs I get completely different first pass stats file (always using Total Commander "Compare By Content" function for side-by-side file comparison).

The avs:

LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\dgdecode.dll")
mpeg2source("g:\t\06.d2v")
trim(40104,40203)
crop(0,72,720,432)

Just for the record, all previous builds of the 1.5.XX.XX branch (I also tried one+ year old build 2366) seem to deliver identical to the latest build 2439 encodes. Also 1.6.4 = 1.6.3. I was about to publish some more extensive Nero-x264 and x264-x264 (not a mistake) codec comparison, but was dissuaded after noticing that my resulting encodes depend not exclusively on the x264 build, but also on the encoding tool I'm using.

[len0x]

Bitrate makes no difference for first pass stats files as you don't have this setting available during first pass, so it must be something else...

[sysKin]

Originally posted by hpn
Can somebody please explain why I get different first pass stats files (and of course second pass encodes later) when using different versions of VirtualDub: 1.5.10.1 build 2439 and the experimental version 1.6.4 build 23174 (both available at http://www.virtualdub.org/ ).(...)
Just guessing, but:
- vdub 1.5 uses YUY2 (not sure) for its "fast recompress" colourspace.
- vdub 1.6 uses YV12
- when you do it manually, you might be doing it on RGB32 (full processing mode).

Or maybe I'm wrong.

Radek

[Sharktooth]

Revision 123 vfw: outdated

Changes:
(rev 121) VfW: x264_log now creates a window for error messages
(rev 122) In N-pass mode if stat_in and stat_out are the same file, instead save to a temp file and overwrite stat_in only when the encode finishes.
(rev 123) Cache half-pixel interpolated reference frames, to avoid duplicate motion compensation.
30-50% speedup at subq=5.

Notes: Build with gcc + nasm and default flags + NDEBUG=1.

[len0x]

Wow, that is indeed a nice speedup! I got more than 30% overall speedup in my tests. Well done, akupenguin! :)

[Doom9]

poor Sharktooth will have to restart his Nero AVC vs. x264 speed test ;)

[Lefungus]

Originally posted by len0x
Wow, that is indeed a nice speedup! I got more than 30% overall speedup in my tests. Well done, akupenguin! :)

Actually, it's the first core patch from syskin.
Everybody would have been satisfied with 10-15% speedup but he thought 30% was fine too :D

[sysKin]

Originally posted by Lefungus
Actually, it's the first core patch from syskin.
Everybody would have been satisfied with 10-15% speedup but he thought 30% was fine too :D Yeah I thought it would be ok ;) but just in case, another 2% with the first sex264 build in some time: http://syskin.is.dreaming.org/x264/ . Diff available.

Have fun,
Radek

[len0x]

Originally posted by Lefungus
Actually, it's the first core patch from syskin.


Kudos to him then :)

Originally posted by Lefungus
Everybody would have been satisfied with 10-15% speedup but he thought 30% was fine too :D

I hope its not as fast as it can get - I want more speedup :D

[igor1st]

Originally posted by sysKin
Just guessing, but:
- vdub 1.5 uses YUY2 (not sure) for its "fast recompress" colourspace.
- vdub 1.6 uses YV12
- when you do it manually, you might be doing it on RGB32 (full processing mode).

Or maybe I'm wrong.

Radek
Original VDub uses only YUY2 (in my tests). But VDubMod is able to use YV12.

[hpn]

Originally posted by len0x
Bitrate makes no difference for first pass stats files as you don't have this setting available during first pass, so it must be something else...

Correct. I'm only going to the second pass (for which I need to specify some bitrate of course) to be able to visually compare the 3 resulting encodes. I can't spot any difference by just watching the vids, but It's there when I make a few screenshots and put the corresponding frames side by side (one over another to be exact).

@sysKin: Thanks for the tip. It must really be connected to the way both versions handle colorspace. I just did 6 different encodes using, fast, normal and full recompress in both VDubs and ended up with 6 different first pass stats files. I'll also try to make some custom build GKnot replacing the 1.5.10 with 1.6.4 and run some more tests, plus some tests with MeGUI. I wouldn't be suprised if I get 10 different encodes at the end at the day, using one and only x264 set of settings :) So I still have to find out which one would be theoretically the best encode of all. Of course it's paranoic, cause all encodes are visually transparent.

[hpn]

Short speed test with the new sysKin 123 build:

Build 117 - 17:30 min. (1050sec.)
Build 123 - 14:59 min. (899sec.)

Edit: (Feb. 24) AthlonXP build 142 - 13:42 min. (822sec.)

14.38% overall speed boost for build 123.

4525 frames, two pass encode, default x264 advanced settings, plus fast first pass disabled, 700kbps

Always nice to see more developers in charge, only hope it won't lead to bunch of x264 branches and complete mess (still not a word at videolan.org x264 repository about any build 123, not that I'm familiar with how exactly these multi-developers projects should work)

EDIT:
Another test, this time at max quality settings (Max reference frames 15, Max B-Frames 1)

Build 117 - 31:02 min. (1862sec.)
Build 123 - 23:25 min. (1405sec.)

24.54% overall speed boost here, so I updated my humble comparison page at
http://www.freewebs.com/hdar/index4.html

[hellfred]

Originally posted by Doom9
poor Sharktooth will have to restart his Nero AVC vs. x264 speed test ;)
Did he already post results of full length encode? I was waiting for that. Did I miss it?

Hellfred

EDIT: OK, I had a look and just found a "Cooming soon" on page 23.

[Sharktooth]

Originally posted by Doom9
poor Sharktooth will have to restart his Nero AVC vs. x264 speed test ;)
... i know :(

@hellfred: no unfortunately i was a bit busy during last week, however at this point i think i will wait a bit to start the test.
New recode version is coming and syskin is on fire :D

[baer999]

wow thats amazing how the time was reduced ! Is there still a time reduce makeable ? That would be great !

[Sharktooth]

Revision 126 vfw is here: http://forum.doom9.org/showthread.php?s=&threadid=89979

I will keep updating that thread daily.

[niamh]

This is nice to have a special sticky for it..the pace this revs are coming out, it's even impossible to try them all :D

[Sharktooth]

i know...i re-started the same encode 5 times... :eek:

[Tommy Carrot]

Originally posted by Sharktooth
Revision 126 vfw is hereAnother nice speedup, almost 10% faster than rev. 123 (11:42 min vs. 12:48)! Well done guys!

Btw, it seems the optimizations on CABAC were very successful, now it's barely slower than CAVLC (~20 sec difference on the above mentioned clip).

[Sirber]

nice nice :D

[igor1st]

I am using a VFW version of x264. And I notice that with long-run encodes the size of file cache grows up and eats almost all free memory.
It can be fixed?

[bond]

Originally posted by igor1st
I am using a VFW version of x264. And I notice that with long-run encodes the size of file cache grows up and eats almost all free memory.
It can be fixed? how much space does it need?

[akupenguin]

Originally posted by Tommy Carrot
Btw, it seems the optimizations on CABAC were very successful, now it's barely slower than CAVLC (~20 sec difference on the above mentioned clip).
CABAC was not modified.

[virus]

Originally posted by igor1st
I am using a VFW version of x264. And I notice that with long-run encodes the size of file cache grows up and eats almost all free memory.
It can be fixed?
If you refer to the file cache, this is perfectly normal. Windows caches both the input and the output file as the encode progresses, using all the free memory available. That memory is freed when needed by other applications. Of course, the file cache is handled by the OS so x264 or any other program has nothing to do with it.

If instead, you refer to the application memory used by vdub(mod) when encoding, then no, it shouldn't grow so high during the encode, not even with latest sysKin's patch, which at 720x576 adds just 441 KB of RAM for each reference frame used.

[igor1st]

Originally posted by bond
how much space does it need?
Today size of file cache reachs ~320MB and amount of free memory decreases to ~30MB.

All my memory - 512MB; OS is XP with SP2; VirtualDubMod 1.5.10.1 build 2439.

[Tommy Carrot]

Originally posted by akupenguin
CABAC was not modified.
I know, but i didn't test the CABAC vs CAVLC performance since rev. ~50, and back then the difference was much bigger.

[igor1st]

Originally posted by virus
If you refer to the file cache, this is perfectly normal. Windows caches both the input and the output file as the encode progresses, using all the free memory available. That memory is freed when needed by other applications. Of course, the file cache is handled by the OS so x264 or any other program has nothing to do with it.

If instead, you refer to the application memory used by vdub(mod) when encoding, then no, it shouldn't grow so high during the encode, not even with latest sysKin's patch, which at 720x576 adds just 441 KB of RAM for each reference frame used.
Most likely that it's my own internal software problem.
A thousand apologies. :)

[peteag]

how's the x.264-codec comparable to the full-featured h.264-standard-codec? looking at the filesize of the codec (app 200kb) and the reference (app 900kb) some features might be missing or incomplete?

[akupenguin]

JM contains an encoder and a decoder, and they don't share any code (anything in common is duplicated). x264 is only an encoder.
x264 doesn't do any of extended profile or high profile. (Though "extended" is misleading, since it's all about error resilience, and is worse in compression ratio than main profile.)
Properly dealing with adaptive interlacing in all cases takes lots of code. x264 doesn't support interlacing.
There are a few other features not supported (weighted prediction, MMCO, B4x4, IPCM)
And the rest is different algorithms (RD aso)

... or we might just be more efficient.

[dragongodz]

or we might just be more efficient.
thats no doubt true. reference software is never really made for speed or efficienty. its simply made to produce the desired output. :)

[Yong]

What's problem with my x264 encoding options?
start /b /low /w avs2yuv %1 -o - | start /b /low /w mencoder - -noskip -nosound -o nul: -ovc x264 -x264encopts keyint=300:nocabac:nodeblock:no4x4mv:nob8x8mv:no8x8mv:subq=1:bframes=3:pass=1:qp_constant=18 -passlogfile "x264.log"

start /b /low /w avs2yuv %1 -o - | start /b /low /w mencoder - -nosound -noskip -o "%temp%\temp.264" -ovc x264 -x264encopts keyint=300:frameref=15:deblockalpha=-3:deblockbeta=-5:ip_factor=1.4:pb_factor=1.3:cabacidc=2:scenecut=40:cabac:8x8mv:4x4mv:b8x8mv:direct_pred=2:subq=5:qp_step=1:qp_min=10:qp_max=51:pass=2:bframes=3:bitrate=800 -passlogfile "x264.log" -ffourcc X264 -of rawvideo

I always get undersized(target bitrate is 800kbps, but result always is around 350kbps) file with some MPEG1 clips:(
I use virus' x264 r123 encoding pack ;)
The x264vfw doesn't have this kind of problem :)

[virus]

Originally posted by Yong
What's problem with my x264 encoding options?

I think you're going way too far with 1st pass reductions.

Leave cabac and inloop on for both passes, and use subq=3 for the 1stpass as well. And if you use 15 ref frames on the 2nd pass, you should use several ref frames on the 1st, too (VfW would use 8 in this case). I'm not surprised VfW works ok: "fast first pass" does a much better job of reducing the 1st pass complexity automatically ;)

If your clip is short and very complex, using 3 bvops (why 3? no more than 1 is recommended, and 0 is even better currently) may make the RC go off quite a bit. Use 3-pass to correct that.

Also you may check that the bitrate is not too high by looking at the average quantizers (use :log=2 to enable that). BTW I've already updated my pack twice since rev. 123 ;)

[pest]

i've tried several versions of x264 but virtualdub 1.6.4
freezes on the first 2nd-pass frame :(
1st pass and single-pass work as expected...nice speed :)

could be my crappy athlon-tb if no one else has
similiar problems

[Yong]

@virus:
Thanks for the advice, try it tonight:)