x264 development

[nm]

I'd like the devs to rethink this action. While there are many guis for win32, you don't have that many for Mac/Linux (or at least not that I'm aware of), and I think it'd be nice to have something around that works "right from the box". Why would someone need to get extra piece of software to do some very simple task (let's just say no cmd guru, or just someone that wants to do a quick encode) when you could have it in the package already. Just my opinions, any feedback is welcome.
Well, I doubt many people were actually using the GTK GUI anyway. Like with the command-line x264, the input needs to be raw video, so the user would need to use a separate tool (MEncoder/MPlayer, avs2yuv+Wine+AviSynth, ...) for decoding the source.

It's much easier to use Avidemux or Handbrake (even the CLI version). Avidemux is already included in most of the big Linux distributions.

[Kurtnoise]

iirc, you have the avs support as input on windows only...

[lexor]

iirc, you have the avs support as input on windows only...

Hence the mention of wine. According to aku, 64bit version is faster under linux through wine with avisynth than the 32bit version is under windows.

[Coolpplse]

On r998 with the MeGui Update I received x264.exe now uses 1.3GB of of 4gb's of ram in Vista 64Bit rather than the usual 600-800MB encoding 1080i MPEG2 -> 1080i x264

On some ocassions with this build even regular SD videos -> x264 use the same 1.3GB's of my ram... Is there a massive memory leak with the latest build or something????

I use Standalone-Blu-ray for 1080i and DXVA-SD-HQ for SD stuff

[Kurtnoise]

Hence the mention of wine. According to aku, 64bit version is faster under linux through wine with avisynth than the 32bit version is under windows.
I spoke for the GTK UI not in general...

[Ramir Gonzales]

Just want to ask,

I installed latest Staxrip beta (StaxRip.beta.1002.1443.exe) and after choosing "HQ Insane" profile and changing subpixel refinement and M.E.Algorithm to the highest level, so Command-Line shows :
--pass 2 --bitrate 1087 --stats ".stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 4 --b-adapt 2 --b-pyramid
--weightb --direct auto --filter -1:-1 --subme 9 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me tesa
--merange 32 --threads auto --progress --no-psnr --no-ssim --output "_EncoderOutput.264_EncoderOutput.264" ""

but when I run x264.exe (x264 core:65 r996 c89bc90) --help it says

me <string> Integer pixel motion estimation method ["hex"]
- dia, hex, umh

In other words, Staxrip proposes to use --me tesa, while x264
tells me it doesn't know --me tesa...:confused:

So my questions are, Is staxrip wrong in it's command-line, and what about these settings, are they as best as possible for a 2-pass encode targeting 700MB output ?

[akupenguin]

--help excludes options that we expect most people not to use. try --longhelp.

[nm]

and what about these settings, are they as best as possible for a 2-pass encode targeting 700MB output ?
You might want to use a higher trellis setting and enable psy-trellis:
--trellis 2 --psy-rd 1.0:1.0

[Ramir Gonzales]

You might want to use a higher trellis setting and enable psy-trellis:
--trellis 2 --psy-rd 1.0:1.0

Thanks nm ! :)
Thanks Akupenguin ! :)

[JarrettH]

Good luck on revision 1000 :devil:

[zyrill]

Hello x264-devs,

I was wondering about the cpu instruction sets being used in the latest (998 - Skystrife) x264 compile: on my Athlon64 x2 (Windsor), when I start an encoding-run, x264 reports
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
and I was wondering whether it did not activate (or show?) all the other sets that could possibly be used, such as - but not only - SSE3!

[J_Darnley]

x264 only uses the features of your CPU which are fastest. You can force it with the --asm switch.

[Ramir Gonzales]

Sorry for another slightly OT question, but are there some special settings to be taken care of when encoding BLACK&WHITE movies ?

I know the "Greyscale Encoding" in xVid, but what about x264 ???

[kemuri-_9]

SSE3 doesn't activate by default on AMD chips because it's not faster than SSE2 and they don't have SSSE3 which would be faster (like Intel counterparts), so it's completely not used at all.

MMX2 and SSE2Slow flags consist of
MMX, MMXEXT, and SSE2 while avoiding SSE2 instructions that are slower on Athlon64s
(There is no SSE1 functions)

[akupenguin]

--no-chroma-me to save a little time. There is no greyscale compression mode because empty chroma compresses to nothing anyway. (There is a greyscale mode in the spec, but it just doesn't matter.)

[zyrill]

Thanks for clearing this up, J_Darnley and kemuri-_9!

Didn't know SSE3 was slower on AMDs in that particular application. Thanks again!

[Ramir Gonzales]

--no-chroma-me to save a little time. There is no greyscale compression mode because empty chroma compresses to nothing anyway. (There is a greyscale mode in the spec, but it just doesn't matter.)

So for B&W movies, --no-chroma-me .
Thanks a lot akupenguin !

[Ranguvar]

And try running the source through a Greyscale() in AviSynth, to effectively remove any rainbows or crap in the otherwise greyscale source :)

[Dark Shikari]

SSE3 doesn't activate by default on AMD chips because it's not faster than SSE2 and they don't have SSSE3 which would be faster (like Intel counterparts), so it's completely not used at all.

MMX2 and SSE2Slow flags consist of
MMX, MMXEXT, and SSE2 while avoiding SSE2 instructions that are slower on Athlon64s
(There is no SSE1 functions)More precisely, SSE3 just isn't useful for anything except working around the cacheline split bug--which only exists on Intel CPUs.

[Ramir Gonzales]

And try running the source through a Greyscale() in AviSynth, to effectively remove any rainbows or crap in the otherwise greyscale source :)
Thanks, I use Staxrip to encode, I have no idea how to do that in the processes of DGIndex, Compressibility Test, 2 x264 passes, and Muxing...:eek:

[zyrill]

I feel a bit dolty asking this since it is a feature only vaguely related to video-encoding. Anyway: is it planned - at any time in the future - to include a "stop and resume"-mode for the encoder? I ask since I thought of a handful of possibilities to stop encoding and resume it at a later point but none is really satisfactory. Two scenarios come to mind:
a) I want to encode during the day when the computer is running anyway but switch it off over night (win hibernation and standby are not to be trusted, I'm afraid - learned so just now the hard way)
b) I want to encode over night when the computer is not in use anyway but want to work during the day having full capacity of my ressources - including RAM - so I want to stop the encoding process completely during the day and not just renice it.
I would guess that it's relatively low on the priority list but then it shouldn't be so hard to implement - if one can code properly, that is (which excludes me) and it'd be a real handy feature, I'm sure.

[Dark Shikari]

I feel a bit dolty asking this since it is a feature only vaguely related to video-encoding. Anyway: is it planned - at any time in the future - to include a "stop and resume"-mode for the encoder? ctrl-s

(and standby and hibernate both work fine for me, no idea what you're talking about)

[zyrill]

ctrl-s

(and standby and hibernate both work fine for me, no idea what you're talking about)

Yesterday evening I hibernated my computer and this morning when I switched it on again, it had a hickup booting windows and the hibernation-state was lost. :s That's why I am a bit vexed with windows hibernation. And that's why I asked about a resume-mode where it wouldn't matter whether the process was dead intermittently.
Thanks for the tip with Ctrl+S didn't know about that...

[Atak_Snajpera]

Yesterday evening I hibernated my computer and this morning when I switched it on again, it had a hickup booting windows and the hibernation-state was lost. :s That's why I am a bit vexed with windows hibernation. And that's why I asked about a resume-mode where it wouldn't matter whether the process was dead intermittently.
Thanks for the tip with Ctrl+S didn't know about that...
If you had Vista SP1 you would not have any problems with hibernation :)

[burfadel]

Depends whether it was hibernation or 'suspend to ram'. A lot of people call the latter hibernation. The difference is hibernation saves the content of the RAM in to a file, which can take some time to shutdown and load up on large memory systems (still loads quicker than normal though), and the computer turns off once the file is saved. Suspend to ram (STR) works by shutting down the computer but keeping a small amount of power to the RAM, keeping the data in RAM instead of writing it to a file. This is almost instantaneous to shut down and load up. Its a feature that has to be supported by the motherboard and hardware (which all do now), and has been around since at least early XP days - just wasn't a very known feature until recently. Which method is your computer currently using? Since they work on a different principle one may be causing a problem whereas the other may not.

[Ranguvar]

If you had Vista SP1 you would not have any problems with hibernation :)

*cough*linux*cough*

[Sagekilla]

Thanks for the tip Dark Shikari. I'd love to go try this now, since I do encodes across 2 computers (the encoding computer being my laptop) and I can't leave my laptop at home since I need to bring it in to take notes when I have class.

I'm just curious if I can do something like: In the middle of an encode, ctrl+s the x264 instance, put laptop in sleep mode, then shut down desktop. When I get back home, start up the desktop and open the AvsP instance of my TCPServer(), and resume x264 on my laptop.

[Atak_Snajpera]

Which method is your computer currently using?
Both methods work.

[Inventive Software]

*cough*linux*cough*

Hibernation's broken with my Ubuntu install. :p

[LoRd_MuldeR]

One could run x264 in a virtual machine via free VirtualBox (http://www.virtualbox.org/wiki/Downloads) tool. It can save and restore the machine state...

[Quark.Fusion]

Built-in save and resume will be better, imho. Installing virtual machine just for one function is too complicated :) Checkpoints also can be useful feature if something going wrong during long encode (power outage or system crash). Sure, virtual machine can do all that, but at very high overhead (more memory, slower speed, time to setup and extra space on disk). Second, if you prefer to save encode over network you risking to broke encode during timeout or other network failure (I'm belive that x264 isn't smart enough to retry write operation).

[burfadel]

Both methods work.

I was commenting to Zyrill :) but its good to know that it doesn't affect it with the 2 different methods!

Due to the time it takes, especially with 2gb or more of RAM for hibernation, hibernation probably is really only beneficial for laptops in battery mode that shut down when the battery is low so no work is lost. STR still uses power because it keeps the RAM charged, hibernation is doesn't. The battery life is quite long in STR mode, but it still uses power and eventually it will shut off losing your work (unless is plugged in to power). For those wondering about power surges and thunderstorms etc, STR is as safe as having the computer off by shutdown (so the power is still connected), as there is always power running through the motherboard :)

[Shinigami-Sama]

One could run x264 in a virtual machine via free VirtualBox (http://www.virtualbox.org/wiki/Downloads) tool. It can save and restore the machine state...

dam beat me to it
<3 VMs

[Ranguvar]

Hibernation's broken with my Ubuntu install. :p

The problem would be that you're using Ubuntu :p

Go with a distro like Gentoo, Arch, Slackware/Slamd64, or even Debian. More minimalist, usually, and easier to get the the root of such problems. Ubuntu IMO hides too much. Certain kernel options need to be enabled to support hibernation.

Trust me, in one day with Arch, you'll learn more than a month with Ubuntu. Such was it with me, at least :) Anyone willing to read and ask questions can learn quite fast, and the feeling of power over your computer that you get is one of the best parts (I'm a control freak over my PC xD).

Then you get to wow Windows users with the likes of the text-editor-on-steroids, Vim, the lean speed of a custom kernel, and quick bash scripts that can do things in seconds that would take a very long time with a GUI.

Anyways, I agree with Quark... VMs seem cumbersome for a task so 'simple' and often-used as encoding. Plus, the overhead.

[Sagekilla]

Actually I think if you're running something as intense as a video encode, the overhead should be very tiny. I don't think x264 makes a lot of kernel calls that would necessitate translation (which would eat up cpu time)

[Ranguvar]

Actually I think if you're running something as intense as a video encode, the overhead should be very tiny. I don't think x264 makes a lot of kernel calls that would necessitate translation (which would eat up cpu time)

/me == efficiency freak :)

But your point is valid. I need to think more :p

[Quark.Fusion]

There is also usability and memory overhead.

[Ranguvar]

Are there any VMs that will let you allocate a dynamic amount of RAM? Actually, Windows would probably hate that... too used to memory hotplugging in Linux.

[Sagekilla]

If you have a nice amount of RAM (4 GB) you can just give an instance of a WinXP VM 1 - 1.5 GB of memory and not worry. Personally I don't think there any VMs that let you dynamically allocate RAM though.

/me == efficiency freak :)

But your point is valid. I need to think more :p

I am too, but most people don't realize how efficient VMs (especially with the proper hardware virtualization) are today. It would've been a very noticeable difference between native vs virtual a few years ago, but it's probably even less when you're dealing with something like x264.

[LoRd_MuldeR]

If you give ~1 GB of RAM to the VirtualBox, but the guest system doesn't use all of that yet, will it still occupy 1 GB of physical memory of the host system?

I guess Windows wouldn't keep those unused pages in the physical RAM, would it?

[zyrill]

One could run x264 in a virtual machine via free VirtualBox (http://www.virtualbox.org/wiki/Downloads) tool. It can save and restore the machine state...
Yeah, one possibly could - but apart from the speed issue, the same applies as for changing os: stupid. Why would anyone with half a mind set up a VM or another OS just so one app runs a fraction of a percent more smoothly?
burfadel: When I said hibernation, I ment it. So to clear things up: suspend to disk. :) And all that isn't even the reason I posted my initial question: I am quite versed with computers and I do not seek help to investigate why hibernation isn't working. It just isn't and I won't do anything about it since I don't need it anyway. I don't even care - I just wanted to know whether a handy stop-and-resume feature was somehow planned.
And I didn't mean something like "Ctrl+S" neither, because although I didn't know about that one, it doesn't really change the situation now, does it? I could just as well use PsSuspend!
Atak_Snajpera: And it does NOT work with my machine (apparently, else I wouldn't have brought up the issue). Also, bringing up OS-choice in this discussion is a bit stupid, really, because no one is going to change his system just because one application might run a bit smoother on that particular system. It's not like the x264-encoder would be the killer app on linux/(input your favourite system here) compared to win 32/xp/whatever.
#########################
Kinda stupid - you'll have to assent - having properly thought it through now yourself?

[Sagekilla]

Zyrill, it's not a question of how "smooth" things run, it's more of being able to start and stop an encode at will. Any one of the various choices will give roughly the same speed and all will be very smooth. The only variation really is memory usage (in the case of a VM) and how well it handles starting and stopping.


Still, I think it would be extremely useful if x264 could save it's state to a file in case it ends before the encode is finish. I could imagine something that would work well would be:

pseduocode:
save current state
Begin encoding of next frame
save current state
save encoded frame
....



I'd imagine the state file could be something as simple as "Last encoded frame: 24000," and that way you could just call it somehow like --state "encoding.state"

[DarkZell666]

You are in a particular situation in which you have few possiblities, and using a VM is one of the few (unless you're ready to drop one of your other requirements because you think it "costs" you less to drop it rather than to set up a VM; which is "only" a 1-hour job afterall).

Another solution would be to investigate the hibernation issue, but you said you don't want that, so you're stuck with "ctrl+s" and no hibernation.

It's your decision, and your decision won't change our lives.
The solutions that have been suggested aren't stupid, they're just not convenient for you :)

[Shinigami-Sama]

Are there any VMs that will let you allocate a dynamic amount of RAM? Actually, Windows would probably hate that... too used to memory hotplugging in Linux.

vmware only uses as much as it need at any given time so...

also if the VM is only doing IO theres very little overhead

also I've played many a 3d games in VMs and they work just as well in a VM as they do on a 'real' box
only thing would be using threads, you have to tell VMware how many threads to use oh noes~
*VM-iphile*

[Sagekilla]

There you go :) The only applications where VM's would be heavily constrained by a CPU intensive activity is one where it's doing a lot of read/writes. Video encoding does read writes, but it's in huge chunks, not millions of tiny ones.. That's the kind of load that a VM can virtualize with little loss in speed.

[akupenguin]

I just wanted to know whether a handy stop-and-resume feature was somehow planned.
There will never be a pause command in x264, because SIGSTOP (or OS-specific equivalent) is the correct way to do that.
There will never be a priority command in x264, because `nice` is the correct way to do that (and if your OS doesn't have a nice-equivalent, too bad).
In order to accept a suspend-to-disk command, someone would have to convince me that there's some use case where neither hibernate nor pause is the correct solution. Furthermore, checkpointing at every frame would be way more trouble than it's worth. Checkpoint at keyframes, then only ratecontrol state needs to be saved.

[zyrill]

akupenguin: I apologize for being a bit fuzzy in my statements again. I was not asking for a SIGSTOP feature in x264 - as I said, there's plenty of ways to do that in nearly every OS. Neither was I looking for a "renice"-feature since apparently, both features are quite superfluous really. I don't know a single OS that doesn't include them. What I was asking is whether something like the checkpoint feature that you mentioned would be implemented. There's a thousand reasons, really - the prominent ones like process crash, power outage, system crash, accidental user error etc. are just a short list. Anyway: it's really annoying if 50 hours or so of raw cpu time just go to waste because x264 just isn't aware that it already encoded a fraction of a clip and starts again from zero.
I don't know if this is the way to go here but the first though that comes to mind is: why not just write the output from keyframe to keyframe saving the info needed intermittently in a separate file so if the encoder process is somehow disturbed in between keyframes i can always pick up on the last keyframe and just continue encoding from there.

[akupenguin]

Ooh, there's a linux kernel patch out which supports checkpointing of arbitrary processes. The previous user-space coredump based solutions were somewhat limited. I may soon get to reject checkpointing too from x264 as having a standard solution. ;)

[Quark.Fusion]

But there is no checkpointing of arbitrary processes on windows :) (If there is please point to me)

Checkpointing at keyframes is acceptable solution if it will not allow cases when termination at saving new checkpoint will destroy also previous one :). Some stop-and-continue button will be also very useful, like Ctrl-C to encode to next checkpoint and exit and double Ctrl-C to terminate at last saved checkpoint. It will make possible multi-day encodes with handy tray gui and/or scheduler.

Avisynth isn't relible and I have situations when it crashes at near 90% of source clip. Usual case is when it run over 2gb userspace limit.

[akupenguin]

Some stop-and-continue button will be also very useful, like Ctrl-C to encode to next checkpoint and exit and double Ctrl-C to terminate at last saved checkpoint.
If the duplicate work of encoding 100 frames bothers you, you need to buy a CPU less than 10 years old.