x264 junk location

[javlak]

Does anyone know where the junk (video headers/footers/info sections) is located in raw x264 video files?

Thanks in advance!

Related to: x265 junk location (https://forum.doom9.org/showthread.php?t=170843)

[LoRd_MuldeR]

Not quite sure what you mean :confused:

But if you are referring to the extra "meta" info x264 embeds into H.264 bitstream (like the encoder parameters that where used), it's stored as custom Supplemental Enhancement Information (SEI) message.

See H.264 standard for details...

[javlak]

Yes, this is what I refer to.

Let me give you an example. Say that I want to encode a video using x264. But I'm not too keen on leaving my PC working on 100% (or close) CPU usage for extended periods of time so I encode just the first 1000 frames of a video. Next day I encode the next 1000 frames and so on.

In theory, I could just concatenate all these files to one file with no problem, but I'd have some extra overhead due to the extra (unnecessary I guess) SEIs and VUIs. That's what I'm interested in; avoiding those "extra" recurring bits of info. They don't hurt, but not all of them are really necessary after the concatenation. They should be in the first video but I guess it would be more efficient to extract them from the rest of the video chunks for efficiency before the concatenation. Perhaps too much work for not that big of a difference in file size, but too many chunks per encoding session, and a few concatenated videos later it would show.

So I guess what I'm looking for is the location of the VUI and SEI, in a raw x264 file. If anyone knows, of course.

[Groucho2004]

I'm not too keen on leaving my PC working on 100% (or close) CPU usage for extended periods of time
Why? If you keep the CPU temperature below the manufacturer recommended values, you can run it at 100% almost indefinitely.

[LoRd_MuldeR]

But I'm not too keen on leaving my PC working on 100% (or close) CPU usage for extended periods of time
Why? If you keep the CPU temperature below the manufacturer recommended values, you can run it at 100% almost indefinitely.

Yeah, CPU's must definitely be able to run many hours under 100% load without any problems - at least as long as you are not "overclocking" manually.

And even if the CPU gets too hot, the only thing that will happen in the first instance is that it will reduce the clock speed. If that happens more than just occasionally, you either have an improper CPU cooler, the CPU cooler was installed wrongly - both of which is a support case - or you eagerly need to clean the dust out of your fans ;)

In theory, I could just concatenate all these files to one file with no problem, but I'd have some extra overhead due to the extra (unnecessary I guess) SEIs and VUIs.

The size of the custom SEI message is negligible, so why worry?

So I guess what I'm looking for is the location of the VUI and SEI, in a raw x264 file. If anyone knows, of course.

The location is not fixed, I suppose. So you will have to parse the H.264 bitstream until you find what you are looking for. Look at the H.264 specifications for details!

Of course it would be much easier to simply never create the custom SEI message. Modifying x264 in that way is trivial, but nobody here will give your detailed instructions how to do this (the developers added the message for a reason).

As said before, it's probably not worth the effort to save a few bytes...

[raffriff42]

I'm not too keen on leaving my PC working on 100% (or close) CPU usage for extended periods of timeIf you can't get your temps under control (laptop?), or you simply want to multitask while encoding, you can: set core affinity (https://www.google.com/search?q=set+core+affinity) to half your cores - instant 50% max usage, or
use CPU throttling with Battle Encoder Shirase (https://www.google.com/search?q=Battle+Encoder+Shirase).

[nhakobian]

If you can't get your temps under control (laptop?), or you simply want to multitask while encoding, you can: set core affinity (https://www.google.com/search?q=set+core+affinity) to half your cores - instant 50% max usage, or
use CPU throttling with Battle Encoder Shirase (https://www.google.com/search?q=Battle+Encoder+Shirase).

Another option is just tell x264 to force a specific number of encoder threads. While it won't force all the encoding to occur on a specific core (the OS can schedule the thread to run on any core it wishes), it would reduce the overall processing done.

[Groucho2004]

Another option is just tell x264 to force a specific number of encoder threads.
And clearly the simplest and smartest option if lower CPU usage is required.

[LoRd_MuldeR]

And clearly the simplest and smartest option if lower CPU usage is required.

If he actually is concerned not about overheating but about x264 slowing down his other programs, then limiting the CPU usage to an arbitrarily chosen percentage is the wrong thing to do! A CPU usage below 100% simply means that some CPU cycles are wasted unused. Instead, he should simply lower the priority of the x264 process to "below normal" or even "idle". This way, no CPU cycles will be wasted pointlessly and x264 will not "steal" any CPU cycles from higher priority processes.

[Groucho2004]

Instead, he should simply lower the priority of the x264 process to "below normal" or even "idle". This way, no CPU cycles will be wasted pointlessly and x264 will not "steal" any CPU cycles from higher priority processes.
Wrong. I always have x264 running at idle priority, makes no difference in CPU power consumption. If the cycles are available, the process (x264) will gladly take them, idle priority or not.

[raffriff42]

If he actually is concerned not about overheating but about x264 slowing down his other programs, then limiting the CPU usage to an arbitrarily chosen percentage is the wrong thing to do!I believe Battle Encoder Shirase (BES for short) works by suspending individual processes for short periods; for 10% throttling it might suspend the selected process for 3 ms every 30 ms.

[LoRd_MuldeR]

I always have x264 running at idle priority, makes no difference in CPU power consumption. If the cycles are available, the process (x264) will gladly take them, idle priority or not.

Exactly! And that's what it's supposed to do. If you are surprised by that, you haven't understood what process priorities are. As said before, CPU usage below 100% means that you are wasting CPU cycles for nothing - unless there really is no process on your system that could use those cycles for something, of course. So using strange tools or workarounds to keep the CPU usage below some arbitrarily chosen percentage is almost always a really bad idea. Priorities is what you want here! But priorities are not about bounding the "CPU usage" of a process. They are about which process gets served first, if multiple processes are ready to execute at the same time. So by setting the priority of x264 to "idle" the x264 process will perfectly be allowed to use ~100% of the CPU time as long as no other process needs those CPU cycles. But as soon as some other process (which is not running at "idle" priority - so pretty much any process) needs some CPU cycles, that process will be served before x264. Then x264 only gets the CPU cycles that the other process(es) couldn't use. Again, that's exactly what you want here: Let x264 use as much CPU cycles as possible but without slowing down other processes.

BTW: For the same reason, setting the priority of x264 to "high" usually doesn't make the encoding process any faster at all - at least when x264 is the only CPU intensive process running on the system.

I believe Battle Encoder Shirase (BES for short) works by suspending individual processes for short periods; for 10% throttling it might suspend the selected process for 3 ms every 30 ms.

Yeah, and that's usually a stupid thing to do - for people who haven't understood how process priorities (are supposed to) work ;)

[Groucho2004]

Exactly! And that's it's supposed to do. If you are surprised by that, you haven't understood what process priorities are. As said before, CPU usage below 100% means that you are wasting CPU cycles for nothing - unless there really is no process on your system that could use those cycles, of course. So using strange tools or workarounds to keep the CPU usage below some arbitrarily chosen percentage is almost always a really bad idea. Priorities is what you want here! But priorities are not about bounding the "CPU usage" of a process. They are about which process gets served first, if multiple processes are ready to execute at the same time. So by setting the priority of x264 to "idle" the x264 process will perfectly be allowed to use ~100% of the CPU time as long as no other process needs those CPU cycles. But as soon as some other process (which is not running at "idle" priority - so pretty much any process) needs some CPU cycles, that process will be served before x264. Then x264 only gets the CPU cycles that the other process(es) couldn't use. Again, that's exactly what you want.
You're preaching to the choir here and completely missing the point. Setting idle priority for a CPU heavy process like x264 will not reduce power consumption. Therefore, the CPU will get just as hot which the OP wants to prevent for some reason.

[LoRd_MuldeR]

You're preaching to the choir here and completely missing the point.

No, I don't think I'm missing the point. It's not clear (yet) what OP wants exactly. And I made clear to what assumption I was referring to:
If he actually is concerned not about overheating but about x264 slowing down his other programs, then...


Setting idle priority for a CPU heavy process like x264 will not reduce power consumption.

Exactly. And there should be no reason to want that.

Therefore, the CPU will get just as hot which the OP wants to prevent for some reason.

I think we already agreed ~10 posts ago that, if he really is concerned because his CPU is getting too hot, then what he actually needs to do is fixing his hardware problem - rather than trying to workaround the symptom by silly things, such as limiting the "CPU usage" of the x264 process or running his encodes in small chunks...

[foxyshadis]

My hardware problem is lack of A/C at home and encoding turning my laptop into a space heater until the air cools down enough to open a window, so I can understand other reasons. :p Even then, it's better to simply suspend the process in process explorer, x264 won't care. It's much safer to put the system to sleep with x264 suspended, as well; less process crashes and system hangs.

You can also throttle the CPU usage just by choosing a maximum battery life power plan. Those force the CPU to never go above idle speed, and do a great job of minimizing heat. You don't need a specialty encoder like Battle Encoder Shirase unless you want maximum performance to the rest of the system.

I posted a method to solve the OP's problem in the HEVC thread he cross-posted to, but realistically, even with low-bitrate SD video the headers will never amount to anything. They're freaking tiny.

[javlak]

Yeah, thank you foxyshadis for the truly detailed reply back in the x265 forum. That was exactly what I was looking for.

The reason why I posted in both x265 and the x264 forums is that the supplemental info for each is different, so I wanted to get info for both of them. I didn't want to post a question on x264 in the x265 forums and vice-versa.

Back on topic, I am not really concerned about CPU usage per se, but I had to produce an example in order to clarify what I needed to do, to get a clear answer. Truth is that I am working on a project that I intend to publish some time later, but it's still in its infancy and I don't want to get people anticipating something that might appear in who knows how many months down the line (if at all).

Thanks to all of you and especially foxyshadis.

[delacroixp]

Say that I want to encode a video using x265. But I'm not too keen on leaving my PC working on 100% (or close) CPU usage for extended periods of time so I encode just the first 1000 frames of a video.
Next day I encode the next 1000 frames and so on.
Encoding is one of the most parallel processing jobs out there (certainly the video component).
One would think it would be easy to incorporate a pause or 'stop and go' option.

Perhaps not.


It's all good
:):devil::D
Pascal