x265 - A question on -info and -repeat-headers

[hellgauss]

I have the following issue while encoding in x265 hdr10:

I do want info nal unit at the beginning of file, i.e. encoding setting log as for -info option. However, to avoid redundance, I just want it once. I use the following command line:


start /b /low /wait cmd /c "ffmpeg -threads 1 -y -hide_banner -nostdin -bitexact -i "hdr10.avs" -map 0 -profile:v main10 -sws_flags accurate_rnd -c:v libx265 -preset fast -crf 16.2 -x265-params "psy-rd=0.8:psy-rdoq=0.9:deblock=0,0:no-sao=1:no-strong-intra-smoothing=1:rc-lookahead=99:bframes=16:vbv-maxrate=80000:vbv-bufsize=90000:level-idc=51:high-tier=1:hdr10-opt=1:hrd=1:chromaloc=2:sar=1:colorprim=bt2020:colormatrix=bt2020nc:transfer=smpte2084:range=limited:max-cll=1000,159:master-display=G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(10000000,50)" -bitexact "t1.hevc" 2>&1 | tee "t1_log.txt""


Even adding, for test, -no-repeat-headers is useless since x265 ignores the option, and however I need other nals to be repeated. The encoding setting are reported every key frame, which I do not want. (Tested with hevcesbrowser tool @github )

MY QUESTIONS (in order of priority)

- Is there a command line setting to disable -repeat-headers only for the "info" nal unit?

- Is there a hack or a third party tool to do so? For example, with SDR, headers are automatically repeated by the muxer.

- I can write simple scripts in c/c++, and it seems that the "info" nal unit is identifyed by the following 6-bytes hex string in the raw hevc stream

00 00 01 (new nal)
4E 01 (header)
05 ("info" identyfier - user data unregistered)

Is it safe to look for that string and just remove it and everything up to next 00 00 01 (new nal), except for the first occourrence?


Thank you very much for any help!

[benwaggoner]

I have the following issue while encoding in x265 hdr10:

I do want info nal unit at the beginning of file, i.e. encoding setting log as for -info option. However, to avoid redundance, I just want it once. I use the following command line:



Even adding, for test, -no-repeat-headers is useless since x265 ignores the option, and however I need other nals to be repeated. The encoding setting are reported every key frame, which I do not want. (Tested with hevcesbrowser tool @github )

MY QUESTIONS (in order of priority)

- Is there a command line setting to disable -repeat-headers only for the "info" nal unit?

- Is there a hack or a third party tool to do so? For example, with SDR, headers are automatically repeated by the muxer.

- I can write simple scripts in c/c++, and it seems that the "info" nal unit is identifyed by the following 6-bytes hex string in the raw hevc stream

00 00 01 (new nal)
4E 01 (header)
05 ("info" identyfier - user data unregistered)

Is it safe to look for that string and just remove it and everything up to next 00 00 01 (new nal), except for the first occourrence?
I've not tried for a while, but my conclusion some years back was that --no-info was pretty much required to use --repeat-headers to get the HDR metadata at the start of every GOP.

[Z2697]

FFmpeg forces repeat-headers on libx265. (no matter HDR or not)

[Z2697]

It should be safe to just remove the bytes, but you need to look for more signatures: the type UUID, because SEI have several types, while removal of them can be safe as well, I think it should be better avoided.

The better solution is to modify the x265 source code to make it no longer repeat that SEI, and maybe submit a patch to the develop team.

[hellgauss]

Thanks @benwaggoner and @Z2697 for answers. So it seems that the answer for the two questions is no, and only third option is left.

@Z2697
For sdr x265 there is no problem: ffmpeg repeats header automatically (at least I check with muxed output .mp4 and .mkv). However only needed headers are repeated. For hdr it is libx265 which automatically repeat unnecessary headers.

I think that UUID parse is not necessary. With standard x265 it seems there is no way to add extra user-data other than info, at least with my standard settings. My concern was about some nal index or counting info somewhere else in the raw stream, but it seems to not be the case.

I will follow the third way [ aargh writing code :( ] , if I succeed I will share the script.

[Z2697]

There're a few options in x265 that can add additional SEI but normally you won't use them.
The most likely ones are HDR cll and master-display, but it's not like they are very critial for the displaying actually.

Try search SEI in the "x265 --fullhelp" output.

And the forced repeat-header, sorry I must have mixed up the different situations, it's only forced when using "hevc bitstream" output. (If you try to disable it using x265-params you get bitstream without any parameter sets)

[benwaggoner]

There're a few options in x265 that can add additional SEI but normally you won't use them.
The most likely ones are HDR cll and master-display, but it's not like they are very critial for the displaying actually.
While they aren't critical, there are enough displays that do something with them that you don't want to not have them. There have been displays that treated "no value" as "could be up to 10,000" and dimmed the image.

[hellgauss]

There're a few options in x265 that can add additional SEI but normally you won't use them.
The most likely ones are HDR cll and master-display, but it's not like they are very critial for the displaying actually.


No, I have checked, at least with simple HDR10 without DoVi. The id for SEI_PREFIX is 39_[Decimal] and it is in bit[1-6] of 4E in the header. The further byte is "last_payload_type_byte", and it is 05 only for user data (info). Other SEI_PREFIX which are in the hdr10 stream are, in decimal:

last_payload_type_byte:
144 - cll info
137 - Master Display info
129 - Other info
0 - Other info
1 - Other info (this is signaled for each frame)

Did not check HLG since header are not automatically inserted and needed, so no problem.

[hellgauss]

It seems to work...

https://github.com/HG3112/x265-info-remove

It is a simple 100-line single-file cpp script. Compiled version in release.

USAGE:

A) x265-info-remove.exe inputFile.hevc outputFile.hevc
B) x265-info-remove.exe all inputFile.hevc outputFile.hevc

The script looks for each occurence of the hex string 00 00 01 4E 01 05 in inputFile and removes it, together with following bytes up to next 00 00 01

If the option 'all' is not specified, the first occurrence is kept

The script is useful to remove unwanted info user-defined nal-units (encoding settings) from raw x265 hevc stream

WARNING: use only on raw .hevc stream ( NOT .mkv NOT .mp4 )

[Z2697]

If you are OK with modifing x265 source code and build it and FFmpeg yourself:
In source/encoder/encoder.cpp starting from Line 3380
You can add

m_param->bEmitInfoSEI = 0;

inside this block to make the "info" option turn itself off after it's being triggered once.

if (m_param->bEmitInfoSEI)
{
char *opts = x265_param2string(m_param, m_sps.conformanceWindow.rightOffset, m_sps.conformanceWindow.bottomOffset);
if (opts)
{
char *buffer = X265_MALLOC(char, strlen(opts) + strlen(PFX(version_str)) +
strlen(PFX(build_info_str)) + 200);
if (buffer)
{
snprintf(buffer, strlen(opts) + strlen(PFX(version_str)) + strlen(PFX(build_info_str)) + 200,
"x265 (build %d) - %s:%s - H.265/HEVC codec - "
"Copyright 2013-2018 (c) Multicoreware, Inc - "
"http://x265.org - options: %s",
X265_BUILD, PFX(version_str), PFX(build_info_str), opts);

SEIuserDataUnregistered idsei;
idsei.m_userData = (uint8_t*)buffer;
idsei.setSize((uint32_t)strlen(buffer));
idsei.writeSEImessages(bs, m_sps, NAL_UNIT_PREFIX_SEI, list, m_param->bSingleSeiNal);

X265_FREE(buffer);
}

X265_FREE(opts);
}
}

[hellgauss]

I build my modded ffmpeg myself for another project of mine @github. However i followed a step-by-step tutorial. The build lacks a lot of features, and I use it only for the extra option I needed, I do not encode with that. Furthermore I would have problems with licensing and package distribution. I'm not that expert in programming.

Since on 4K I get Second per Frames encoding, it is not important to wait an extra minute to parse the output. I feel more safe with my script (also: is "if (m_param->bEmitInfoSEI)" used elsewhere in ffmpeg code?).

I would try to submit a bug report / feature request to x265 teams, but it seems difficult to submit it, at least for me.

Thanks anyway for your suggestion!