Question about elementary AVC streams

[eXtremeDevil]

So, I have two .avc streams, extracted from a BD, they are both a remux, no reencode whatsoever, the number of frames are the same, and comparing side by side the frames are virtually the same, no differences at all, same source, same result.

But for some reason, the hashes of the files are different. I was wondering, why could that be?

Thanks in advance!

[Selur]

So, I have two .avc streams, extracted from a BD, they are both a remux,..

not clear what you did in detail, so my guesses are:

a. If there were encoding involved: Multithreading,.. encode the same source with x264 and use '--non-deterministic' and the outputs will not be identical.

b. If I remember correctly, even without encoding, muxing into mkv and extracting with different mkvtoolnix versions might result in non-identical outputs due to header handling,...

Cu Selur

[eXtremeDevil]

Originally Posted by Selur

not clear what you did in detail, so my guesses are:

a. If there were encoding involved: Multithreading,.. encode the same source with x264 and use '--non-deterministic' and the outputs will not be identical.

b. If I remember correctly, even without encoding, muxing into mkv and extracting with different mkvtoolnix versions might result in non-identical outputs due to header handling,...

Cu Selur

Hi, thanks for the answers, let me clarify:

a. No encoding, both should be just a remux.

b. I extracted the avc streams them with the same mkvtoolnix versions, but maybe they were muxed into mkvs before with different mkvtoolnix versions, would that affect the result? I guess it shouldn't...

It's not like I want to achieve something here, I'm just curious.

[Selur]

Yes, different version or muxing setting could alter which headers got stripped.

[Z2697]

Originally Posted by Selur

Yes, different version or muxing setting could alter which headers got stripped.

...or duplicated

I recommend YUView for "inspect the inside of the bitstream" and many other features it has.

[eXtremeDevil]

Originally Posted by Selur

Yes, different version or muxing setting could alter which headers got stripped.

I get that, but if I demux the elementary streams, shouldn't they be the same again??

[Selur]

No, afaik only the actual video data would be the same, none essential headers might be changed.

[eXtremeDevil]

Sorry, but I still don't get it. Isnt the "the actual video data" the extracted avc stream? Are you saying that the .avc file has headers inside that cannot be removed/ignored in order to create a hash for only video data?

[Selur]

SEI (supplemental enhancement information) messages do not affect the core decoding process, but can indicate how the video is recommended to be post-processed or displayed. So parts of SEI messages can be dropped, modified,... (repeated more or less frequent) when extracting the video stream.
So if you have a source stream that had the (complete) SEI data only in the first IDR frame but upon extraction is added to each IDR frame the extracted stream would be more 'robust' (to cutting/damage) than the original, while the data for the core decoding process hasn't changeg.
=> you might want to read up on SEI

[eXtremeDevil]

You mean that, even with the same extraction tool, the elementary avc stream could have different SEI/IDR data inside?

[SeeMoreDigital]

It's a shame there's not a tool for adjusting the properties of AVC and HEVC elementary streams like 'Re-stream' has been able to do for many years ago with MPEG-2 elementary streams.

[eXtremeDevil]

Originally Posted by SeeMoreDigital

It's a shame there's not a tool for adjusting the properties of AVC and HEVC elementary streams like 'Re-stream' which was available many years ago for adjusting the properties of MPEG-2 elementary streams.

Is that a yes to my last question? xD

[Z2697]

SPS and PPS can also be duplicated.

[ivanb]

You can try to open both files in the analyzer like https://media-analyzer.pro/. Perhaps you might spot some differences right away. Once you find them, it’ll be easier to fix them if needed.

[Selur]

You mean that, even with the same extraction tool, the elementary avc stream could have different SEI/IDR data inside?

If you use different version or setting in the tool: yes, you can have different SEI data.

[eXtremeDevil]

Originally Posted by Selur

If you use different version or setting in the tool: yes, you can have different SEI data.

No no, I used the exact same tool and options for extracting both .avc out of both. mkv.

[eXtremeDevil]

Originally Posted by ivanb

You can try to open both files in the analyzer like https://media-analyzer.pro/. Perhaps you might spot some differences right away. Once you find them, it’ll be easier to fix them if needed.

Great tool, but I can't see the differences side by sides, there's too many data

EDIT

I actually managed to find differences, seems like one of the streams have a few of these:

00324107b7AVC End Of Sequence #31789

Is there a way to edit them or something similar, remove that data (or whatever it is), just to check again both files and see if they have the same hash?

If not, in any case, which stream would be better, if I had to keep one?

Thanks guys!

[Z2697]

Originally Posted by eXtremeDevil

Is there a way to edit them or something similar, remove that data (or whatever it is), just to check again both files and see if they have the same hash?

FFmpeg bitstream filter "filter_units".

ffmpeg -i bfd.264 -bsf:v filter_units=pass_types=1-5 -c copy out.264

NALU type id 1-5 in AVC is VCL. The result file is not decodeable due to the lack of SPS and PPS, it only contains all of the encoded frames' data.

[ivanb]

Originally Posted by Z2697

ffmpeg -i bfd.264 -bsf:v filter_units=pass_types=1-5 out.264

I guess for this particular case we would need something like

ffmpeg -i bfd.264 -bsf:v filter_units=remove_types=10 out.264

to filter out EOS NALUs

[eXtremeDevil]

I tried both commands, and the results of the "different" avc files are identical, same hash. Can we then say the videos are virtually the same? Since one has those "End Of Sequence" data, which video could be better to keep? I don't quite understand what "End Of Sequence" data do, and if it's better or irrelevant to have it.

EDIT

BTW, why the result file is 3gb and the .avc file is 28gb?