Remuxing {~Transcoding~} AVCHD (MTS) to MP4.

[mustardman]

Hi,

I have done a bit of googling, and reading this (and other) forums, it appears to be quite well known that AVC content can be stored in a variety of containers, ranging from MKV to MTS/M2TS to MP4.

However, remuxing {~transcoding~} from one container to another does not seem to be very often done, but it can be. There seemed to be multiple solutions, which I tried with varying success. In order, these were...

Mencoder : (suggested on a Ubuntu forum). This failed with my video source, but I discovered MediaInfo (a very nice program) along the way.
MKVtoMP4 : Although it says MKV, it will also accept MTS and M2TS for remuxing {~transcoding~}. It wanted to REcode the audio until I got the settings right, but it did work with my source.
FFmpeg : This also worked with my source.

The interesting part is that both working 'solutions' drastically change the size of the video! I can see there will be a change, as containers have different overheads. However, my change is in the order of 5%, for a file with about 1500 frames!

If I have not turned you away with my lengthy intro, here is some more...
The source video is taken with a Canon G30, 1920x1080 at 50fps progressive. AVC is running 'high@L4.2', audio is of inconsequential size (AC3 for the interested).
mencoder videofile.mts -demuxer lavf -oac copy -ovc copy -of lavf=mp4 -o videofile.mp4
Mencoder didn't like it for some reason, and skipped a bunch of frames, which is why I went looking for others.

Along came MKVto MP4, and it took a few goes to just get a remux {~transcode~} going (it wanted to mess with the audio). I got it right soon enough, but the file size was way different... 90,648K to 86,197K (close enough to 5 megs). The result plays perfectly. However, no container should be that inefficent for such a short file, something must be wrong. So I went looking for something else.

Enter FFmpeg : It generated an MP4 of almost identical size... 86,197K. It also plays perfectly.
ffmpeg -i videofile.mts -vcodec copy -acodec copy videofile.mp4

I have started to look at where these bits are going missing, but so far have not proceeded very far. What I would like to do is extract the bits/bytes used to encode each frame, and see if there is a difference (I would imagine there is) at the frame level. I have tried 'Elecard Streameye', and it has some really nifty stuff in there, way over my head! But doesn't seem to allow export of frame size to a CSV (or equivilant) file. I am currently looking at FFprobe, but have not yet deciphered enough of the command line parameters it needs to be able to give me what I want.

My (half baked) theory at this stage is that the camera is stuffing a lot of extra bits into the video stream that are not strictly required to reconstruct a proper image, although why this is happening, I have not a clue.

What is going on? The operations I am doing have all been supposedly lossless. Why are so many bits being lost? And what are/were those bits supposed to do?

Anybody have any experience with "lossless" transcoding not being lossless?

MM.

EDIT: I have corrected the bad wording and marked the original {~~} (since there is no strikethough). Otherwise people may wonder what vivan and fvisagie were taking...

[Sharc]

You can't expect equal sizes, because the "overhead" is different between the various containers. A few % difference is not at all unusual.
However, if your "transcoder" is re-encoding the video and/or audio for whichever reason the file size of the re-enocoded files can be very different, depending on the encoder settings. Re-encoding should be avoided if you just want to pack the video+audio+subtitles into a new container and the new container accepts the existing formats.

[vivan]

5% is a typical overhead of transport streams.

However, transcoding from one container to another does not seem to be very often done, but it can be.Because it's called "remuxing" ;)

[fvisagie]

...transcoding from one container to another...
mencoder videofile.mts -demuxer lavf -oac copy -ovc copy -of lavf=mp4 -o videofile.mp4


Your examples and the rest of your post seem to indicate that you are actually referring to remuxing as opposed to transcoding?

I am currently looking at FFprobe

If you really need to examine file content, that's one of your better options. You may consider ExifTool (http://www.sno.phy.queensu.ca/~phil/exiftool/); see below.

And what are/were those bits supposed to do?

My guess would be that they stored container- and/or application-specific data, inside the container but external to the video and audio streams.

Remuxing a test M2TS file to MP4 and examining both with ExifTool:

(Truncating a very long listing for the input file containing data for each frame)
ExifTool Version Number : 9.60
File Name : 2014051015411110.mts
Directory : .
File Size : 43 MB
File Modification Date/Time : 2014:05:10 15:41:24+02:00
File Access Date/Time : 2014:06:11 11:49:18+02:00
File Creation Date/Time : 2014:06:11 11:49:18+02:00
File Permissions : rw-rw-rw-
File Type : M2TS
MIME Type : video/m2ts
Video Stream Type : H.264 Video
Audio Stream Type : A52/AC-3 Audio
Audio Bitrate : 256 kbps
Surround Mode : Not indicated
Audio Channels : 2
Image Width : 1920
Image Height : 1080
Time Code : 15:41:11:06
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Audio Sample Rate : 48000
Time Code : 15:41:11:05
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Time Code : 15:41:11:05
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Time Code : 15:41:11:07
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
...

(Complete listing for the output file)

ExifTool Version Number : 9.60
File Name : output.mp4
Directory : .
File Size : 41 MB
File Modification Date/Time : 2014:09:09 13:45:00+02:00
File Access Date/Time : 2014:09:09 13:45:00+02:00
File Creation Date/Time : 2014:09:09 13:45:00+02:00
File Permissions : rw-rw-rw-
File Type : MP4
MIME Type : video/mp4
Major Brand : MP4 Base Media v1 [IS0 14496-12:2003]
Minor Version : 0.2.0
Compatible Brands : isom, iso2, avc1, mp41
Movie Data Size : 42748199
Movie Data Offset : 48
Movie Header Version : 0
Create Date : 0000:00:00 00:00:00
Modify Date : 0000:00:00 00:00:00
Time Scale : 1000
Duration : 12.99 s
Preferred Rate : 1
Preferred Volume : 100.00%
Matrix Structure : 1 0 0 0 1 0 0 0 1
Preview Time : 0 s
Preview Duration : 0 s
Poster Time : 0 s
Selection Time : 0 s
Selection Duration : 0 s
Current Time : 0 s
Next Track ID : 3
Track Header Version : 0
Track Create Date : 0000:00:00 00:00:00
Track Modify Date : 0000:00:00 00:00:00
Track ID : 1
Track Duration : 12.96 s
Track Layer : 0
Track Volume : 0.00%
Matrix Structure : 1 0 0 0 1 0 0 0 1
Image Width : 1920
Image Height : 1080
Media Header Version : 0
Media Create Date : 0000:00:00 00:00:00
Media Modify Date : 0000:00:00 00:00:00
Media Time Scale : 90000
Media Duration : 12.96 s
Media Language Code : und
Handler Type : Video Track
Handler Description : VideoHandler
Graphics Mode : srcCopy
Op Color : 0 0 0
Compressor ID : avc1
Source Image Width : 1920
Source Image Height : 1080
X Resolution : 72
Y Resolution : 72
Bit Depth : 24
Video Frame Rate : 50
Track Header Version : 0
Track Create Date : 0000:00:00 00:00:00
Track Modify Date : 0000:00:00 00:00:00
Track ID : 2
Track Duration : 12.99 s
Track Layer : 0
Track Volume : 100.00%
Matrix Structure : 1 0 0 0 1 0 0 0 1
Media Header Version : 0
Media Create Date : 0000:00:00 00:00:00
Media Modify Date : 0000:00:00 00:00:00
Media Time Scale : 48000
Media Duration : 12.99 s
Media Language Code : und
Handler Type : Audio Track
Handler Description : SoundHandler
Balance : 0
Audio Channels : 2
Audio Bits Per Sample : 16
Audio Sample Rate : 48000
Handler Type : Metadata
Handler Vendor ID : Apple
Encoder : Lavf55.10.100
Avg Bitrate : 26.3 Mbps
Image Size : 1920x1080
Rotation : 0


In this example it seems that at least camera settings data wasn't copied across during the remux. Possibly the MP4 container doesn't support it.

If your interest is not so much determining the purpose of the removed content as it is confirming that video and audio content have been copied verbatim, instead use something like Avisynth's Compare(). As you probably noticed however, the remux by ffmpeg was lightning-quick; most certainly no re-encoding was involved. Therefore, the only other possible source of any changes would be corruption during the copying of frames, and chances of such a bug having gone undiscovered all these years is nil. Therefore, investigating this (for the purpose of verifying video and audio integrity) is a waste of time :).

[Sharc]

For a quick check you may also open the container in MediaInfo (http://www.videohelp.com/tools/MediaInfo). It will list the files which are packed in the container, together with more information about the files like size and settings. You should easily see if the remuxing should have skipped or altered (re-encoded) any files.

[mustardman]

Yes, you are perfectly correct, and I should have realised it a lot sooner, like when I was typing... the operation I am speaking of is indeed remuxing, and not transcoding! Doh.

I am a bit rushed at the moment, but will look further into this later today. I think a detailed reply is in order (and polite too).

Several comments to follow...

[mustardman]

A more comprehensive reply...

@vivan : Yes, a Transport Stream will have large overheads, after all a TS is meant to be almost independant of where in the stream you start, it doesn't have the luxury of a file header. But for a mere (less than) 1500 frames? Over 3K of extra info per frame? Is that reasonable (asking, not nit-picking)?

@fvisagie : I have used EXIFtool in the past (on video & images), and found it to be just fantastic. It would be nice to include it in MediaInfo. However, on this occasion I have found MediaInfo more informative. FFmpeg was indeed lightning quick, and MKVtoMP4 not quite as quick. Simply going on the time it took for processing, I suspected that no re-coding was done.

@Sharc : MediaInfo was where I first noticed some really strange things...

I have attached a PDF of MediaInfo output with all three files side by side (it makes comparison much easier than scrolling up and down). Some fields are greyed out, as they weren't listed for that format. The items in red are what caught my eye.

The very first thing I noticed was the file size in explorer, so I used MediaInfo to get more details.
Second I noticed the video duration... in the remuxed files it was longer. Interesting...
Third, for both MP4 versions, the overall bitrate is less than the source (MTS). Could this all be TS container overhead?

But then I looked at the video bit rate and video stream size... the values for each MP4 file are significantly different (especially interesting since the overall file size is extremely close). The audio streams are reported identical for all three, so no contribution there.

How accurate/reliable are these figures? I don't believe there is a bug in FFmpeg. I am not a coder, and never will be. All the numbers reported are calculated at some stage, whether by the video camera, FFmpeg when it remuxes, or MediaInfo when it reads the file. Could it simply be a matter of MediaInfo calculating some fields (that are not stored in the file) based on other fields put there by other entities, and hence getting different values?

If I get a chance, I'll try my luck with FFprobe and see how I get along.

MM

[fvisagie]

In this example it seems that at least camera settings data wasn't copied across during the remux. Possibly the MP4 container doesn't support it.

It turns out my guess was incorrect; camera settings data is preserved. Remuxing back to M2TS and examining with ExifTool:
ffmpeg -i 2014051015411110.mts -c copy avchd.mp4
...
ffmpeg -i avchd.mp4 -c copy -f mpegts -bsf h264_mp4toannexb avchd.mp4.mts
...
exiftool -a -ee avchd.mp4.mts
ExifTool Version Number : 9.60
File Name : avchd.mp4.mts
Directory : .
File Size : 44 MB
File Modification Date/Time : 2014:09:10 12:05:32+02:00
File Access Date/Time : 2014:09:10 12:05:32+02:00
File Creation Date/Time : 2014:09:10 12:05:32+02:00
File Permissions : rw-rw-rw-
File Type : M2T
MIME Type : video/mpeg
Audio Stream Type : A52/AC-3 Audio
Image Width : 1920
Image Height : 1080
Time Code : 15:41:11:06
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Time Code : 15:41:11:05
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Time Code : 15:41:11:05
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
Time Code : 15:41:11:07
Date/Time Original : 2014:05:10 15:41:11+02:00
Aperture Setting : 4.0
Gain : 0 dB
Exposure Program : Program AE
White Balance : Auto
Focus : Auto (0)
Image Stabilization : Off
Exposure Time : 1/100
Make : Canon
Recording Mode : Unknown (8)
Capture Frame Rate : 50
Video Frame Rate : 50
...

This shows that camera settings data is preserved inside the video stream. The difference in file size after remuxing is therefore purely due to multiplexer overhead as mentioned earlier.

Here's an even more instructive comparison: demuxing the video tracks from the original M2TS and remuxed MP4, followed by a comparison.
ffmpeg -i 2014051015411110.mts -c:v copy -an 2014051015411110.m4v
...
ffmpeg -i output.mp4 -c:v copy -an output.m4v
...
fc /b 2014051015411110.m4v output.m4v
Comparing files 2014051015411110.m4v and OUTPUT.M4V
FC: no differences encountered
The two video streams from before and after remuxing are byte-identical.

[mustardman]

@fvisagie : That is a cool idea indeed. However, if FFmpeg was used to generate the MP4 stream (video+audio+data) you have, then it would follow that it should extract the exact same data from the generated MP4 as it did when doing the M2TS -> MP4 -> video only.

But... using the other program (MKVtoMP4) - as long as it uses different libraries to do the extraction - and looking for a difference...

Judging from the almost identical sizes I got using FFmpeg & MKVtoMP4, it is either using the same library to extract, or there is indeed no difference in the video.

I should try this myself, so I am just rambling here, but can FFmpeg take an MP4 and remux it back into a MTS/M2TS...?

MM.

[fvisagie]

I should try this myself, so I am just rambling here, but can FFmpeg take an MP4 and remux it back into a MTS/M2TS...?

From above:

Remuxing back to M2TS and examining with ExifTool:

...
ffmpeg -i avchd.mp4 -c copy -f mpegts -bsf h264_mp4toannexb avchd.mp4.mts
...

[mustardman]

Obviously you know FFmpeg much better than I ever will. :)

I ran your suggestion, and I must say that I was very surprised that the resulting file came out larger than the original!

Original MTS (90.7M) -> Remux to MP4 (86.2M) -> Remux to MTS (93.1M).

At least FFmpeg gave me an estimation of the overhead, which was much higher than I expected... around 8%. I am no stream expert, but that seems an awful lot. I would have expected a modern bitstream format to be a little more efficent! After all, when we are talking high power transmitters, bits cost money!

So, I can see that there will be differences in the overhead between the formats, even between the two MTS files - there are no doubt things the camera does & does not do that the software does or does not do. I am OK with that.

Despite that, I still wanted to check the frame sizes, to see if they were the same throughout, and according to the program I was using (Elecard Streameye 1.1), they are different. Not just one every so often, but every single one.

Orig
MTS ->MP4 ->MTS
FFmpeg FFmpeg

Elecard Streameye v1.1

1 I 441216 417024 449320
2 B 25152 23130 24816
3 B 33408 31576 34404
4 P 99456 93718 103588
5 B 36480 34541 37036
6 B 37056 34170 37036
7 P 117120 110634 121636
8 B 35520 33242 36096
9 B 33600 31956 34404
10 P 119424 112315 123704
11 B 33024 30784 33276
12 B 31680 29357 31584
13 P 120000 113577 125208
14 B 31104 29477 31772
15 B 30336 28733 31208
16 P 123840 116244 125396
17 B 29952 27274 31960
18 B 28416 27032 28952
19 P 119232 112832 122012
20 B 28032 26418 30644
21 B 27264 25320 27448
22 P 118080 110962 119756
23 B 28992 26893 31584
24 B 28416 26819 29516
25 I 350784 332014 357952
26 B 27456 26000 30268
27 B 28608 26272 28388
28 P 126720 119073 128592
29 B 25920 24484 28952
30 B 26880 25229 27260

Now it could be Elecard is getting it wrong, likely with MTS/MP4, possibly even with the two MTS streams, and I am going to try something else (maybe FFprobe) to get some numbers.

But back to my original query, and actually adding to it... are bits being stripped out of (or pushed back into) the video stream? Yes, the resulting streams play fine, and I can see no difference, but IS there one?

MM

[fvisagie]

But back to my original query, and actually adding to it... are bits being stripped out of (or pushed back into) the video stream? Yes, the resulting streams play fine, and I can see no difference, but IS there one?

From above:

Here's an even more instructive comparison: demuxing the video tracks from the original M2TS and remuxed MP4, followed by a comparison.
ffmpeg -i 2014051015411110.mts -c:v copy -an 2014051015411110.m4v
...
ffmpeg -i output.mp4 -c:v copy -an output.m4v
...
fc /b 2014051015411110.m4v output.m4v
Comparing files 2014051015411110.m4v and OUTPUT.M4V
FC: no differences encountered
The two video streams from before and after remuxing are byte-identical.

[SeeMoreDigital]

Original MTS (90.7M) -> Remux to MP4 (86.2M) -> Remux to MTS (93.1M).
How about uploading your AVCHD source somewhere, so others can try re-muxing it to .MP4?

[Sharc]

... At least FFmpeg gave me an estimation of the overhead, which was much higher than I expected... around 8%. I am no stream expert, but that seems an awful lot. I would have expected a modern bitstream format to be a little more efficent! After all, when we are talking high power transmitters, bits cost money!

Transport containers like .ts and .m2ts add more overhead to the stream than other containers. About 8% seems right, but it depends also on the packing of the particular audio format. The larger overhead makes these streams generally more robust against errors (bit errors, short signal interruptions ...) which may be introduced along the transport path from the TV station to your home TV. In case of a signal failure the stream can quickly re-synchronize for example when the signal returns. Hence the "extra" overhead is not just a waste of precious bits.

[mustardman]

@Sharc
True. Having done some error correction stuff, I know how much overhead it generates. When put in terms of error correction and 8% overhead (even without other bits for sync, etc), then 8% actually seems rather small!

@SeeMoreDigital
It is a big file, so I'll have to revert back in a few days... I might get a shorter clip - I am pretty confident it will exhibit the same characteristics.

@fvisagie
I did read that particular comment (unlike your other one about remuxing from MP4 back to MTS:rolleyes:)
But the reason I did not do this is because it uses the same remuxing code (unless I am really mistaken) as was used to strip the video from the MTS and put it into a MP4 container in the first place. It makes sense that using the same process that generates an "error" to check if there was an "error" gives 'no change'. Using a cheap calculator to evaluate 1+2x3 (=9) and using the same cheap calculator to verify 1+2x3 (=9 again) would suggest the answer is right - even though we both know it isn't.
Note: I am not saying FFmpeg has any errors in it, only that you should never use the same process to both determine and verify an answer. I've fallen into that trap before.
This is part of my motivation to find other "tools" (even if they are quite manual - to get frame sizes from Elecard was typing numbers into a spreadsheet). I don't know how FFprobe will do, and I haven't had a chance to try it yet, because it - like the example above - will probably use the same libraries as FFmpeg did to remux in the first place.

Please note that I am not trying to be argumentative, difficult or pig-headed about this thing. I started off wanting to know why there was such a big difference in the file size (and it seems overhead is the answer), but started to get really curious when different frame sizes started popping up.

Cheers,
MM.

[poisondeathray]

If you want to prove it to yourself to a per frame md5 check on each. These are decoded to uncompressed video and compared. If the actual video streams are differerent in any meaningful way (i.e not just padding , header differences, metadata, or overhead) , then the hash check will be different . Demuxing and remuxing don't always give the same results from a basic file compare, because sometimes sequence header data is altered (e.g. in MKV container)

ffmpeg -i input.mts -f framemd5 output_mts.framemd5
ffmpeg -i input_ffmpeg.mts -f framemd5 output_ffmpeg.framemd5
ffmpeg -i input_ffmbc.mts -f framemd5 output_ffmbc.framemd5

Alternatively, you can do PSNR tests comparing to the original. ffmpeg can do this now as well

[fvisagie]

Demuxing and remuxing don't always give the same results from a basic file compare, because sometimes sequence header data is altered (e.g. in MKV container)

Indeed. The suggested comparison above uses raw MPEG-4 video output format in an attempt to minimise the likelihood of that.

[poisondeathray]

Indeed. The suggested comparison above uses raw MPEG-4 video output format in an attempt to minimise the likelihood of that.

That's only works in this specific case because he used a transport stream. I was suggesting a procedure for more general use. The headers aren't stripped when demuxed from a transport stream. For example, if you used raw MPEG-4 stream from a MKV container, the file would be detected as "different" using fc because the sequence headers will have changed . per frame MD5 or PSNR will detect them as the same, because decoded image is the same

[fvisagie]

Thanks for teaching me something new :).