Videos appear truncated unless with VLC and Films & TV video readers

[Mathilda]

Hello, I apologize in advance for this long message. I have videos of maximum 24h each that I suspect to be corrupted because on every video there are specifically 13h15m22s missing, no matter the length of the video. I know the ~13h part is not completely broken because I can see it in VLC media player and in the default video player in Windows (Films & TV).

But actually when I tried to check if the videos were corrupted using ffmpeg, I didn't get any issues as outputs :

ffmpeg.exe -v error -i "/Users/myname/Desktop/myvideo.mp4" -f null >error.log 2>&1
    ffmpeg.exe -v error -i "/Users/myname/Desktop/myvideo.mp4" -f null - >error.log 2>&1
    ffprobe -show_entries stream=r_frame_rate,nb_read_frames,duration -select_streams v -count_frames -of compact=p=1:nk=1 -threads 3 -v 0

I also tried to see if Python was reading my videos length correctly :

 from moviepy import VideoFileClip
    clip = VideoFileClip("C:/Users/myname/Desktop/myvideo.mp4")
    duration = clip.duration
    print ("duration : ", duration)

But there are still these 13h15m22s missing according to Python. So I still considered these videos as corrupted, and I tried to repair these videos with many ffmpeg command lines.

What I tried with ffmpeg :

I tried just copying the video to see if it solved the problem :

ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -c copy "/Users/myname/Desktop/myvideo_repaired.mp4"

But there were the ~13h missing. And apparently there was a problem with key frames so I tried re-encoding it and forcing the key frames creation :

ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -c:v libx264 -c:a copy -force_key_frames "expr:gte(t,n_forced*2)" "/Users/myname/Desktop/myvideo_repaired.mp4"

But there were still ~13h missing. And I also tried reencoding completely the video :

 ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -c:v libx264 -c:a aac "/Users/myname/Desktop/Test export vidéos/myvideo_reencoded.mp4"

And I still had the same problem. I also tried the mkv extension :

  ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -c:v libx264 -c:a aac "/Users/myname/Desktop/Test export vidéos/myvideo.mkv"

And the ~13h are still missing. I also tried to generate new timestamps, and used the `-loglevel debug` parameter to print all the hidden messages :

 ffmpeg -fflags +genpts -i "/Users/myname/Desktop/myvideo.mp4" -c copy -loglevel debug "/Users/myname/Desktop/Test export vidéos/myvideo_timestamps.mp4"

I got outputs saying "Invalid DTS/PTS Combination" and "EOF while reading input" which apparently means that there is a problem with timestamps and that the end of the input file has been reached quicker than expected. So I tried regenerating the timestamps :

  ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -vf "select='eq(pict_type,I)'" -fps_mode vfr "/Users/myname/Desktop/Test export vidéos/keyframes_%04d.png"

Which only gives me the frames as an output, with the ~13h of frames missing. And I also tried forcing the 1st frame to be a key frame :

   ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -vf "select='eq(pict_type,I)'" -force_key_frames "expr:gte(t,n_forced*10)" -fps_mode vfr "/Users/myname/Desktop/Test export vidéos/keyframes_%04d.png"

And still had ~13h of missing frames in my output frames. I eventually tried to analyse deeper the problems :

 ffmpeg -report -i "/Users/myname/Desktop/myvideo.mp4" -f null -

And got as an output hundread of lines looking like :

[null @ 0000020391541580] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1293 >= 1293

I've not listed all that I tested but this is a pretty good summary. I'm running out of ideas at this point (my last idea is explaned at the end of this message).

I'm listing here the weird points I found about these videos :

- If my video is longer than 13h15m22s, there are always the last 13h15m22s missing. Otherwise, there is no missing part.
- The 13h15m22s problem occurs with different cameras (all from the same brand, I don't have its name but I can check if it's important).
- There are always 13h15m22s missing, no matter which software I use (except for VLC and Films & TV) : ffmpeg, python (with cv2 and moviepy), Movavi (or any other video edition software), etc.
- The fps are variable and sometimes their value has a lot of decimals (like 18.122...) so maybe it can cause an issue. But if I set the broken video to constant fps it doesn't give me back the missing 13h15m22s, I just have a truncated video with fixed fps.

And here is what I think about it :

- The problem should come from within the camera encoding, since the cameras were old
- However, the truncated data is not lost because I can still read it on VLC and Films & TV apps
- But the usual ways to repair corrupted videos do not work (I tried Video Repair, ffmpeg, and Untrunc)
- The only thing I managed to do is using Untrunc to repair my 24h video using a shorter working video as reference sample (length is 4h which is < 13h15m22s so it's working well). The output video file has a length of 24h and contains all the missing frames.

But to do that I had to set the variable fps of the working 4h video to the mean fps value of the broken 24h video, using ffmpeg :

 ffmpeg -i "/Users/myname/Desktop/myvideo.mp4" -r 20 "/Users/myname/Desktop/myvideo_20fps.mp4"

And once I repaired the 24h video using the 4h video with constant fps, the "repaired" 24h video was setting the video time depending on the frames amount. For example let's say the mean fps of the 24h video was 20 fps, then the repaired video was like "ok so each 20 frames there is 1second". But this is not true : since the fps mode is variable, sometimes a second gets only 5 frames and sometimes it's 30 ! So my video eventually lengths 24h, but the seconds with a lot of frames are counted as several seconds, while the seconds with few frames are counted as less than a second.

So about this point, I was wondering if I could use ffmpeg to set the fps rate to variable for the 4h video (after setting the fps rate to constant by giving it the mean fps value of the 24h video). This would allow the 24h video to keep a variable frame rate when I use the 4h video as a sample to repair it with Untrunc. At this point this is my last hope to get back these missing ~13h, unless you guys have other guesses about this issue.

Unfortunately I cannot share my videos here since I work on confidential data, but if someone has any idea I can test it and come back to tell the result quickly !

Thanks a lot for every piece of advice !

[StainlessS]

Unfortunately I cannot share my videos

But you can presumably post output of MediaInfo, let the guys/gals see what MediaInfo thinks about the output.

[Mathilda]

Originally Posted by StainlessS

But you can presumably post output of MediaInfo, let the guys/gals see what MediaInfo thinks about the output.

Sure, I attached the MediaInfo output for a video of 21h25m37s. I hope this helps !

[Emulgator]

It is just text, just post it and wrap it in CODE tags, no need to attach it.
Camera brand/maker/model ? If it is CCTV you may need the codec that came with the cam/system, GEO (Geovision) being an notorious example.
And if playable in VLC: Convert/Save

[Mathilda]

Ok here it is in code tags :

General
Complete name                            : renard4383 Video 1 04_07_2022 10_47_16 1.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (mp42/isom/qt  )
File size                                : 39.3 GiB
Duration                                 : 21 h 25 min
Overall bit rate                         : 4 372 kb/s
Frame rate                               : 22.725 FPS
Encoded date                             : 2022-07-04 08:47:16 UTC
Tagged date                              : 2022-07-04 08:47:16 UTC

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Baseline@L4.1
Format settings                          : 1 Ref Frames
Format settings, CABAC                   : No
Format settings, Reference frames        : 1 frame
Format settings, GOP                     : M=1, N=32
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 21 h 25 min
Source duration                          : 21 h 25 min
Bit rate                                 : 4 370 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 22.725 FPS
Minimum frame rate                       : 4.524 FPS
Maximum frame rate                       : 25.175 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 1)
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.209
Stream size                              : 39.2 GiB (100%)
Source stream size                       : 39.2 GiB (100%)
Writing library                          : AVC Coding
Language                                 : English
Encoded date                             : 2022-07-04 08:47:16 UTC
Tagged date                              : 2022-07-04 08:47:16 UTC
Color range                              : Full
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
mdhd_Duration                            : 77137208
Codec configuration box                  : avcC

I tried to get the cameras brand and model but they have been threw away and no one has been able to give me this information. According to the MediaInfo output I think that the codec is avcC but I can't confirm it without the cameras.

I know this is not an optimal situation to work but I still have hope to get these data.

[Emulgator]

Full Range... and VFR...I am guessing on a CCTV / surveillance / babycam system.
well, 4Mbps @ 720p25 are rather generous for these apps, but you never know which implementation sits on the hardware encoder FPGA.

VLC -> Media -> Convert/Save

[Mathilda]

Yes, a sort of surveillance system is very possible since it is a part of my team's missions.

About the VLC part, which information do you want me to get when I'm in the Convert/Save window ? In the profile section I have several encoders available, like "Video - H.264 + MP3 (MP4)", "Video - VP80 + Vorbis (Webm)", etc. Is that the list you needed ?

Also, fyi :

• I tried the encoders "Video - H.264 + MP3 (MP4)" and "Video - H.265 + MP3 (MP4), the first one shortened my video and the second one stretched it.

• About the heigh, the videos coming from this specific camera are all 720p, but I also have videos from 3 other cameras that are 450p & 1080p. It's not my purpose to repair these videos but I'm just saying, in case you need another video size to test something.

• Also, the Mbps can change from a video to another, but with this specific camera it seems to be always ~4Mbps (while for the 3 other cameras it can be ~1 or ~8Mbps, but again I care less about these 3 cameras, it's just to let you know).

[Emulgator]

As long as you get delivered full length you just choose the encoder you see fit.
And .mp3 in avc: avoid that.
Just running a 29m59s test under profile ".mp4 capable device 720p":
VLC encodes from player output at source playing speed using iGPU,
so it doesn't look like real frameserving to me.
A good chance to get your streams rendered in one piece, methinks.
Takes a full encoding day per recorded day, though.

Test finished. Well it still came out as x264 + .mp3 in .mp4.
Well, I did not bother to make a new profile.
What iGPU was working on...can't tell. Decode ?
Not even smoothly playable at the end.
New test.

[clsid]

This can sometimes help with ffmpeg:
-ignore_editlist 1

[Mathilda]

@Emulgator Thanks for the update, I tried several encoders since yesterday and got no satisfying restults with H264+MP3 in MP4 & TS formats, VP80+MP3, and Theroa+Vorbis (OGG). There were either no output or a very bad quality video sometimes with a grey screen.

But good news : I just tried the "Video for MPEG4 720p TV/device" as you suggested. For me it runs faster than the playing speed of the video, and the video output is completely readable with no loss of quality or stretching !

I don't celebrate to quickly because I let it run only for a short time. I have to let it run on the 24 hours of videos to see if the problem is still there. But this is very promising ! I'm coming back to you with the update as soon as I got it.

@clsid thanks for this code line, I tried it this way :

ffmpeg -i "/Users/myname/Desktop/myfile.mp4" -ignore_editlist 1 -c copy "/Users/myname/Desktop/myfile_repaired.mp4"

I still have the 13h missing problem, but I don't know if I used this instruction well.

---
Update : The MPEG4 720p encoding ran for 8 hours. The 1st time I tried to open the output video it came out completely black with no metadata at all. Then I retried and the video appeared to be only 1hour 8minutes long, so it didn't work. I'm going to try other VLC encoders in the following days but I'm not very convinced about it.