Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 31st August 2020, 04:35   #1  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Is there a way to recover an interrupted ProRes 4444 encoding?

I was running a Vapoursynth script for about 7 hours with a few more hours to go when I got the BSOD.

Is there a way to recover the frames that were already encoded?

FFMpeg message I get is:

Code:
moov atom not found
video.mov: Invalid data found when processing input

Last edited by Cary Knoop; 31st August 2020 at 04:51.
Cary Knoop is offline   Reply With Quote
Old 31st August 2020, 10:21   #2  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 184
You are very lucky, prores frames have size and signature at start. so you can recover it easily with little of coding. Is there audio too?

Next time use better container for encoding, like mkv.
richardpl is offline   Reply With Quote
Old 31st August 2020, 16:12   #3  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Quote:
Originally Posted by richardpl View Post
You are very lucky, prores frames have size and signature at start. so you can recover it easily with little of coding. Is there audio too?
Ok, so how do I recover?

No audio.
Cary Knoop is offline   Reply With Quote
Old 31st August 2020, 16:19   #4  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 184
I'm unaware of existing tool that does it already. But as already said, program can be coded and build in few minutes and than it will recover all video frames in few seconds when run on such file.
richardpl is offline   Reply With Quote
Old 31st August 2020, 18:58   #5  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,031
Quote:
Originally Posted by Cary Knoop View Post
Ok, so how do I recover?

No audio.
Remuxing it through mp4box might do the trick. Or with ffmpeg.

Some tools add audio while writing video, others may do all the audio at the end. The .mov file format is extremely flexible. If there is an audio track in the file, remuxing might salvage that as well.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 31st August 2020, 19:06   #6  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Quote:
Originally Posted by benwaggoner View Post
Remuxing it through mp4box might do the trick. Or with ffmpeg.
I tried that without success.

It's not the end of the world, just a lot of wasted electricity and time.
Cary Knoop is offline   Reply With Quote
Old 31st August 2020, 19:37   #7  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,536
I tried different tools for the exact same problem but it was a mp4 instead of a mov. Never could rescue anything. That is why I use mkv now, lesson learned!
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository
ChaosKing is offline   Reply With Quote
Old 31st August 2020, 22:01   #8  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Quote:
Originally Posted by ChaosKing View Post
I tried different tools for the exact same problem but it was a mp4 instead of a mov. Never could rescue anything. That is why I use mkv now, lesson learned!
Cool, using Matroska works!
Interrupting the encoding still allows viewing the already encoded frames!

Code:
ffmpeg -i test.mov -c:v prores_ks  -profile:v 4444 -pix_fmt yuva444p10le test.mkv
Cary Knoop is offline   Reply With Quote
Old 1st September 2020, 21:25   #9  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,601
If mov is readable by ffmpeg then you should not need to re-encode. You can try to use copy.

Last edited by kolak; 1st September 2020 at 21:29.
kolak is offline   Reply With Quote
Old 2nd September 2020, 00:41   #10  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,233
Quote:
Originally Posted by Cary Knoop View Post
Cool, using Matroska works!
Interrupting the encoding still allows viewing the already encoded frames!
It's because in MP4/MOV all the frames/samples are actually stored in one giant monolithic block, that contains no "packet headers", "sync bytes" or whatsoever to indicate the frame boundaries

The really only way (except maybe by using some heuristics) to locate individual frames within a MP4/MOV file is by using the "sample size", "sample to chunk" and "chunk offset" tables.

Unfortunately, these tables usually will be written at the very end of the file, after all the frames have been written. So, if the encoding process is interrupted, these tables have not been written yet - and you are pretty much screwed.

MKV, on the other hand, is based on EBML, a "binary" variant of XML. It means that, in a MKV file, each frame is wrapped in its own separate EBML element. This can be recovered quite easily, even from an "incomplete" file.
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 2nd September 2020 at 01:01.
LoRd_MuldeR is offline   Reply With Quote
Old 2nd September 2020, 01:02   #11  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Quote:
Originally Posted by LoRd_MuldeR View Post
It's because in MP4/MOV all the frames/samples are actually stored in one giant monolithic block, that contains no "packet headers", "sync bytes" or whatsoever to indicate the frame boundaries

The really only way (except maybe by using some heuristics) to locate individual frames within a MP4/MOV file is by using the "sample size", "sample to chunk" and "chunk offset" tables.

Unfortunately, these tables usually will be written at the very end of the file, after all the frames have been written. So, if the encoding process is interrupted, these tables have not been written yet - and you are pretty much screwed.

MKV, on the other hand, is based on EBML, a "binary" variant of XML. It means that, in a MKV file, each frame is wrapped in its own separate EBML element. This can be recovered quite easily, even from an "incomplete" file.
Thanks for that info.

From now on it is going to be mkv all the way!
Cary Knoop is offline   Reply With Quote
Old 2nd September 2020, 02:54   #12  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,893
Quote:
Originally Posted by LoRd_MuldeR View Post
It's because in MP4/MOV all the frames/samples are actually stored in one giant monolithic block, that contains no "packet headers", "sync bytes" or whatsoever to indicate the frame boundaries

The really only way (except maybe by using some heuristics) to locate individual frames within a MP4/MOV file is by using the "sample size", "sample to chunk" and "chunk offset" tables.

Unfortunately, these tables usually will be written at the very end of the file, after all the frames have been written. So, if the encoding process is interrupted, these tables have not been written yet - and you are pretty much screwed.

MKV, on the other hand, is based on EBML, a "binary" variant of XML. It means that, in a MKV file, each frame is wrapped in its own separate EBML element. This can be recovered quite easily, even from an "incomplete" file.
That's actually fantastic information! Thanks for sharing that.
Blue_MiSfit is offline   Reply With Quote
Old 2nd September 2020, 09:18   #13  |  Link
richardpl
Registered User
 
Join Date: Jan 2012
Posts: 184
My offer still stands to recover complete file, unfortunately I do not work for free.
richardpl is offline   Reply With Quote
Old 2nd September 2020, 15:33   #14  |  Link
Cary Knoop
Cary Knoop
 
Cary Knoop's Avatar
 
Join Date: Feb 2017
Location: Newark CA, USA
Posts: 396
Quote:
Originally Posted by richardpl View Post
My offer still stands to recover complete file, unfortunately I do not work for free.
I have already reprocessed the file.
Cary Knoop is offline   Reply With Quote
Old 2nd September 2020, 15:49   #15  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,536
In some cases, like x264 with mkv output, the mkv file has "no index". That it why seeking can take very long with big files. I noticed it with my 40gb lossless encodes. Remuxing the mkv fixes it.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database || https://github.com/avisynth-repository
ChaosKing is offline   Reply With Quote
Old 2nd September 2020, 17:36   #16  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,031
Quote:
Originally Posted by LoRd_MuldeR View Post
It's because in MP4/MOV all the frames/samples are actually stored in one giant monolithic block, that contains no "packet headers", "sync bytes" or whatsoever to indicate the frame boundaries

The really only way (except maybe by using some heuristics) to locate individual frames within a MP4/MOV file is by using the "sample size", "sample to chunk" and "chunk offset" tables.

Unfortunately, these tables usually will be written at the very end of the file, after all the frames have been written. So, if the encoding process is interrupted, these tables have not been written yet - and you are pretty much screwed.
If fragmented MP4 (fMP4) is used, recovery is much easier since it will have that data. That's how adaptive bitrate streaming works, as each GOP is independently accessible via a byterange request.

That said, a good remuxing tool can build the index post-hoc by analyzing an existing non-fMP4 file.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:25.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.