Isochroma
25th October 2006, 02:23
The following is a highly condensed summary of the full tentative specification, whose complete paper with example file is available here: x264 Resume (http://isochroma.com/Testfiles/Misc/x2R/x2R.htm)
The Current Scenario
The x264 encoder is an excellent choice for video compression. Depending on the parameters supplied via commandline or GUI, the encoder can compress video at a wide range of speeds. The best-quality configuration is capable of using large quantities of CPU cycles per frame.
On even the fastest machines, video sequences can take long periods of time to compress to completion using the highest quality settings (exhaustive motion search, RDO, etc.) A recent 256,576-frame video compression run using best settings was to complete in just over seven days. At about noon on the third day, the encoding machine (an Athlon XP 2500+) suffered a write-induced crash (hard drive corruption) from a second unrelated process performing routine network transfer.
After such an interruption, the output file was in an unusable state; the user is forced to delete the file and restart encoding from the first frame.
x264 encoding can be interrupted by at least three classes of events:
1. Unplanned system shutdown/restart: crashes, hard locks, etc. caused by hardware or software
2. Third party (including OS) software crashes that affect x264 or cmd.exe, or require manual reboot to resume other functions.
3. Unexpected accidental or volitionary termination of the x264 process by:
3.1. Window Close
3.2. CTRL-C/CTRL-Break
3.3. Task Manager's End Process/End Process Tree
3.4. Running Software: other processes request termination using API calls or the TaskKill executeable.
Due to write-induced crashing which occurs on average every three days, the machine will likely never complete a compression run of this particular video and is thus unusable for the purpose.
--------
The Proposal
It is proposed that to overcome the difficulties encountered in maintaining a stable environment for the x264 encoder during long encoding runs, a resume feature be implemented.
This resume feature has some desireable characteristics, and the costs are low:
1. Resumption of compression can occur in all cases (1-3) listed above.
2. Encoding normally resumes at the last keyframe successfully written to completion in the output stream.
3. If the recovery file is damaged, encoding resumes at the keyframe before the last keyframe.
4. The recovery file is small (<100k), and need only be updated at each keyframe.
5. The resume feature uses very little CPU, and only periodically at each keyframe.
6. The resume feature's activation is optional and predicated on the appearance of a currently-unused switch in the commandline.
The meat of the idea is a recovery file, written in plaintext and having an optional extension of .x2r. This recovery file contains various things, among them a header with the x264 version used, commandline passed which initiated the reference encoding session, and one or two FRS (Frame Recovery Segments), depending on the stage of encoding.
User X, who wishes to use the resume feature specifies the new -z filename.txt or -z filename.x2r argument anywhere in the commandline:
x264.exe (parameters) -z filename.x2r
Encoding begins, and when the encoder finishes the first keyframe, it creates filename.x2r. This file allows the encoding to resume/recover at a future point in time, when x264 is restarted.
At some point in the encoding process, our happy User X is now sad because his computer crashed / virus wiped it out, or his mom made him shut it down before bed :) Due to premature termination, the x264 process produced an incomplete .264/MKV/MP4 file, which has no header and would likely be difficult to parse, not to mention play.
But this time, luck is on his side! The next morning after brushing his teeth and eating a nourishing breakfast, he sits down at the box and brings up a cmd window. Into it he types this line:
x264.exe -zr filename.x2r
The magical -zr parameter tells x264 to ignore anything else on the line and begin a resume process by first opening filename.x2r. In filename.x2r the process will find first a header (initial scan starts from the beginning) which provides certain information:
---- CUT ----
See full paper (http://isochroma.com/Testfiles/Misc/x2R/x2R.htm) for the remaining juicy details.
The Current Scenario
The x264 encoder is an excellent choice for video compression. Depending on the parameters supplied via commandline or GUI, the encoder can compress video at a wide range of speeds. The best-quality configuration is capable of using large quantities of CPU cycles per frame.
On even the fastest machines, video sequences can take long periods of time to compress to completion using the highest quality settings (exhaustive motion search, RDO, etc.) A recent 256,576-frame video compression run using best settings was to complete in just over seven days. At about noon on the third day, the encoding machine (an Athlon XP 2500+) suffered a write-induced crash (hard drive corruption) from a second unrelated process performing routine network transfer.
After such an interruption, the output file was in an unusable state; the user is forced to delete the file and restart encoding from the first frame.
x264 encoding can be interrupted by at least three classes of events:
1. Unplanned system shutdown/restart: crashes, hard locks, etc. caused by hardware or software
2. Third party (including OS) software crashes that affect x264 or cmd.exe, or require manual reboot to resume other functions.
3. Unexpected accidental or volitionary termination of the x264 process by:
3.1. Window Close
3.2. CTRL-C/CTRL-Break
3.3. Task Manager's End Process/End Process Tree
3.4. Running Software: other processes request termination using API calls or the TaskKill executeable.
Due to write-induced crashing which occurs on average every three days, the machine will likely never complete a compression run of this particular video and is thus unusable for the purpose.
--------
The Proposal
It is proposed that to overcome the difficulties encountered in maintaining a stable environment for the x264 encoder during long encoding runs, a resume feature be implemented.
This resume feature has some desireable characteristics, and the costs are low:
1. Resumption of compression can occur in all cases (1-3) listed above.
2. Encoding normally resumes at the last keyframe successfully written to completion in the output stream.
3. If the recovery file is damaged, encoding resumes at the keyframe before the last keyframe.
4. The recovery file is small (<100k), and need only be updated at each keyframe.
5. The resume feature uses very little CPU, and only periodically at each keyframe.
6. The resume feature's activation is optional and predicated on the appearance of a currently-unused switch in the commandline.
The meat of the idea is a recovery file, written in plaintext and having an optional extension of .x2r. This recovery file contains various things, among them a header with the x264 version used, commandline passed which initiated the reference encoding session, and one or two FRS (Frame Recovery Segments), depending on the stage of encoding.
User X, who wishes to use the resume feature specifies the new -z filename.txt or -z filename.x2r argument anywhere in the commandline:
x264.exe (parameters) -z filename.x2r
Encoding begins, and when the encoder finishes the first keyframe, it creates filename.x2r. This file allows the encoding to resume/recover at a future point in time, when x264 is restarted.
At some point in the encoding process, our happy User X is now sad because his computer crashed / virus wiped it out, or his mom made him shut it down before bed :) Due to premature termination, the x264 process produced an incomplete .264/MKV/MP4 file, which has no header and would likely be difficult to parse, not to mention play.
But this time, luck is on his side! The next morning after brushing his teeth and eating a nourishing breakfast, he sits down at the box and brings up a cmd window. Into it he types this line:
x264.exe -zr filename.x2r
The magical -zr parameter tells x264 to ignore anything else on the line and begin a resume process by first opening filename.x2r. In filename.x2r the process will find first a header (initial scan starts from the beginning) which provides certain information:
---- CUT ----
See full paper (http://isochroma.com/Testfiles/Misc/x2R/x2R.htm) for the remaining juicy details.