Distributed Encoding: Joining mpegs without A/V glitching?

[eslave]

I'm trying to use mencoder (http://oss.netfarm.it/mplayer-win32.php) to re-assemble several small mpegs into one large mpeg (for streaming to my DVR). My problem is that every joint exhibits a small (but noticeable) audio glitch (squeak/warble).

mencoder -noskip -mc 0 -oac copy -ovc copy -of mpeg -mpegopts format=mpeg2:tsaf:muxrate=3000 out\Job1.mpeg out\Job2.mpeg out\Job3.mpeg out\Job4.mpeg -vf harddup -o testout.mpg


All mpegs are exactly the same frame length (encoded by ffmpeg SVN-r18639 (http://ffmpeg.arrozcru.org/builds/) from an AVS script). I'm trying to make them exactly k*GOPsize frames to prevent breaking the GOPs.

FFMpeg command:
ffmpeg.exe -y -i "%~dpn1_%COMPUTERNAME%.avs" -target ntsc-svcd -acodec mp2 -ac 2 -ar 48000 -ab 192k -s 352x480 -r 29.970 -sc_threshold 1000000000 -flags +cgop+ilme+ildct+alt -top 1 -bf 2 -g 15 -flags2 +sgop+fast -aspect 1.3333 -b 2500000 -maxrate 2500000 -threads 1 "%JOBOUT%\%~n1.mpeg"
(MediaInfo reports my original DVR recordings to be 29.970fps. Although I haven't yet tried the DVR will probably accept 30000/1001 fps video. I'm using low-res encoding purely for speed while testing.)

AVS file: (JobNum starts at 1)
DirectShowSource("SomeVideo.AVI", fps=29.970, convertfps=true)
<resize, auto-letterbox, and make it interlaced>
Trim((JobNum-1)*1500, (JobNum*1500)-1 )

I believe AVIsynth (http://sourceforge.net/projects/avisynth2/) uses a 0-based frame count, seemingly confirmed by ffmpeg's on-screen messages, but MediaInfo always reports 1 frame less than desired. If I change the AVS Trim() line (to add one additional frame):
Trim((JobNum-1)*1500, (JobNum*1500) )
then MediaInfo reports 1 frame more than desired. I've tried other JobSizes (1005 frames, or any other multiple of 15), but the results are the same.

The other curiosity I've encountered is the audio-interleave produced by ffmpeg is 500ms (reported as Delay by MediaInfo), but mencoder always limits the preload to 0.4s. MediaInfo also claims all the small mpegs have audiolength shorter than the video (same for all, but ranging from 0.02s to 0.08s in various tests). No matter what I try to do the framecount of the merged mpeg (from 4 parts) is off by 5-12 frames (1 to 3 frames per part), and the audiolength is shorter than the video (by as much as 0.874s as reported by MediaInfo), although the audio sync appears to be just fine. Just passing a single part through mencoder alters the framecount and audiolength. I've tried mencoder's -audio-delay and -audio-preload options but noticed no difference in the final output. I've tried several different versions of mencoder (Athlon, RTM, RC1, preRC1) with practically the same results. I've tried ffmpeg's -itsoffset and -map options (http://howto-pages.org/ffmpeg/#delay)without improvement.

I've also tried mpgtx -j (http://mpgtx.sourceforge.net/), but this produces video glitching as well. I tried the copy /b trick (equivalent to unix cat (http://ubuntuforums.org/archive/index.php/t-85718.html) command) to no avail. Even assembling the parts in Womble produces the same audio glitching as joining with mencoder. The individual parts all seem to play fine in every player I've tried (MPC2, VLC, Womble, VDubMod 1.5.10.2/2542), but I can't be entirely certain that I'm really hearing the first few ms of audio.

I'm disinclined to demux and assemble the A/V separately before remuxing. Having two 'copies' of the finished video on my HDD is enough, I don't care to have 3 or 4 'copies' laying around during reassembly.

If I could find a solution to the audio glitching then distributed encoding would be worthwhile (which was my original intent), but hearing a squeak every 33s to 50s throughout the show isn't really acceptable. I would like to stick to my CLI/batch process for simplicity (because I already know it works for 1 computer), but I did try Media Encoding Cluster (http://codergrid.de/) only to discover it doesn't support fractional fps, dvd::rip (http://www.exit1.org/dvdrip/doc/cluster.cipp) seems to go the wrong way (from MPEG2), RBfarm looks a little dated and rather complex, and I just noticed AVStoDVD (http://forum.doom9.org/showthread.php?t=126647&highlight=distributed+mpeg).

Any ideas appreciated.

-eslave

[setarip_old]

Hi!I'm trying to use mencoder to re-assemble several small mpegs into one large mpegWhy don't you just rely on the original undivided MPEG that you created?

[eslave]

What undivided MPEG? I'm converting from <anything> TO mpeg, and I'm trying to do it piece-wise so other computers on my network can participate (this part already works). There is no undivided MPEG source at the beginning of this process.

The problem is putting the pieces back together.

-eslave

[netmask]

You can use ProjectX to join up as many mpeg files as you like. Instead of the default Demux select as an output M2P

[eslave]

Thanks!

ProjectX (http://www.videohelp.com/tools/ProjectX) assembled them without glitching, but it did jump-cut both the video and audio. The timestamps would still need fixing as well (minor problem).

At least I can listen to it now, but I still wouldn't want to watch it.

I'm thinking that I still need to fix something with ffmpeg or the AVS file to get the exact framecounts desired, but mencoder's audio behavior eludes me.

-eslave

[netmask]

maybe you do need to demux the files - ProjectX will then fix any errors.

[Ghitulescu]

The reason is usually the audio, probably is 1 audio frame less than needed.
If you demux the things into ESs, you'll see that the lengths are probably not equal.
The only solution I know to work is to cut 1 video frame from the MPEG (the last one) with a "smart" cutter.

[eslave]

Yes, I too am beginning to think there is a fundamental issue with MPEG packetization/clocking which prevents this from working as expected.

I messed around with ffmpeg's -intra, -vframes and -aframes options without success. Although when encoding 1 extra GOP per small mpeg ffmpeg's -intra option helped mencoder's -frames option cut them to the exact framelength desired (which I was unable to do before). Each small mpeg still reported audiolengths different than the videolength (before and after cutting), and the assembled mpeg still had audio problems.

I even re-wrote my encoding line to let mencoder do the heavy lifting (instead of ffmpeg):
mencoder.exe -noskip -mc 0 -noautoexpand -audio-delay 0.0 -audio-preload 0.0 -audio-density 2 -fps 30000/1001 "%~dpn1_%COMPUTERNAME%.avs" -oac lavc -af lavcresample=48000:channels=2 -ovc lavc -lavcopts threads=1:acodec=mp2:abitrate=224:vcodec=mpeg2video:ildct:ilme:alt:top=1:sc_threshold=1000000000:cgop:keyint=15:vbitrate=2500:aspect=4/3 -vf harddup,dsize=4/3 -of mpeg -mpegopts format=mpeg2:tsaf:muxrate=2724:vaspect=4/3:vframerate=30000/1001 -ofps 30000/1001 -o "%JOBOUT%\%~n1.mpeg"
Although mencoder actually encoded the exact number of frames desired the resulting small mpegs exhibited the same audio issues (too long or too short by a few ms).

So the root problem appears to be some kind of audiolength or muxing issue. The audiolength issues may even be caused in some way by AVIsynth, but I can't think of any test to eliminate this possibility. I'm going to explore encoding the audio separately from the video, and then try to reassemble the whole mess with mencoder. I suspect the total audiolength will be several ms shorter than the videolength, but at the end of the show I won't care as long as its in sync.

Thanks again to all for your input, and keep your fingers crossed! I'd be disappointed to find out that its impossible to distribute this process with so many tools available.

-eslave

[eslave]

Well, I finally experienced some success on Sunday with distributed encoding. I think I'm on the right track, and the audio sync is perfect.

The general workflow is:

Determine the video's final framecount at the desired target framerate (with MediaInfo).
For the video Jobs create (framecount/JobSize)+1 AVS files (integer math). A special Job0 is only the audio with a 4x4 BlankClip(). The remaining video Jobs are (k*GOPsize)+1 frames using:
Trim((JobNum-1)*JobSize, JobNum*JobSize)
with the exception of the last Job's trim being:
Trim((JobNum-1)*JobSize, 0).
Encode all the AVS files to mpeg using mencoder.
Join all the mpegs together with the audio:

a) Demux the video mpegs into elementary video streams with BBDMUX (http://members.cox.net/beyeler/bbmpeg.html).
b) Join all video mpegs with mencoder (-nosound -ovc copy)
c) Overlay the audio onto the merged mpeg with mencoder (-audiofile audio.mpg -oac copy)



Steps 1, 2 and 4 are done with a 'distribute/merge' batch file on the host which also produces a secondary 'encoding' batch file for step 3. The encoding batch file can be executed by any computer on the network with AVIsynth and FFDshow (same configuration), provided all computers have mapped the netshare (hosting the source video, all the AVS files, the encoding batch, and mencoder) as the same drive id, including the host (I use Z:) because of the filenames referenced within the encoding batch and AVS files. The primary host begins processing step 3 (by running the encoding batch). Other computers can manually join the encoding process by running the encoding batch (presently I do it all from 1 keyboard using TightVNC (http://www.tightvnc.com/download.html) or RDP (http://remotedesktoprdp.com/) and WakeOnLan (http://www.aquilatech.com/Software/wakeonlan)). When the host detects all AVS files are exhausted and all mpeg encodings are complete it will begin step 4.

The 'extra' frame in each Job is a workaround for mencoder's 'drop the last frame' quirk when using -ovc copy, so the final mpeg produced in step 4c is actually 2 frames shorter than it should be (losing 1 frame for each pass through mencoder with -ovc copy). Supposedly this can be overcome with -ovc raw if you declare the format/fourcc, or I can add two extra blank frames at the end of the final AVS. I hope to eliminate the BBDMUX step if I can get mencoder to behave properly when joining the mpegs (it unpredictably drops too many frames, sometime even an entire GOP). I also hope to merge the mpegs and overlay the audio with a single command instead of 2, but mencoder restarts the audio at the beginning of each video being merged when I try to issue such a command. I would also like to make some kind of 'watch' batch for all the nodes so that they will automatically join on their own once the encoding batch becomes available.

In short duration tests so far I've been increasing the JobSize from 150 frames to 1050 frames, and have successfully joined up to 8 video mpegs together (with appropriately truncated audio for testing). I still need to get mencoder to honor my requested 30000/1000 framerate instead of assuming 29.970, which is strange because when I request 29.970 fps it exits with some kind of "illegal fps for mpeg-2" error. So there is a shudder/judder in the video every few hundred frames, but the audio appears to remain in perfect sync (for the total test duration of nearly 5 minutes so far). I may also opt to encode the audio with ffmpeg instead as it allows bitrates other than 224k for mpeg-2 audio.

For some reason the completed mpeg reports a Delay of 233ms for both the video and audio (according to MediaInfo). This is not a problem for any Windows video player I tested, but unfortunately there is a 'Step 5' required before my DVR will accept the file for streaming: sanitizing. After passing my finished video through the sanitizer (without error) the audio leads the video by nearly half a second (I can't tell exactly), and MediaInfo reports some wacky Delays (V:933ms, A:-143ms+V0:790ms). I've tried muxing with several other programs, but so far most cause my DVR's mpeg sanitizer fits, and the problem remains even with mplex (http://download.videohelp.com/liquid217/dvdauthorgui.pl?p=download) and ffmpeg:
ffmpeg -y -i audio.mp2 -i video.m2v -vcodec copy -acodec copy -map 1:0 -map 0:0 output.mpg
(Curiously this results in a blank video for VLC, but works in Womble.)


If I can solve the last sync issue and clean-up my script a little more I may even post it. I find it almost laughable that I was able to write the batch for steps 1-3 in under 30 minutes, only to have step 4 turn into a multi-day dilemma. So much for 'trivial' concerns.

So close...

-eslave

[eslave]

OK, I realized my original source video suffered from some drifting audio sync to begin with, and I also had so many REM'd-out lines in my test batch I didn't see that I was explicitly telling mencoder to skew the audio. The following weekend I dropped the BBDMUX step, switched back to encoding with ffmpeg because it didn't barf all over my fps/bitrate requirements the way mencoder did, and combined the 2-step mencoder merge into one long piped command-line. I then tested the script on a full-length 94 minute show (without commercials) and saw no appreciable A/V sync issues after merging, even though I know some frames were dropped after further scrutiny with MediaInfo.

Happily my DVR's mpeg sanitizing program hasn't had any problems with these encodings, and over the past few weeks I've only had noticeably bad sync in 2 out of over 30 distributed encodes (before sanitizing). One was due to the cluster host failing to completely encode Job47/156 due to excessively heavy HDD usage (easily fixed), and the other was out-of-sync by 14-20 seconds (but only missing 21 frames overall) and I was unable to find an obvious way to fix it. About 10% of the time ffmpeg still likes to drop one or more of the last frames when encoding without apparent rhyme or reason even though the on-screen messages stubbornly report that the entire Jobsize+1 frames were encoded, but typically no more than 1 frame is dropped so the effect is minimal. For encodes that I'm only going to watch once or twice and then throw-away this solution is perfectly adequate, but probably not if you're doing archival or commercial encoding. Try encoding with mencoder or another command-line encoder if you really need frame-perfect performance.

I'm using 3 single-core machines in my cluster with about 7.7Ghz of raw processing power (3.2G P4 desktop, 1.7G AMD desktop, and a 2.8G P4 laptop). Depending on how much black space (letterboxing) is on the screen my distributed encodes finish in about 30-55% of their normal single-machine encoding times (45-70% faster). The overall speed improvement wasn't quite as dramatic as I would have hoped, but its still much faster than encoding with only my fastest machine. Now while I'm watching one show the next encoding will finish and get added to my DVR's playback guide before I'm even ready, and I can still make use of my DVR's fast-forward, rewind, and other navigation features (unlike connecting the computer's A/V output to my DVR's input).

Under current testing my 100Mbps network averages 12% utilization, with momentary spikes to 86% whenever a node starts a new Job. I could probably add 10 or more comparable network nodes before hitting any serious network I/O bounds. Faster processing nodes will likely use more bandwidth. The encoding script will try to use all cores of a multi-core system with Ffmpeg's -threads %NUMBER_OF_PROCESSORS% option, but AVIsynth will only run on a single core unless you get the multi-threaded version of AVIsynth. Because encoding is far more intensive (CPU bound) than your average frameserving script I think this is an amicable distribution of resources.

All this hinges on every machine running the same (or at least a compatible) version of AVIsynth (2.58 works for me), Ffdshow needs to have the SAME configuration on all machines ("Export all settings" to a .REG file if you need to). The same copy of Ffmpeg will be provided by the cluster host for encoding. All machines (including the cluster host if its encoding too) need to map the host's read-write netshare to the SAME drive letter. This way all file-references (paths and filenames) are the same for all machines. AVIsynth 2.58's DirectshowSource supports simultaneous reads (FILE_SHARE_READ) from multiple applications/clients, but I believe 2.57 does not (I noticed one machine stopped all the other nodes after it opened the video the very first time I tested this).

Attached is a collection of test scripts I've been working with. I know everyone has their favorite scripting language, but I opted for the 'zero-overhead' aspects of batch with command extensions although I did make a concession for the cscript/VBS routine to wait without eating CPU. This isn't intended as a turn-key solution, but you should be able to customize this with a few changes to get what you need (ECHO, PAUSE, and REM are your friends).

DistMerge.CMD is a heavily stripped-down version of my DVR-specific script which does everything from beginning to end, but if it breaks midway then the other scripts may be of use. You can add your own AVS brains to MyScript.avsi and then uncomment or add a few lines in DistMerge.CMD to tailor as needed. As mentioned in the previous post the Audio will be encoded separately as Job0 and then all the Video Jobs will follow. By default DistMerge.CMD will encode on the cluster host as well, and the audio will almost always be encoded by the cluster host. Be certain to double-check the audio codecs and downmix settings on the cluster host before (re-)encoding. If your merge has bad sync check for a significant number of missing frames with Finfo.CMD and re-encode the offending Job(s) by renaming their source AVS files, rename ClusterNodexyz.TMP to CMD and run it, then use Merge.CMD to re-assemble the final video. If all else fails and the sync offset is constant you should be able to fix it with mencoder's audio options. Watcher.CMD is for execution by the client nodes (through the mapped netshare) to monitor the current directory for the encoding batch and then execute it when found. MapDrive.CMD is a sample command-line for mapping netshares. I make no attempt yet to clean-up all the temporary files because I'm still testing and would hate to re-encode everything because of 1 bad Job or some other unexpected batch error. BTW, simultaneous writes to multiple files is a great way to fragment your HDD, so trash all the temporary (small) videos once you've verified the final (merged) output.

I recommend testing first on only 1 computer with the following changes to DistMerge.CMD:
if %MOD10% EQU 0 set MAXJOB=%CURJOB% & echo SHORT-CIRCUIT TEST: Creating Job%CURJOB%.avs... & goto AVSDistDone
rem if %MOD10% EQU 0 echo Creating Job%CURJOB%.avs...
Fix the 4 file/path variables at the top of the script to suit your system's layout and video folder location, then drag-and-drop a video onto DistMerge.CMD and quit Ffmpeg's audio encode (Job0) after 3 or 4 hundred seconds of encoded audio by pressing the "Q" key. For the first single-node test(s) wait for just 2 or 4 video Jobs to complete and then "Q"uit the rest. Inspect the short-duration merged encode for any problems (use Merge.CMD if needed). If everything looks OK then add 1 node and try again, then add 1 or 2 more nodes and try again, then you'll know the drill for adding more nodes. Once you're satisfied that all nodes are participating and performing properly undo the short-circuit changes to DistMerge.CMD and try a full-duration encode of a short sample-clip. If that works then shows of any duration should work. Remember, you can use CTRL-S and CTRL-Q to pause and resume (just don't quit Ffmpeg by mistake). Also, most of these scripts assume they're being executed from the same folder as the source video.

Enjoy.

-eslave