Batch encoding of DV AVIs

[Pakmenu]

Thanks to Didée for the 2x6Mb samples of Nvidia cuda v. x264.

My hardware is 7+ years old and I do a lot of (re-)encoding and publishing on torrents of movies.

So naturally I'm having to decide where to put my money: CPU, APU or GPU? It does look like Quick Sync is promising, but as it comes with sandy bridge, which is at the moment the best value for money, I'll have 2 in one when buying an i5 2500k.

half the energy consumption and better quick sync sound like nice promises for the Ivy platform, but I'm sure it'll take a year or so before release and prices that are way higher... or is it around the corner and affordable? (=not more then 30% more then what i'd pay for an i5 now) anyone?

Anyways...As I have a stack of PAL DV homevideotapes in a box (which i can't even play back as my DV cam broke!)

AFTER 15 YEARS, I'm still waiting for the one step encoder that can transcode a RAW 14GB interlaced AVI into bob-deinterlaced 50p h264, using a constant quality setting..

It looks like Mediacoder should in principle be able to do it, but have never been able to get the right settings to succesfully double the framerate from 25i to 50p... if anyone has advice, i'll pay a 20 euro bonus (if they have a bankaccount in europe or china as I dislike paypal)

Thanks everyone for contributing, and helping me sort out the advertising from the cream (x264 in this case).
AFAIR (r=researched) about the most effective (speed while retaining quality wise) way to encode to h264 AVC video:

encoding to h264 seems best done by x264 on a x86 CPU.

My version of why this is: because the linear coding of video is not possible to scale to >10 processes without losing video quality. as the video would be divided up smaller regions (let's say 128x72pixel) and the seperate threads can not look at the other regions for similarities. Nevertheless the idea of stanley huang has merit (Note: though I didn't like his move to closed source and paid versions and am sticking with the opensource portable version). The idea: encode different parts, say every 4 minute segment of a movie is coded by a seperate process. This looks like a good solution to me, to break encoding up in to several tasks (provided you either use cq coding and don't care about the final size, or use an average bitrate and not care about the final distribution of bandwith over the whole movie)

not sure anyone else has though of it before but anyway... here's a thought! could x264 have a commandline where it divides itself into n copies and each copy transcodes 1/nth of the movie length? Ofcourse as it stands now it doesn't look like intel is moving to ever more cores... (I was hoping for the 50core general purpose integration of CPU and GPU).

Compared to my 10hour encode times i have now on 1.6Ghz single core, a new sandy bridge would already be a miracle in comparison!

[Dark Shikari]

My version of why this is: because the linear coding of video is not possible to scale to >10 processes without losing video quality. as the video would be divided up smaller regions (let's say 128x72pixel) and the seperate threads can not look at the other regions for similarities. Nevertheless the idea of stanley huang has merit (Note: though I didn't like his move to closed source and paid versions and am sticking with the opensource portable version). The idea: encode different parts, say every 4 minute segment of a movie is coded by a seperate process. This looks like a good solution to me, to break encoding up in to several tasks (provided you either use cq coding and don't care about the final size, or use an average bitrate and not care about the final distribution of bandwith over the whole movie)

This is not suitable to a GPU-like architecture. The problem is not merely "how to utilize thousands of cores", it is "how to use thousands of GPU cores". The latter is a much more difficult problem because GPUs only have thousands of "cores" by using a very restricted definition of what a "core" is.

[Guest]

Originally Posted by Pakmenu

I do a lot of (re-)encoding and publishing on torrents of movies.

Please read and follow our forum rules. You cannot discuss that stuff here.

[nm]

Originally Posted by Pakmenu

Anyways...As I have a stack of PAL DV homevideotapes in a box (which i can't even play back as my DV cam broke!)

AFTER 15 YEARS, I'm still waiting for the one step encoder that can transcode a RAW 14GB interlaced AVI into bob-deinterlaced 50p h264, using a constant quality setting..

Just use QTGMC for bobbing and encode with x264 in CRF mode.

[Pakmenu]

Ok, why i'm asking for a ONE STEP process, is because really I'd like to batch process the uncompressed AVI's to 50fps bob-deinterlaced CRF x264...

Why?
the only way to get the video separated into the various clips, is to do so while transferring from DV tape to uncompressed AVI as after transferring to the PC the time-code is lost, unless again: really complicated steps are taken including making of a special file that keeps track of the timecode and then using software compatible with that, and software compatible with that is not x264 enabled.

Result: every hour of DV tape gets cut into about 50 uncompressed AVI fragments, since i have >50 tapes= over 2500 fragments.

See why I don't look forward to manually using Avisynth and then x264 without a GUI that can batch process it for me?

[nm]

Originally Posted by Pakmenu

Result: every hour of DV tape gets cut into about 50 uncompressed AVI fragments, since i have >50 tapes= over 2500 fragments.

See why I don't look forward to manually using Avisynth and then x264 without a GUI that can batch process it for me?

I'd write a shell script to do such work.

[Blue_MiSfit]

Yep. Even a simple batch file would suffice.

http://commandwindows.com/batchfiles-iterating.htm

Your steps for each iteration would be as follows:

1) Write an AviSynth script that loads the AVI in question, and applies your desired filter chain
2) Call x264, feeding it the name of the AviSynth script as input, with your desired options, and the output file name as something similar to the input

Or you could replace avisynth + x264 with ffmpeg, libav, or something similar - both of which implement libx264 and have their own deinterlacing libraries. Quality is probably not up to par with QTGMC in any case.

[Pakmenu]

Thanks for the pointer Blue_MiSfit !! - I might be able to get this done the way you discribe (though i doubt without any serious setbacks) and i'll try as far as time permits, though as I have a wife asking attention, complaining when i spend hours and hours of time without visible results on the computer.

Avysynth is simple but i often take hours to get the syntax right, and the plugins installed and working right. I just know I'm going to get stuck with all the (4?) dependencies i see that QTGMC relies on! some dll missing somewhere or some syntax errors i can't get my head around.

My mind is not one that can easlitly wrap itself around commandline programming because often a clear instruction or syntax or 'howto' is NOT included with such software, written by geeks who think something like below commandline is *obvious*

Loadplugin("C:\Program Files\ASXGui\tools\dgdecode.dll")
Loadplugin("C:\Program Files\ASXGui\tools\decomb.dll")
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\autocrop.dll")
DirectShowSource("D:testvid.VOB")
AssumeTFF()
Telecide(guide=1)
Decimate(5)
Lanczos4Resize(720,528)
Autocrop(mode=0,threshold=18,samples=45)

These kind of things (loading of plugins, the syntax etc) is NOT obvious for me... i have tried, but I got as far as making AVS files work for me, but not as far as batching them. (I do know basic DOS commands, but how to iterate or make variables, no idea!)

To me such commandline programming can only be understood if i'm basically held by the hand and told what to do..
finally I ask myself why the only program (Avysinth) that CAN do what i want so damned complicated? Why is there no GUI!?

I figured that I'm SURELY not to the only one with interlaced material waiting to be archived.
And so I have waited and waited... and tried program after program for the functionality i'm looking for, which is de-interlace and transcode to 50fps quality based WMV/x264... I tried everything from avidemuc, avysynth, super, Megui, Handbrake, VideoStudio 9,10,12, Canopus Pro, Mainconcept, ASXGui, TMPGEnc, MEncoder, Super, Pinacle Studio & about a dozen more obscure ones like StoikVideoConverter, and my favorite: MediaCoder 6.9-7.3 (newer versions got worse) All of these have cost me thousands of hours (REALLY!) to experiment with, every one of them resulting is dissapointment concerning video editing software in general. Only the fact that most professional encoders refuse to encode to a given resolution and force you to encode to PAL/NTFS even in x264! Pathetic!

Nevertheless with literally hundreds of silly to (so called) proffessional video editing softwares coming out every year, your'd think ONE atleast would have the same functionality as the batch script using Avysinth QTGMC + x264 crf encoding, you propose I write and... which i was trying to avoid...... as i don't know how to write scripts.

I guess it was the same with casette tapes to mp3, I have wondered for years why No-one is coming up with a hardware device where you can enter a casette and it can encode to mp3?
And for that matter, why not a hardware DV tape to HDD x264 transcoder? (I have bought 2 HDD-DVD recorders but mpg2 quality sucks)

I start to think that there's just a conspiracy to prevent people from *updating* their archived material to newer formats. The only exception i have seen is my mother who has paid over 100 euro to 'convert' some of our old super8 filmrolls into DVD, but the service did a lousy job (unsharp & instead of the sound of the movie, it's silent with occasional strangely spooky sounds and unclear voices on the soundtrack)

Anyway sory for kind of 'complaining'. Seriously i have thought if i only knew a programmer who can do what i want, i'd pay him to write a program to do this and do humanity a favour. (countless people don't even know what 'interlaced' is, and just use something like WMV, (which weave-blends interlaced material!) and... get stuck with below par- captures of their home-video.

Cheers Everyone for being on the cutting edge of video encoding quality here!

[blubb444]

I recently started to do something very similar with my DV-AVIs from an old 1999 Panasonic NV-DS12 (I sometimes still use it as a second cam). I'm really a total noob when it comes to scripting/programming/batching stuff but eventually I figured it out how to do everything in a batched way with CLI only!

Just some things to consider:
- The time code is usually NOT lost when capturing to DV-AVI (it might depend on the software though). I'm usually capturing with my NLE (Magix MEP 17), I always get the timecode with it. How can I tell? Well, there's a software called Scenalyzer (newer versions are $$$ but you can get an old but for this purpose still functional free version from here: http://www.chip.de/downloads/Scenaly..._13000791.html ) into which I simply import my unsegmented (up to 12GB) DV-AVI and it can split it into smaller DV-AVIs, with a systematic date/time-based filename! It exactly cuts where there are scene changes on the tape.
- Never resize anamorphic material to square pixels, encode as anamorphic instead, x264 supports it! You'll get better encoding efficiency - quality/(encoding time * bitrate) ratio - with that. Almost all "4:3" DV material is aimed at a 15:11 resolution resulting from the original 720x576 (PAL) or 720x480 (NTSC) frame, the difference to actual 4:3 is apparently an artefact from the analogue days where some pixels from the left and right would be cropped or something (to 704 or 702), resulting in the narrower actual 4:3 frame. But the pixels are there in the digital stream/file so no need to throw them away, I can't understand why people sometimes crop them off. That means that you simply have to tell x264 to use --sar 12:11 for PAL or --sar 10:11 for NTSC. It irks me when people stretch to 768x576 (PAL), giving the encoder more work uselessly, or even worse, downscale to 640x480 (NTSC), discarding valuable video information. I guess it makes sense if you have a 640x480 display, but at least in computer monitors/laptops/even netbooks, this had already been low-res back in '98 (my first, cheap CRT in that year was set to 1280x1024). Again: Resizing to 1:1 pixels may have been necessary back in the xvid in avi days thanks to container and/or stream limitations, but with x264 in mp4/mkv it's useless. Most hardware players seem to support anamorphic too (even my TV).

But back to the topic, here's how I did my whole conversion (all DV-AVIs to be encoded are in one folder, the DV decoder I use is cedocida, Avisynth is multithreaded, very important for my QTGMC setting!):

- You'll want to have in the same folder as the DV-AVIs you're about to process: x264 (I'm using 10-bit depth 64-bit program, but if you want good compatibility with hardware players/share the files etc. use 8-bit depth instead), avs4x264mod, ffmpeg, neroaacenc (assuming the audio shall be encoded to aac) and mkvmerge (assuming we want to make MKVs, you can make MP4s too but IMO mp4box is slow as hell). You don't *NEED* to have them in the same dir but I prefer it because it makes my AVSs and BATs look cleaner.

- First I made a batch file that created one Avisynth script per AVI, like this:

for %%a in ("*.avi") do echo SetMTMode(5,10) > %%~na.avs
for %%a in ("*.avi") do echo AviSource("%%~na.avi",pixel_type="YV12") >> %%~na.avs
for %%a in ("*.avi") do echo SetMTMode(2) >> %%~na.avs
for %%a in ("*.avi") do echo QTGMC(Preset="Placebo", EdiThreads=2) >> %%~na.avs
for %%a in ("*.avi") do echo levels(16, 1.0, 255, 0, 255, coring=false, dither=true) >> %%~na.avs

This results in AVS scripts looking like this:

SetMTMode(5,10)
AviSource("20050822_151017.avi",pixel_type="YV12") 
SetMTMode(2) 
QTGMC(Preset="Placebo", EdiThreads=2) 
levels(16, 1.0, 255, 0, 255, coring=false, dither=true)

You must tweak the parameters in "SetMTMode" and "EdiThreads" yourself to get maximum performance, according to the .html file that is included with QTGMC (open the avs in VirtualDub, run a video analysis pass and look how fast it goes)
I'm using the slowest preset (placebo) because it does some pretty decent denoising too, you can of course also choose a faster preset or even tune single parameters)
I'm doing the levels adjustment because my camera, like many other consumer cameras, has strange luma values from 16 to 255 which is neither limited nor full range so I expand it to full range and write that info to the h264 stream later on. If you don't do that, you'll get either a) clipping of the bright pixels when the decoder is set to use or assumes limited range or b) "blacks" that are actually more dark grey if full range is assumed. See http://forum.doom9.org/showthread.php?t=165003 . For testing purposes use histogram() on a clip rich in contrast to see what levels your camcorder records at. Of course this might just not matter to you and you can leave it as it is.
For the pixel type, choose "YV411" instead of "YV12" if you have NTSC files. But then another problem arises, because x264 doesn't support 4:1:1 from what I've gathered so you have to convert the colourspace by adding ".ConvertToXX()" at the end of the line where it loads your video. Here you have two options (use that for XX): Converting to YV12 (4:2:0) and losing half of the vertical chroma in the process so your effective chroma sampling will be down to 1/8 of the luma. Or you could convert to YUV2 (4:2:2) at the possible expense of hardware compatibility (doesn't matter anyway if you use 10 bit). In the latter case, this has then to be specified in the x264 parameters but I had some problems with avs4x264mod with that (it ignored my --input-csp and --output-csp settings and simply converted to YV12, in that case use the avs script with 32-bit x264 directly). Maybe you can leave out the conversion altogether and configure cedocida for the desired subsampling and specify that as pixel_type but I'm not 100% sure, just try it and use whatever works best.

- In the next step, I made another batch to create yet another batch which will do the video encoding:

for %%a in ("*.avs") do echo avs4x264mod --x264-binary "x264-64-10.exe" --preset slow --crf 18 --tune film 
--keyint 500 --min-keyint 50 --sar 12:11 --input-range PC --range PC -o %%~na.264 %%~na.avs >> x264enc.bat

(inserted a break here to not blow up the width, remove it)


Of course you can tweak the x264 however you like, but keep in mind that a faster preset won't benefit that much with speed because you'll be bottlenecked by QTGMC. For NTSC, use --sar 10:11 instead. I used the keyint for better seeking, you can leave it out if you want. If you didn't do the level adjustment above then leave out the --range and --input-range. The CRF I used led to a bitrate of about 3500 kbit/s on average, of course it depends on the material. Change it if you like. Of course you have to use the correct x264 binary name too.

- The created batch then looks like this:

avs4x264mod --x264-binary "x264-64-10.exe" --preset slow --crf 18 --tune film --keyint 500 --min-keyint 50
 --sar 12:11 --input-range PC --range PC -o 20000716_202220.264 20000716_202220.avs 
avs4x264mod --x264-binary "x264-64-10.exe" --preset slow --crf 18 --tune film --keyint 500 --min-keyint 50
 --sar 12:11 --input-range PC --range PC -o 20000716_204116.264 20000716_204116.avs
...

(here I inserted breaks too, they won't and mustn't be in the .bat-generated .bat)


- Now you can run that batch directly to start video encoding, but I'd like to call it with another batch to keep the CPU priority low so that I can do other stuff on the PC (of course this can be done from cmd, but I'm too lazy to cd myself all the way there every time :

START /LOW x264enc.bat

- Now for the audio processing, I first separate the audio tracks from the AVIs and store them in WAVs with ffmpeg to make them compatible to the Nero AAC encoder with following batch:

for %%a in ("*.avi") do ffmpeg -i "%%a" -vn -acodec copy "%%~na.wav"

- Then they can be encoded:

for %%a in ("*.wav") do neroaacenc.exe -if "%%a" -of "%%~na.aac" -q 0.52

The quality value -q goes from 0 to 1 float and here, as opposed to x264, higher means better quality/higher filesize.

- Finally the raw audio and video tracks can be muxed together:

for %%a in ("*.aac") do mkvmerge -o "%%~na.mkv" "%%~na.aac" "%%~na.264"

Now if everything looks and sounds right you can delete the .264 and .aac files, and if you have no different plans with the DV-AVIs you can delete them as well (but it's always a good idea to keep the tapes of course)
Of course there may be more efficient ways to do all this, like I said I'm relatively new to it all too.

[nm]

Originally Posted by blubb444

Of course there may be more efficient ways to do all this, like I said I'm relatively new to it all too.

Thanks for sharing what you've learned in that thorough post! Many people here use similar workflows (mine is almost identical to yours except for running everything on Linux), but were just too damn lazy to write about it.

[blubb444]

Yeah, I think what this site needs are some more noob-friendly guides/tutorials on such very general and common issues like for example this one (getting your DV captures to a reasonable size and make them look good). Sure there are lots of very special questions which cannot be answered generally but then again I often see returning issues like this one which I think newbies (like myself) can be informed about to get a rough idea to start with.

[Guest]

I've broken these posts out to a new thread as they were off-topic for the original thread.