Does anyone have a Windoze batch "universal" x264 encoder script set?

[FredThompson]

Long story short, I'm developing a Windoze batch routine to take all files in a directory (AVI,VOB,MPG, etc.) and create MP4 where only the video is re-encoded. Subtitles and audio will be untouched. NTSC CC will be converted to subtitle. Default languages (video, audio, subtitle) will be set to English.

avg target video bitrate comes from this formula:

0.000145*width*height*framerate

That "magic number" works pretty well for me.

CCExtractor, MediaInfo and x264 are already in the mix. Looking for a multi-container demux right now.

Will make separate sub-routines for specific containers if needed.

MeGUI, VidCoder, Hybrid, etc. won't handle the bitrate formula, hence, the need for a custom routine.

If anyone has developed something similar to what I describe, please consider sharing the results. I'll share what I build, just looking to avoid duplicating effort and learning about gotchas before they show up.

[turbojet]

I use a batch script often that gets info with ffmpeg -i, parses the info for fps/res/audio codec, demuxes with ffmpeg, creates an avs, feeds it to x264 for crf encode and muxes with mkvmerge.

I've used ccextractor in the script by adding 1 line and modifying the muxer line. Also used mp4box in the same script but never tried figuring out average bitrate as it's not needed in my situation (not burning) and it makes things much more complicated with a potential for quality loss over crf. If you want help on anything else though feel free to ask.

[FredThompson]

Will you share the script?

I'll scan the current design flowchart and add it to this thread today. It's really divided into 2 main sections. The first is analysis, the second is encoding and muxing.

Analysis is where source codec is analyzed. If DivX 6, x264 or similar, encoding is bypassed if the avg bitrate is < 1.5 the magic number. If MPEG2, encoding avg bitrate is the lesser of half the source bitrate or the magic number calculation, subject to a similar 1.5 comparison.

Basic idea is that x264 should yield near-identical visual appearance at 1/2 the bitrate of older codecs. Setting destination avg bitrate as a function of frame size has good potential savings in destination file size.

Don't care to deinterlace, crop, or resize.

Wondering if there's a way to sense visual junk at top of streams extracted from DVR and cover them with black. See attached screenshot. This is not always present. The number of scanlines used varies.

Plan to include ColorMatrix.

Toying with the idea of deblocking if source bitrate is below a threshold. Same with deringing and dehaloing.

Any automatic filtering (other than colorspace adjustments) has potential damage. Some "defects" might be intentional. A video which discusses compression artefacts could be screwed up by filtering.

Keeping source audio saves time and avoids any potential timing and/or encoding problems. If there is a major advancement in audio codecs, can always redo the audio portions if that is desired. Cost of storage will probably make that irrelevant to me.

All this assumes the source has plenty of variation in the video streams. Wouldn't work well for screen recordings of software instruction or other low-motion source.

VHS capture and known all-telecine material will have some other processing. Automatic detelecine is easy enough (thanks, Neuron2) but auto VHS is quite another.

Toying with the idea of an AutoIT GUI to set parameters in a text file which is then read by the scripts to start the queue. This might make it more useful for other people. It's the last step, though.

[smok3]

something from 2009, probably not exactly what you want;
http://paste.debian.net/63983/
(I had this linked to some total commander buttons via another loopy.bat with various avs templates and such)

[FredThompson]

Some helpful stuff in there. Thanks.

[turbojet]

My script is way off from what you are doing and wouldn't help much due to lack of comments but let's go step-by-step. For example:


:: Analysis
%~dp0ffmpeg -i %1 >"%~dpn1.txt" 2>>&1 ::send analysis of input file to text
for /f "tokens=4 delims= " %%a in ('find /I "Audio:" "%~dpn1.txt"') do set ac=%%a ::Find audio codec and set it to %ac% variable
for /f "tokens=1 delims=." %%a in ("%~n1") do set fn=%%a ::Setup up custom preset triggered by filename
DEL "%~dpn1.txt"

:: Demux audio
::if not exist "%~dpn1.%ac%" "%~dp0ffmpeg" -i %1 -acodec copy "%~dpn1.%ac%" :: this might work, untested
if %ac%==ac3 if not exist "%~dpn1.ac3" "%~dp0ffmpeg" -i %1 -acodec copy -f ac3 "%~dpn1.ac3"
if %ac%==aac if not exist "%~dpn1.aac" "%~dp0ffmpeg" -i %1 -acodec copy "%~dpn1.aac"
if %ac%==mp3 if not exist "%~dpn1.mp3" "%~dp0ffmpeg" -i %1 -acodec copy "%~dpn1.mp3"

:: Make avisynth if filtering
del "%~dpn1.avs"
if "%fn%"=="ivtc" echo dss2(%1)>>"%~dpn1.avs" & echo tfm().tdecimate()>>"%~dpn1.avs" ::if filename starts with 'ivtc.' it adds tivtc
if "%fn%"==echo dss2(%1)>>"%~dpn1.avs" & "deint" echo tdeint()>>"%~dpn1.avs" ::if filename starts with 'deint.' it adds tdeint
goto :AVS

:: Send directly to x264 if not filtering
if not exist "%~dpn1_VID.264" start "" /b /low /wait "%~dp0x264_64.exe" --output "%~dpn1_VID.264" %1 --crf 18
goto :MUX

:: Encode with avs2yuv->64 bit x264 at low priority
:AVS
"%~dp0avsInfo.exe" "%~dpn1.avs" >>"%~dpn1.txt"
FOR /F "tokens=10 delims= " %%a IN ('find /I "Length:" "%~dpn1.txt"') DO SET FRAMES=%%a
DEL "%~dpn1.txt"
if not exist "%~dpn1_VID.264" start "" /b /low /wait "%~dp0avs2yuv_x86.exe" "%~dpn1.avs" -o - | start "" /b /low /wait "%~dp0x264_64.exe" --output "%~dpn1_VID.264" - --demuxer y4m --frames %FRAMES% --crf 18

:: Mux with mp4box
:MUX
::if not exist "%~dpn1.mp4" "%~dp0mp4box\mp4box" -add "%~dpn1_VID.264" -add "%~dpn1.%ac%" "%~dpn1.mp4" :: this might work, untested
if %ac%==ac3 if not exist "%~dpn1.mp4" "%~dp0mp4box\mp4box" -add "%~dpn1_VID.264" -add "%~dpn1.ac3" "%~dpn1.mp4"
if %ac%==aac if not exist "%~dpn1.mp4" "%~dp0mp4box\mp4box" -add "%~dpn1_VID.264" -add "%~dpn1.aac" "%~dpn1.mp4"
if %ac%==mp3 if not exist "%~dpn1.mp4" "%~dp0mp4box\mp4box" -add "%~dpn1_VID.264" -add "%~dpn1.mp3" "%~dpn1.mp4"



I didn't include a bitrate variable, deblocking or bpp calculation. Are you burning to a DVD or something that restrains the size of the encode? Otherwise I'd and many others would strongly suggest using crf, it's a much better way of getting that you want then going by bitrate or bpp.

Autocrop or Robocrop (http://forum.doom9.org/showthread.php?t=168053) might work well for removing the junk lines from the top. Autocrop hasn't been updated in years but Robocrop dev is active.

[FredThompson]

Plan is to keep the files as files on drives on a media server. Been through CDRs, DVDRs, and BluRay.

Hadn't thought about crf. I played with various methods of constant quality over the years and was always frustrated with how fuzzy, low-quality (such as aged VHS) sources would yield huge files. I used to encode NTSC DV to 422 MPEG2 with TmpgEnc at a cbr of 8000 which worked great. Did a few rough test with clean sources and came up with 1500 bitrate x264 as sufficient for D1 NTSC and figure x264 should match source MPEG2 at half the rate so...that's where the ideas came from. I've been using full first pass, not fast first pass. Will investigate crf.

I know the flow chart isn't up yet. I was researching various ways to make a GUI and to add some logic which might be difficult with batch language. Found QB64 today and it seem useful. I also have linux, android and ios devices so this could be a nice investment of time for other personal projects.

[FredThompson]

OK, looked at CRF. The idea has merit. I seem to recall something similar with one of the popular MPEG2 encoders a few years ago. Maybe that was CCE or was CCE the one where people would run multiple passes and there was debate about how many to run.

I really like the 2-pass concept so there's advance knowledge of which parts of the stream are the most and least complex. Something more like a fast first pass and a CRF second pass would appeal to me. Wonder if x264 has motion analysis for the 2-pass avg bitrate methods or if it's just looking at cumulative deltas between frames.

The encoding portion really doesn't seem that difficult to script. The separation/analysis of source files and support for different aspects such as CC, subtitles, default languages, those seem to be the real "work" of this design. I also noticed that VOBs that MeGui and Hybrid will not process are compressed by VidCoder. MeGui's one-click file checker doesn't seem to ignore non-video files. At least, it doesn't when it shows a file count and asks if it should proceed. I really like MeGui and would use it if there was a way for its AviSynth script to set the encoder properties and if the VOB reading was more robust. Sometimes it fails just after the audio analysis pass, sometimes the second video pass runs beyond 100% completeness. I've also seen everything marked as completed except for the cleanup but there's no muxed result file.

Going to look at Robocrop now to see if it would be helpful on those DVR captures with the extra data in the overscan. Those are from DirecTV. Not sure what's in there, just that it can be really distracting during playback.

OK, Robocrop might work for this type of junk removal. The top few lines of a DirecTV SD feed sometimes have what looks almost like morse code of white on black. It's not part of the source signal, it's extra data. The data varies by frame so it "dances" around as the video is played. Maybe Robocrop could help find this. The goal is to overlay any dancing lines with black. I suppose an assumption of 480x480 MPEG2 with an added verification that the lines tagged by a modified Robocrop are have no chroma and only black and white would be sufficient. Source material that is B&W video would have lots of gray in those lines. Good link, thanks. I'll table this while the overall structure is being designed and come back to it later.

[turbojet]

Pulling CC with ccextractor and muxing them is simple but if you are dealing with many different subtitle formats it could be complicated.

[FredThompson]

The more I think about it, the more it makes sense to build a pre-processor which builds AVS and CCExtract(ed) SRT files. It might even be possible to include something like RoboCrop along with border padding to properly center source and remove the visual junk at the top of some DirecTV captures. (Must pay attention to alignment so frame order isn't screwed up.) NTSC DV needs field order change and 4:2:2 to be as lossless as possible. Not sure what to do with ut screen and analog captures, 4:4:4 might be an option if really needed or the source is important.

The key would be to build custom job queues for an existing GUI and let it do the "hard" parts of the demux, encode, mux, cleanup, etc. tasks.

Am testing GUIs now. MeGUI and Hybrid fail at various points with some of the VOB sources, their indexing seem to be less robust than what is needed.

VidCoder (Handbrake) work with the audio and video but insufficient subtitle support.

Testing TX264 now. It claims to handle subtitles and I really like the "add folder" and "add folder tree" queue options. Don't see how to set a preferred default language (to include assigning a language if none exists) for either audio or subtitles nor how to import a job queue.

Looked at mini264 a while ago and will test it.

[FredThompson]

tx264 doesn't give a proper output. Seems to get through the encoding and fails at the mux point.

Handbrake and VidCoder don't directly support AviSynth scripts and a frame server for batch operations? Ugh. Looking into it, though, to leverage Handbrake's CLI interface. Handbrake does have a pretty nice batch pre-scan which I'm testing.

[FredThompson]

Getting closer.

Found Avisynth Virtual File System http://forum.doom9.org/showthread.php?t=133313

Now it looks like building a job queue for Hybrid is the way to go. It uses mplayer to read and hasn't choked on any of the valid files which choke other GUIs.

Also did some tests and think the threshold for re-encoding should be a 35% reduction in video space.

FLAC (Yes, I have a 720p x264 mkv with a FLAC track) and MP2 will be re-encoded. MP2 will go to AC3 with the same sample rate and resolution. FLAC, well, I've only found one such example and it's from a 1960s Italian movie so AC3 as well, but probably 224/48k.

Plan for VSRip and CCExtractor for VOB sources. SRT, SSA and ASS to be kept if they exist in the stream.

Default language will be English if unspecified or English is present but there are other audio tracks. This could get a little tricky for multi-tracked source files. Maybe they should be reported for additional manual tweaking.

Work continues.

[turbojet]

I thought you were looking for a script, what role does a gui play?

Or have you decided to just go with a gui?

[FredThompson]

For a while, it seemed building a queue to be run through Hybrid would speed up development by leveraging what's already been done.

Found out last night Hybrid isn't properly handling some of the MP2 audio so it's back to Handbrake CLI. Design on that and replace CLI calls as parts of the script are developed. That way, encoding can start sooner and refinements can still be made.

I guess some of my posts were more "stream of thought" than I'd realized. Sorry about that.

[FredThompson]

Alright, GUIs are out. Too limiting for what needs to be done.

Am working on some things. Will post them when I get a rough version going.

[FredThompson]

OK, here are some changes:

This is for general natural video, not for high-motion, simple animation or simple screengrabs. Most of my source are live action with relatively low motion.

Keep all audio, video, subtitles, chapters (if they exist)

Add NTSC CC if it exists (convert to srt with CCExtractor)

If no exiting chapters, add new chapters:
if source length <35 minutes, every 3 minutes
otherwise, every 5 minutes

Audio WILL be re-encoded to AC3 if it is one of these formats:

MP2 -> ac3 with same sample and bitrates
FLAC,PCM -> ac3 192 48000 (Would prefer an analysis pass to set "best fit" bitrate, 192 is fine for most "talking" source but too low for complex source)

Video will be re-encoded to x264 4.1, fast, film with AviSynth for deblocking with these conditions:

MPEG2 colormatrix (Not sure if MPEG2 from SD or HD DVRs need this - assume SD DVD source for both PAL and NTSC need this - need to see if there is a way to auto-detect)

video bitrate is lower of these conditions and must be 0.75% or less than source bitrate:

defaultvideobitrate = int((width*height*framerate*magicnumber)/10)*10

case{

If XviD or DivX video, int((sourcebitrate*magicnumberXviD)/10)*10
magicnumberXviD default = 0.7

if DIV6, int((sourcebitrate*magicnumberDivX6)/10)*10
magicnumberDivX6 default = 0.75

if AVC/h.264, defaultvideobitrate

otherwise, int((sourcebitrate*magicnumbermpeg2)/10)*10
magicnumbermpeg2 default= 0.5

}

Default language settings are kept as specified in the source. Too often I see default languages which do not have a specified language. Can't assume anything as a default other than matching the source. We speak English, Spanish and Italian. If I'm encoding Dad Boot, for example, I'd want the default languages to be English. How do I ensure that unless there is a guarantee that all source languages are properly labelled? Can't be done. Better to keep the source settings and manually change defaults as desired.

[FredThompson]

Here's another example of the overscan data which is sometimes in DirecTV SD broadcasts:

http://forum.doom9.org/showthread.php?p=1654968#post1654968

Imagine how visually annoying this dancing "bar code" is. Human vision is drawn to motion.

Still working on a way to detect and overwrite these lines in an AviSynth script. Not so easy. Might be easier to have a prescan which reports files for which the top few lines are only grayscale but there is color in the rest of the frame. Better to have a list of files to manually check than to have a false positive.

[FredThompson]

Asked RoboCrop author for help: http://forum.doom9.org/showthread.php?p=1654968#post1654968
Discussions with Selur about Hybrid and its job queue structure.
Methinks the least work/quickest completion route is now a directory parsing routine which creates a job queue for Hybrid. This was not the original intent of a batch file but it's a quicker route to the desired goal than recreating most of the process.

It became apparent that the only additions needed to Hybrid were:

determine video bitrate based on source file (magicnumber result vs. existing rate vs. codec-specific weight, lesser of the two is selected)
Auto-determination of ColorMatrix use based on source stream
RoboCrop to get rid of unnecessary letterbox data
determine audio passthrough/encode based on source streams - mp2, flac, pcm get re-encoded

assumption is all source is NOT lower than mpeg2/mp2/lossless quality. I'm sure there are lesser data compression files in my collection. Goal is to get the bulk of this "smart" re-encoder working then identify the outlier formats and add them to the process.

The way I envision this, anyone could take the resultant processor and add their own changes for default values and AviSynth calls. My preference is no deinterlace or resizing, keep all source audio/subtitles. Tried to determine a way to automatically set default languages. That's not easy. Probably better to have a separate routine that does this to the mp4/mkv files afterwards (i.e., if more than one stream, set default to user-chosen language if it exists.)

Waitaminute...maybe that is possible. Hmmm... Can't assign a specific language designation to a default stream which does not have a language in ALL cases but might work in some cases. For example, if languages are und, Spa, Fr and the program is from an English source, assume und is really English and set the flags appropriately. Outliers such as the movie Das Boot, would not be run through the same process.

[Selur]

Auto-determination of ColorMatrix use based on source stream
probably doesn't help you, but Hybrid can already do that, if Avisynth is used. (isn't supported otherwise since, last time I checked mencoder didn't have a colormatrix support; ffmpeg does)

[turbojet]

Are you planning on gaining quality by encoding a lower bpp source to a higher bpp x264? Unless it requires something like cropping, derainbowing, deinterlace or ivtc it's not going to be better because of the higher bpp.

Speaking of colormatrix detection would it be possible in avisynth to measure source colors, convert to 601 and 709 and measure the converted source colors to detect what the original source color matrix is? Or does a convert to always change colors?

[FredThompson]

probably doesn't help you, but Hybrid can already do that, if Avisynth is used. (isn't supported otherwise since, last time I checked mencoder didn't have a colormatrix support; ffmpeg does)I wasn't aware Hybrid autodetected. OK, that's great. I assumed the frame size options for Hybrid's ColorMatrix use were the only determinant. ColorMatrix can be safely enabled for every source and Hybrid applies the conversion when needed? Great!

[FredThompson]

The goal is a general use re-encoder for a bunch of files to x264 if there is at least a 25% savings in video bitrate.

As much as possible, all constant values will be open for user configuration.

All audio tracks will be kept EXCEPT MP2 and lossless formats will be re-encoded.

All subtitles will be kept. NTSC CC titles will be generated and added. I don't have any PAL CC source so I can't personally add that capability.

The quickest path to this goal appears (to me) to be a standalone process which creates a job queue for Hybrid encoder GUI.

--

Once the basic flow is working, functions will be expanded.

The first expansion will probably be RoboCrop AviSynth package.

Scanning and testing lossless source audio to determine optimal bitrate would probably come after that.

Maybe a GUI to easily modify constants would be useful.

I never deinterlace but other people might want this so it will be supported.

IVTC is interesting but it must be robust because this is a set-it-and-forget-it type of process. A few years ago there were lots of discussions here about mixed-format sources such as some of the Star Trek TV episodes. People other than me have far more knowledge about this process.

I'm not sure how this will play out for lossless screen captures. 4:2:2 would give more accuracy with a potential problem at playback.

Similarly, NTSC DV is 4:1:1 so 4:2:2 seems to be the best x264 format to preserve quality.

Should screencasts and NTSC DV yield BOTH 4:2:2 and 4:2:0?

Are the problems with PAL NTSC and red fixed? I don't know. I have a few PAL DV files but haven't done anything with them in years.

There will be plenty of room for tweaking and expanding source file scope after the first revision is stable.

These statements assume I'm putting everything together. Other people will have their own priorities and maybe someone else will help with this project.

I hope to do all of this through batch files, MediaInfo and FFDiag. This keeps the process as open to modification as possible. Kernalization into individual batch files helps during initial design with a hit in speed. Once things look pretty tight, the batch files will be rolled into a single file. I hope this can be done with regular batch files and possibly some helper files. If not, PowerShell might be needed. Haven't really thought that through completely. Hope to keep the process as standalone as possible.

--

Are you planning on gaining quality by encoding a lower bpp source to a higher bpp x264? Unless it requires something like cropping, derainbowing, deinterlace or ivtc it's not going to be better because of the higher bpp.
No. Re-encoding will happen if the target video bitrate is significantly smaller than the source video bitrate. Two numbers are calculated:

MaximunmTargetVideoBitrate := SourceVideoBitrate * CodecSpecificRatio

My choice for MPEG2 is 0.5 so an advanced x264 bitrate would be 0.5 of an MPEG2 source.

I haven't fully compared Xvid, DivX and other simple mpeg4 sources. I have settled on a value of 0.75. More information comparing simple and advanced mpeg4 might give a more useful value. That's not as important to me as getting the workflow...working.

CalculatedTargetVideoBitrate := FrameWidth * FrameHeight * FrameRate * MagicNumber

My MagicNumber is 0.00015

Re-encoding of video happens ONLY IF CalculatedTargetVideoBitrate < (0.75 * SourceVideoBitrate )

ReEncodingTargetVideoBitrate := the lesser of MaximumTargetVideoBitrate and CalculatedTargetVideoBitrate

Speaking of colormatrix detection would it be possible in avisynth to measure source colors, convert to 601 and 709 and measure the converted source colors to detect what the original source color matrix is? Or does a convert to always change colors?I understand Selur's comments to be that Hybrid does auto-detection.

[Selur]

ColorMatrix can be safely enabled for every source and Hybrid applies the conversion when needed? Great
The "From" field will be set depending on the detected ColorMatrix (if the source has no color matrix infos, it will be guessed based on the resolution)
The "To" field will be adjusted by the output resolution, assuming the 'Adjust by resolution' box is checked.
-> no clue, if that method is 'save', but that is how Hybrid does it. :)

[FredThompson]

Agreed. We can assume SD and lower MPEG2 is always Rec.709 but we can't be certain. One of the DVRs here is HD MPEG2. I wonder if somebody has a list of known defaults. VideoHelp has a great table of formats and resolutions but I don't think it has this detail.

All we can really do is make assumptions. We can't automagically detect everything. This is the same dilemna with audio and subtitle tracks that don't have a specified language. We could use an assumption and the user takes the responsibility. This is why the example of the movie, "Das Boot" is a good example. My household speaks English, Spanish and Italian. We don't speak German. Suppose the default audio track for "Das Boot" doesn't explicitly state it is German. I'd still want that track but it would be a mistake to give it a default setting of English because there would already be an English track.