Star Trek TNG NTSC DVD to x264 + 5.1 AAC MKV (VFR)

[fields_g]

As many others have found over the past years, I recently discovered the "joy" of encoding mixed / Hybrid / VFR (variable frame rate) sources. I'm going to highlight my procedure to get this from a DVD to a MKV container by leaving it hybrid and without conversion to 23.976, 29.97, or 120 fps. Since I'm just starting a large conversion project, I'd like to have comments on the procedure I have devised and help with video encoding settings.

Here's what I'm doing:
1) DVD Decrypter 3.5.4.0
Individually Rip Each episode
Stream process to remove AC3 2.0 right away.
2) DGIndex (1.4.0, 1.4.1rc1 doesn't work with TIVTC 0.9.9.7)
Create AC3 file
Honor pulldown
3) AVISynth 2.55 with TIVTC 0.9.9.7 plugins
Create two AViSynth scripts
First Script

mpeg2source("E:\NEXTGEN\VIDEO_TS1-1\episode1-1.d2v")
tfm(d2v="E:\NEXTGEN\VIDEO_TS1-1\episode1-1.d2v",output="E:\NEXTGEN\VIDEO_TS1-1\ep1-tfm.txt")
tdecimate(mode=4,tcfv1=false,output="E:\NEXTGEN\VIDEO_TS1-1\ep1-tdec.txt")

# this next line assumes the source is dvd 720x480
# and crops it down to 32x32 to save processing time
# since we're only interested in letting tfm/tdecimate
# gather info and not the actual result

crop(344,224,-344,-224)

Second Script

mpeg2source("E:\NEXTGEN\VIDEO_TS1-1\episode1-1.d2v")
tfm(d2v="E:\NEXTGEN\VIDEO_TS1-1\episode1-1.d2v",input="E:\NEXTGEN\VIDEO_TS1-1\ep1-tfm.txt")

# If your source is not anime or cartoon then add
# vfrDec=0 into the line below
tdecimate(mode=5,hybrid=2,tcfv1=false,vfrDec=0,input="E:\NEXTGEN\VIDEO_TS1-1\ep1-tdec.txt",tfmIn="E:\NEXTGEN\VIDEO_TS1-1\ep1-tfm.txt",mkvOut="E:\NEXTGEN\VIDEO_TS1-1\ep1-timecodes.txt")


4) VirturalDubMod 1.5.10.1 (or any other script playing program)
Play First Script (disabling input window speeds process greatly)
Close Program
5) MEGUI
Audio
Encode AC3 to MP4 5.1 HE AAC, Normal, Quality
Video
Input Second Script
Use Bitrate calculator (load created MP4 audio and select 1/2 CD)
Configure video (also verify bitrate setting)

MEGUI x264 config Main page:
Mode: Turbo Automated 2pass
Bitrate 1081
Keyframe Interval 250
# of Ref Frames 8
# of B-Frames 4
B-Frame options Pyramid Adaptive
Deblocking Filter
Alpha Deblocking 0
Beta Deblocking 0
CABAC
Subpixel Refinement 6
Weighted Prediction
Chroma ME
Macroblock Options All
AVC Profiles High

MEGUI x264 config Advanced page:
Min Quant 10
Max Quant 51
Max Quant Delta 1
Credits Quant 40
Factor between I and P 1.4
Factor Between P and B 1.3
Chroma QP offset 0

VBV Buffer Size (undefined)
VBV Max Bitrate (undefined)
VBV Internal buffer 0.9
Bitrate Variance 1.0
Quant Compression 0.6
Temp Blur of est Frame complexity 20
Tem blur of Quant after CC 0.5
SCD Sensitivity 40
B-Fame bias 0
B-frame mode Temporal
ME Algo Multi hex
ME range 16
Nb. Threads 1
Min GOP size 20
SAR (undefined) x (undefined)
fourCC x264

Encode as MP4 file
6) MMG (part of Mkvtoolnix 1.5.5)
Merge MP4 Audio, MP4 Video and timeframe file (under Video Stream)(Subs and Chapters?)

Play and Enjoy

QUESTIONS SECTION:
---------------------
Is there some automated way (without manually separating film and video segments and recombining) to make a hybrid AVC MP4? Would someone be interested in making such a tool? I believe that future hardware will require such a container to play these files and would rather use mp4 instead of repakaging to mp4 later.

Is there a way to have MEGUI to play the first script using the encoding function to eliminate the need to manually running everything thing through VirtualDubMod before MEGUI?

I've been looking for suggestions for MEGUI x264 configs. Could someone make some specific MEGUI suggestions (preferably for a 1050 to 1100 kbit video bitrate (45 minutes with 192 kbit audio on 1/2 CD)). Encoding speed not much of a concern (within reason), but quality and the ability to decode 2500 or greater processor and future mp4 hardware is.

Any other comments would be appreciated!

References:
http://www.avisynth.org/VariableFrameRateVideo#create-vfr-mkv
http://forum.doom9.org/showthread.php?t=98247
TIVTC Documents

[Kostarum Rex Persia]

Everything about your settings looks good,except number of Ref frames(value 8 is too high,I reccomend to use 5) and B-Frames(I don't sure why you put value 4)

I suppose that your goal is to archive a very good quality of output video.Can you post some pictures of input and output file,in BMP or TIF format,of course.

[TheBashar]

tdecimate(mode=5,hybrid=2,tcfv1=false,vfrDec=0,input="E:\NEXTGEN\VIDEO_TS1-1\ep1-tdec.txt",tfmIn="E:\NEXTGEN\VIDEO_TS1-1\ep1-tfm.txt",mkvOut="E:\NEXTGEN\VIDEO_TS1-1\ep1-timecodes.txt")


fields_g,

I too have been enjoying the joys of VFR AVC MKV. One difference between what I have been doing and what you described is that I prefer to use DeDup instead of TDecimate. TDecimate, even in its VFR modes, is still based around the idea of periodic decimation.

Much of the hybrid material I have (TNG, B5, SG1) have sections where the pattern changes or there is no pattern. For instance, there is usually the 3:2 pulldown, then there are some CGI sections with no duplicate frames at all. Then sometimes there are slow motion seuqences of KDDKDDKDDKDD.

I find DeDup to be much more versatile. It has much more sophisticated tuning controls for duplicate detection and its algorithms are not premised on cycle patterns.

I also use TDeInt instead of TFM's internal deinterlacer, but I have much less rationale reasons for chosing one over the other.

Cheers!

[fields_g]

Admittedly, not much time was spent on the AVC config yet. Kostarum, thanks for your recommendations. I feel this is where most of my project tweeking is yet to come. When I complete some encodings, I'd be willing to share some shots in order to demonstate quality.

TheBashar, I also have seen DeDup as a possibility. I went with TIVTC route because of the "out-of-order frames (b-frames)" message on the avisynth page. After getting that package and reading the AVCTcA readme, it looks like mkvmerge (Aug 3, 2005 or later) now supports these natively. I just stuck with that package. It seems like DeDup has some benefits. I'll look at it closer! Thanks!

[tritical]

I too have been enjoying the joys of VFR AVC MKV. One difference between what I have been doing and what you described is that I prefer to use DeDup instead of TDecimate. TDecimate, even in its VFR modes, is still based around the idea of periodic decimation.

Much of the hybrid material I have (TNG, B5, SG1) have sections where the pattern changes or there is no pattern. For instance, there is usually the 3:2 pulldown, then there are some CGI sections with no duplicate frames at all. Then sometimes there are slow motion seuqences of KDDKDDKDDKDD.
I think you are slightly confused as to what exactly Dedup and TDecimate do, because they do not do the same thing.

Dedup takes an input stream, identifies duplicates, and then replaces runs of duplicates with a single frame from that run and uses the timecodes to extend the display time of that one frame to take up the same amount of time as the original string of duplicates. It does not identify 24p vs 30p or adjust the timecodes for that.

TDecimate, in vfr mode, identifies 24p vs 30p sections, decimates the duplicate frame in 24p sections, and then outputs a timecode file so that the 24p and 30p sections play at the correct rates. It does not try to eliminate or replace duplicates that arise for any reason other than field matching of telecined footage.

The difference between those two operations is very important. In fact, you can use TDecimate/Dedup in combination by running dedup after tdecimate and with the timecode file tdecimate outputs as input to dedup via its "timesin" option.

And, indeed, AVCTcA is now unneeded and useless as mkvmerge from recent versions of mkvtoolnix handles the combination of b-frames and external timecode files correctly.

[TheBashar]

I think you are slightly confused as to what exactly Dedup and TDecimate do, because they do not do the same thing.

I don't think I was confused or alternatively I still am. I was voicing a preference for DeDup because I preferred it's ability to handle duplicate detection (and removal) in all cases, not just from 24 vs 30 fps sections. I find this superior for my hybrid material than just 24/30 section handling.

In fact, you can use TDecimate/Dedup in combination by running dedup after tdecimate and with the timecode file tdecimate outputs as input to dedup via its "timesin" option.

That's something I hadn't considered, but why not just let DeDup remove the all the duplicate frames (ie the ones the 1st pass through TDecimate would remove)?

And, indeed, AVCTcA is now unneeded and useless as mkvmerge from recent versions of mkvtoolnix handles the combination of b-frames and external timecode files correctly.

Really? I didn't know that either. I had tested it with 1.5.0 and the problem still existed. I will have to try the new 1.5.5 release. Thanks for the heads-up!

[tritical]

I don't think I was confused or alternatively I still am. I was voicing a preference for DeDup because I preferred it's ability to handle duplicate detection (and removal) in all cases, not just from 24 vs 30 fps sections. I find this superior for my hybrid material than just 24/30 section handling. But that is just it... in terms of playback, dedup does not remove duplicates, it replaces a run of duplicates with a single frame and then adjusts the display duration of that one frame to be the same length as the original run of duplicates. That means if you feed dedup with the undecimated output from tfm() or telecide() the 24fps parts will never be decimated, because instead of removing the duplicate and then adjusting the framerate of the remaining frames (what tdecimate does), it will remove the duplicate and then double the display duration of the frame that there was duplicate off... so the 24fps parts will still be jerky. i.e. it would be just like running telecide() w/o decimate() on a normal movie.

Think about it this way, you would never replace decimate() with dup(), but you would use dup() after decimate()... it is the same relationship here between tdecimate() and dedup().

That's something I hadn't considered, but why not just let DeDup remove the all the duplicate frames (ie the ones the 1st pass through TDecimate would remove)?
Because then, as explained above, the 24fps parts will not be correctly decimated. You will only have done field matching and no decimation.

[TheBashar]

But that is just it... in terms of playback, dedup [...] replaces a run of duplicates with a single frame and then adjusts the display duration of that one frame to be the same length as the original run of duplicates.

I understand. That was my goal.

instead of removing the duplicate and then adjusting the framerate of the remaining frames (what tdecimate does)

Aahhh! Now there is my misunderstanding! I incorrectly thought that TDecimate used M-in-N cycles for duplicate determination and then created an output in the same manner as DeDup. I did not realize that it adjusted the remaining M frames timestamps to smooth over the whole N frame time!

Now I get it. Now your suggestion about following TDecimate with DeDup makes perfect sense!

Thank you for taking the time to get that through my thick skull!

[fields_g]

tritical, thanks for the explanations. You mentioned the option of running dedup after tdecimate. Would this catch many frames on this type of source material? Is it worth the extra processing? Would it disturb future AVC hardware decoding?

[Teegedeck]

From my bit of experience dedup doesn't save a lot of bitrate on natural video. Not if you don't set thresholds so low it gets choppy. And I guess we consider non-choppy playback of hybrid material the biggest advantage of vfr in matroska, right?

[bond]

Is there some automated way (without manually separating film and video segments and recombining) to make a hybrid AVC MP4? Would someone be interested in making such a tool? its pretty easy to create a vfr .mp4 file with xvid (via n-vop dropping), but for x264 there is no method existing till afaik

and yes, any .mp4 player handles vfr streams in .mp4 unless its borked

[tritical]

@fields_g
I would agree with Teegedeck, it probably wouldn't help much on natural video. Even on animation I tend to not use dup()/dedup() simply because I prefer the look w/o it.


its pretty easy to create a vfr .mp4 file with xvid (via n-vop dropping), but for x264 there is no method existing till afaik I think he was referring to the ability to make a vfr file with mixed 24p/30p sections, not vfr via dup dropping. AFAIK, there is currently no way to do it with mp4 except for encoding the different sections separately as cfr and then concatenating them. It would be nice to be able to use a timecode file in the same way you can with mkvmerge... this inability is one of the major reasons I don't use mp4.

[bond]

I think he was referring to the ability to make a vfr file with mixed 24p/30p sections, not vfr via dup dropping. AFAIK, there is currently no way to do it with mp4 except for encoding the different sections separately as cfr and then concatenating them.i am refering to the ability of some mp4 muxers (like mp4box or 3ivx) to drop n-vops from asp streams and create a vfr stream instead during muxing