Before I start encoding this 732 episode project, could I get a check of my workflow?

[kuchikirukia]

I'm starting a rather large project on a source that needs some cleaning. Since I'm new to this x.264/Avisynth stuff and a month of study and trial and error still hasn't gotten me to the point where I really know what half of this stuff is doing, I'd like someone to double-check that I'm not doing anything stupid or pointless -- especially the whole "extract to mpeg2 and run DGIndex" part, since I really have no idea if the d2v file is adding anything to what TFM and TDecimate are doing, and it saves me a ton of work if I can skip steps 3, 4, and 5 and just run everything from the mkv file using ffms2.

Here's what I'm preparing to use:

1. Use MakeMKV to extract whole disc to MKV so I have something I can work on.
2. Use mkvmerge GUI chapter splitting to extract the episodes.
3. gMKVExtract GUI to extract the video and audio.
4. Play .mpg through in DGIndex, IEEE-1180 reference, honoring pulldown flags (no idea what that does), maximum speed. *(I have no idea if this playthrough step does anything for me -- I just noticed it puts Field Repeats in the log file, which you don't get that if you skip right to Save Project. I don't know if Field Repeats is useful or even being used anywhere.)
5. Save DGIndex project.
6. Open first pass AviSynth script in VirtualDub and Run Video Analysis Pass:
#load plugins
DGDecode_mpeg2source("E:\Video\source.d2v", cpu=0, idct=5)
tfm(d2v="E:\Video\source.d2v",output="E:\Video\tfm.txt"), .tdecimate(mode=4, output="E:\Video\stuff.txt")

(VirtualDub is significantly faster than MeGUI's analysis pass in this. 4:10 vs 6:00)

7. Run encoding script in MeGUI:
#load plugins
DGDecode_mpeg2source("E:\Video\source.d2v", cpu=0, idct=5)
edeintted = SeparateFields().SelectEven().EEDI2(field=-1)
tdeintted = TDeint(edeint=edeintted)
tfm(d2v="E:\Video\source.d2v", input="E:\Video\tfm.txt")
tdecimate(mode=5, hybrid=2, tcfv1=false,vfrDec=1,input="E:\Video\stuff.txt",tfmIn="E:\Video\tfm.txt",mkvOut="E:\Video\mkv-timecodesfile.txt")
RemoveGrain(mode=1)
crop(4, 0, -4, 0)
bicubicresize(640,480,0,0.5) # Bicubic (Neutral)
lsfmod(ss_x=2.0, strength=100, edgemode=1, soft=0, soothe=false, overshoot=0,smode=5)
lsfmod(ss_x=3.0, strength=120, soft=0, soothe=false, overshoot=0,smode=5)
Edgecleaner(strength=22, smode=1)

"C:\Program Files\MeGUI_2507_x86\tools\x264_10b\x264-10b.exe" --preset veryslow --tune animation --crf 13.5 --keyint 240 --qcomp 0.7 --aq-strength 0.7 --nr 100 --no-fast-pskip --sar 1:1 --output "E:\Video\encoding vfr.264" "E:\Video\encoding vfr.avs"

8. Encode audio.
9. Mux audio with video+timecodes with mkvmerge.

--------------------------------------------------------------------------------------------------------------------------------------

So, questions:
1. Does the DGIndex playthrough do anything for me? (If I can skip that and only that it saves me 4 minutes of waiting)
2. Does the d2v + DGDecode give me a more accurate IVTC (or anything), or am I wasting my time with those steps. If they're a waste, should I use ffms2 or is there something better?
3. Is there a better way to do sharpening/antialiasing than LSFMod's supersampling? I tried MAA2 to smooth out a lower-AA LSFMod, but it was unbearably slow and it Dr. Watson'd MeGUI every time it started. (though the encode would still go through to finish.)
4. Am I using LSFMod stupidly? Since max sharpening=200 at 3.0 supersampling doesn't give me all the sharpening the source needs I added another instance of sharpening. But I see there are different kernels to LSFMod's sharpening, but I have no idea what those are or do.
5. Is the edeintted/tdeintted being in there going to help/hurt anything? After a week of banging my head trying to get VFR to work I ran MeGUI's AVS Script Creator, picked the TIVTC+ TDeint(EDI) option, and tweaking the script it gave me with TIVTC's VFR options somehow gave me working VFR over what I had been doing earlier (even though I see absolutely no difference over what I gave TFM and TDecimate before), so since it's working I haven't wanted to mess with it.
6. Avisynth 2.6 MT. I've read some places that it doesn't play well with TIVTC. Is this still true, or can I use it? I did play around with it back when I was doing CFR and it seemed to be working (LSFMod was having a problem until I disabled the post-processing, after which it was good to go with setmtmode(2)), though I noticed a frame jump here and there, I'm not sure if it was the multithreaded or just my settings were wrong.

And general question: What does the period do in Avisynth? I see TFM().TDecimate() a lot. Is that just the same thing as a new line or does it say, "Hey, we're using the same DLL here!" On the chance of the latter I went LSFMod().LSFMod(), though I have no idea if it does anything.

[kuchikirukia]

AviSynth 2.6MT testing on Pass 1:

Setting mode 2 before the source screws everything up. TFM and tdecimate outputs are 1/4 the size. (OK, that was stupid.)

setmtmode(5,2) before the source (switched to 2 threads seeing that it was only using 10% per thread of 4) and mode 2 after, TFM's output file looks fine from spot-checks, though it's missing the entire last section of remarks on OVR HELP INFORMATION:.
Tdecimate is still missing frames in its output file.

Also, this is only a hair faster than single-threaded.

Mode (3,4) before source and mode (2) for TFM/Tdecimate:
Speed is back up. (3:12 vs 5:20)
TFM and tdecimate outputs are missing frames.

Hmmm... I wonder if it was the mode 3 for the source that changed TFM vs mode 5 or the change from 2 to 4 threads.
-- It looks like it was the 4 threads. Going to (3,2) before source gives us the same output as single-threaded using mode (2) for TFM, including the remarks section. Gonna run it a couple more times to see if it's 50/50 whether we get those remarks.
Tdecimate missed half the frames.
-- ran three more, TFM came up with the remarks in each. Not conclusive, though.

Only a slight speedup over single-threaded. (4:40 vs 5:20)


Let's see if TFM will do quad threads when it is running in mode 3 rather than just the source.
Nope, virtualdub locks up.
Tried 2 threads and 1. TFM won't work in setmtmode(3). Putting it back to (2)

Let's see if tdecimate works in (3) with TFM in (2):
-- no, that locks up virtualdub as well.


Mode 4 on tdecimate gives access violation.

Mode 5, barely any speedup (a couple secs; within the margin of error). I guess it was the broken tdecimate that gave us the speedup in mode 2, not TFM's operating in mode 2.
Tdecimate's output is good, though

Tdecimate with setmtmode(6):
Slight speedup. (4:53)
Output good.
Note, TFM has been operating under mode 2 and we just lost the remarks section at the end of its output file again. Looks like it may be 50/50 whether that gets added properly if running it 2 threaded in mode 2.

TFM under mode 6, quad threaded:
Horry shit we are off to the races again! 3:17.
BOTH TFM AND TDECIMATE OUTPUTS LOOK GOOD

LoadPlugin("C:\Program Files\MeGUI_2507_x86\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\Program Files\MeGUI_2507_x86\tools\avisynth_plugin\TIVTC.dll")
SetMemoryMax(512)
setmtmode(3,4)
DGDecode_mpeg2source("E:\Video\source.d2v", cpu=0, idct=5)
setmtmode(6)
tfm(d2v="E:\Video\source.d2v", output="E:\Video\tfm.txt")

setmtmode(6)
tdecimate(mode=4, output="E:\Video\stuff.txt")


Looking at Process Explorer, it seems that all TIVTC instances are spawned under one VirtualDub thread in mode 6, while DGDecode gets run across 4 Avisynth threads.
A second Virtualdub.exe thread runs almost maxed, mostly with just virtualdub.exe under it, although the occasional avisynth.dll.

Since process explorer is only showing those 4 avisynth threads using 5-10%, let's see what happens if we go to two threads.
(3:12)

Going to setmtmode(6,2) for the source brings everything back to single-threaded. No avisynth.dll threads are spawned and everything avisynth related gets run under a single virtualdub thread. Though the second virtualdub thread seems to be slightly heavily loaded than under single-threaded and the time is 5:12 vs 5:20.

Now I guess the question is whether TIVTC works correctly under mode 6 when encoding.
e: Running on the absurdly hybrid opening of this series, it all looks good.

[foxyshadis]

I edited your script's lines to fit the window; as you suspect, a period is basically the same thing as a newline. (Not 100% -- there are times when later functions can access earlier inputs with a period -- but that's confusing and it's always better to explicitly assign the clip instead.)

1. DGIndex playthrough does nothing, all that matters is that the d2v is always the same. Honor pulldown is an absolute must if you have hybrid video.
2. It doesn't much matter. DGDecode is nice because it can work right off the vobs, but if you're pulling them into mkv anyway, you could use ffms2 or LWLibavVideoSource directly on the original mkv, then trim for episodes, or directly on the split segments. Always use repeat=true with ffms2 or libav for hybrid.
3 & 4. Sharpening AND antialiasing is a tough bugger, and yeah, that's overkill and slow as hell. Doubling up on LSFmod isn't bad, but at such a huge supersample it's a waste. Better to sample it up yourself and then do processing on it. A better option to start off with might be DAA (http://avisynth.nl/index.php/DAA), substituting EEDI2 for NNEDI3 if necessary (EEDI2 tends to be better for anime), then use LSFmod with a much smaller supersampling (less than 2), then end with FineSharp or SeeSaw for final sharpening. If that re-introduces too much aliasing, you might have to drop them, but it's worth trying.
5. External deint is for better quality with lousy fieldmatching, or for the 30/60fps parts of hybrid video. Since you never even use tdeintted anywhere (it would go in the TFM line as clip2=tdeintted), you don't get the benefit or the speed hit.
6. The MT problem was more that it slowed the whole script down, but that may not still be the case. I dunno.

Now I have to ask: Have you checked that this project is actually hybrid anywhere? You could save a lot of time if it's not (or just barely is, and you're willing to force the little bit that isn't).

Another comment: Double-check the A/V sync after you split the mkv, because the nearest keyframe might not exactly match the split point, whereas the audio will always be exact. I would split within Avisynth to avoid that problem, using AudioDub(LWLibavAudioSource("yourfile.whatever",av_sync=true)) immediately after the video source. Saves you time splitting too.

[kuchikirukia]

"FFVideoSource does not have a named argument "repeat"."

I tried sampling it up myself, but it always introduced aliasing -- LSFMod would alias like crazy at supersample of 1.0 which would remain on the downsize.

Thanks for the DGIndex info. It saves me so much to be able to use MeGUI's one-click.

[foxyshadis]

Whoops, I forgot that ffms2 uses rffmode=1 for Honor Pulldown.

[kuchikirukia]

Does that flag only have to be in the first metrics pass, or does it have to be in the encoding pass as well. (The opening seemed to encode fine without it in either)

If it needs to be in the encoding pass, is there a way to add it to MeGUI's one-click avisynth <input>?

[kuchikirukia]

What Masktools should I be using? I, erm, seem to have both mt_masktools-26.dll from masktools-v2.0a48.zip and masktools2.dll from masktools2-x86.zip.

-- I switched to the masktools2 b1 since its wiki mentions it's an improvement on 2.0a48.

E: Ok, no, that gives me endless errors.

[kuchikirukia]

From what I'm reading and seeing I can't use FFMS2 for VFR.

First off, using the rffmode=1 gave me interlacing artifacts. Also, if I don't use fpsnum/fpsden, it runs at a constant speed that is the average of the VFR framerate. If I do use it, from what I'm reading, ffms2 is creating duplicate frames to achieve it. I don't want that!
https://ffmpegsource.googlecode.com/svn/trunk/doc/ffms2-avisynth.html
int fpsnum = -1, int fpsden = 1
Controls the framerate of the output; used for VFR to CFR conversions. If fpsnum is less than or equal to zero (the default), the output will contain the same frames that the input did, and the frame rate reported to Avisynth will be set based on the input clip's average frame duration. If fpsnum is greater than zero, FFVideoSource will force a constant frame rate, expressed as a rational number where fpsnum is the numerator and fpsden is the denominator. This may naturally cause FFVideoSource to drop or duplicate frames to achieve the desired frame rate, and the output is not guaranteed to have the same number of frames that the input did.

No fpsnum/fpsden, frames = 2159
fpsnum 30000/fpsden 1001, frames = 2212

The latter gives me a timecodes.txt file that's usable to result in a VFR output, but I question whether it's giving me the correct framerates for the individual sections if it's making room for duplicate frames.

Hmmm... dgdecode gives me 2212 as well, but it has 30000/1001 in its d2v file. Why does Handbrake give a 2340 frame VFR file?

[kuchikirukia]

Ok, masktools2 b1 gives me frames here and there like this:
http://s30.postimg.org/lrf49t2y5/Bleach_2_mkv_snapshot_13_27_2014_11_02_15_14_31.jpg (http://postimg.org/image/lrf49t2y5/)

It also seems to be slower than 2.0a48.

e: wait, I'm getting it with both versions? Where the heck is this coming from all of a sudden. I've done hundreds of tests and never seen this.
Let's try setmtmode(3) for lsfmod and edgecleaner. If that doesn't work, I guess I'll try going back to avisynth+. I'm not horribly bottlenecked by single-threaded except for the VFR first pass.

e2: oh well THAT didn't work. Setmtmode(3) gives me flashing.
I'm going to try running the encoding pass without any setmtmodes. Thankfully it doesn't look like that's going to slow me down any. Even though CPU util is dipping down to 70% a lot as it's waiting on the avisynth thread, speed is still comparable. If I was trying to encode any faster than Very Slow I'd be bottlenecked and certainly want to double up on the workers, but with my current settings I'm good.
Now hopefully this gives me a clean encode.

[foxyshadis]

Starting with the interlaced frames is integral to working with VFR DVDs -- you can't stick to pure progressive because then you won't know which areas are and which aren't. You have to let TDecimate determine that. Otherwise the audio sync will be all off, even if the average framerate is correct.

If you are going to force a single framerate, better to force 24 since the main episode is that, instead of 30 which is normally only used for the credits. You can just force TFM to deinterlace what it can't match.

[kuchikirukia]

I'm not sure if I'm accidentally forcing anything, thus the question. It seems to work with 30000/1001 -- tdecimate is giving me 24fps in the IVTC film and 30fps in the video, I was just wondering if it was adding additional spurious frames to the stored 24p sections trying to raise it to 30fps. And Handbrake giving me significantly more frames while still retaining a 1:30 clip is also throwing me.

But anyway, both ffms2 with fpsnum/fpsden 30000/1001 and dgindex with its d2v saying "Frame_Rate=29970 (30000/1001)" give me outputs with good 24/30p.

e: Oh, and if someone with an avisynth wiki account wants to change it, the entry for supersampling for SeeSaw is wrong:
http://avisynth.nl/index.php/SeeSaw
It's ssx/ssy, not ss_x/ss_y.