DTV tools -- .TS -> AVISynth?

[dvgeek]

This has been a very enlightening thread. I currently use the modified ProjectX for 59.94fps, which seemed to fix my sync issues. (My caps are mostly ABC-Alias and Fox-24).

I demux in ProjectX, encode in WMV9 single pass (for a faster processing - 2hrs HD in about 6 hours, XVID - 2pass takes about 24 hours for the same avs script, which is just a one-liner for loading the .d2v).

I use VirtualDubMod for encoding with Fast Recompress - no audio.

I addin the ac3 file after encoding - edit out the commercials and save the avi with a direct stream copy. I always have audio interleave at 32ms.

Does anyone know if the latest ProjectX releases have got the 59.94fps mod incorporated ? Thanks.

[FreQi]

Seriously, you should not be cutting commercials after you encode. You're easily encoding 17 to 20 minuets of material that you are going to throw away. Cutting commercials first will shave 30% or more off of your current encode times.

Also, for shows like 24 and Alias, you should also be Decimating it to 23.96fps. The show is origiannly filmed in 24fps, so you are not loosing anything. You only gain more bits to allocate to the frames and ending up with a higher quality encode.

[dvgeek]

Thanks for the suggestion. I've had so many audio sync problems when I edited out commercials with HDTV2MPEG that now I've given up on that tool. It's a great tool - fast and all - but sadly for me, I could never get audio sync - no matter what method I tried.

So, now I clean up the transport stream by demuxing with ProjectX, then use DGIndex to create the D2V file and use the latest version of AVISynth/VirtualDub to encode into WMV9 HD. Then I use VirtualDubMod to mux in the AC3 audio and cut out the commercials - I do not mind the extra time as it gives me audio in sync.

Also - if I use fdecimate() or any other form to decimate to 23.976fps, my video gets choppy as if it suddenly decides to switch field priority. So what if my video is 10% more - I get smooth motion by keeping the original frame rate. I've all but given up on decimating to 23.976fps or do edits before encoding. I'll keep experimenting though. Thanks again.

[Leehro]

actually your decimate problem could be because of commercials. if you're encoding with commercials (often interlaced content, or at least a different field/frame offset) that can screw with decimate.

i had some sync problems too, but i eventually used VDubMod to make all the edits, save the frame ranges into avisynth, and fix the chopped up ac3. after that i think it was just a constant audio offset. i'm not using hd2mpeg anymore at all. just dgindex, vdumbod, avisynth, and whatever the latest greatest ac3 cut fixer is.

edit: still looking for anyone with a good script for 1080i->480i

[nnigam]

Originally Posted by dvgeek

Also - if I use fdecimate() or any other form to decimate to 23.976fps, my video gets choppy as if it suddenly decides to switch field priority..

Is your original source at 23.97fps. Before using Fdecimate, use Fdecimate(metrics=true) in the avisynth script. Then curser through your video. You will see a large change in the number when a new frame is found instead of a duplicate frame. You can use the largest of the small numbers as the threshold for fdecimate, and see how many actual distinct frames you have to get what the actual frame rate should be. Most of my hd stuff is 23.976 fps which has been converted to 29.97 by adding duplicating one frame, and then every other frame is duplicated. I also have some which are at 29.97 and only every other frame is duplicated. If this is converted to 23.976 it will be choppy.

I also agree with you that a little larger video to get better audio synch and smoother picture is worth it. Many times

[Ac3Dc3]

This has got to be one of the best threads on converting HDTV out there, thanks for all the info!

does anyone know where i can get FreQi's tool theeo? and is it up to date with the current version of DGIndex?

[FreQi]

Interesting that you ask about theeo. I was working on it last night for the first time in maybe 6 months. It is true that theeo exists, and it does work, pretty well. However, when DGIndex stopped putting AC3 in the name of the files it demuxed, it threw a wrench in there that I've been just working around ever since. I finally got tired of it and decided to fix it, but instead of fixing it, I redesigned the interface, and pretty much started over. I am currently trying to decide if I want to actually start completly over and use Visual Studio 2005 and .NET2.

If you're interested, I could get you up and running with what I'm calling "theeo-v0.00" as I rework the tool. In any case, I would like to get your feedback on it. Send me a PM.

[FreQi]

I've just posted the first release of theeo. I'd be interested in getting your feedback in that thread

[coolyhu2]

Thx


~~

[G_M_C]

Hmmm, reading throught this thread becaus this week im gonna try my first .TS to XviD.

As I understand it it's only needed to use HDTV2Mpeg2, and than DVD2AVI to make it possible to open the stream in avisynth through the Mpeg2source() command ?

And if true does it really need to be (the special version of) DVD2AVI ? Or can DGindex (DGMPGDec) do the same thing ? This because this thread says DVD2AVI has to be considerated "obsolete" (because it makes errors). And I've allready got it installed (and having several different versions of MPEG2DECx.dll is a problem waiting to happen). And if DGIndex can work, does the "hints" switch work too (for use with Colormatrix) ?

If I cant use my allready installed DGIndex/DGMPGDec, does the "hints-switch"(for the use with Colormatrix) work with that special version of DVD2AVI ?

I want to have this info, so i cam make my own AviSynth script, so the only thing I do need to be shure of is the bvest way to open the .TS in the script

[squid_80]

DGIndex can handle unprocessed .TS streams ok, as long as there's no corruption (common with DVB-T, I don't have experience with anything else).

[FreQi]

There are of course several ways to get your .TS's into an .AVS script, but the "best" way is purely subjective. Some people use HDTVtoMPEG2 to do their trimming, others use ProjectX. In either case, I would suggest that you use DGIndex to generate the .D2V that you load in your .AVS since it is true that DVD2AVI has been pretty much abandonded in favor of DGIndex.

The way I process the .ts's has changed in the recent past. I used to use just DGIndex then cut the video in AVISynth and the Audio with VDMod. This was perfect in most cases, but would ocassionaly result in desync'd audio, especially if there were problems with the recording (drops, etc). It seems the more predictable method that I have been using is to use ProjectX to trim the unwanted segments then demux the .TS to it's MPV and .AC3 elements. Because ProjectX "fixes" drops and maintains A/V sync, you never have to worry about things like AC3 Delay anymore because it's allways 0ms. Then I use DGIndex to generate a .D2V to open then .MPV with AVISynth for trans-encoding since DGDecode is faster than other mpeg2source() plugins. The main downside to this process is the near doubling of hard drive space required to do the work, but it's a small trade-off for the reliability of the end product.

[G_M_C]

Well thx guys

All the answers I needed. So hope to be able to give it a try this weekend

[jmac698]

I record the .ts, then use mpg2cut2 to take out commercials. This app is very fast as it simply does a binary copy of the remaining parts of the file, there is no processing at all. The only thing fancy about this file splitter is that it understands mpeg2 enough to find the exact byte to cut the file on. This byte is called a "GOP boundary", roughly speaking. A GOP is a set of, say, 15 pictures that can be independently decoded from any other part of the file, so you can call it one "atom" of film.
This should tell you the disadvantage of this method - you can only cut to the nearest 15 frames or 1/4 of a second, or so. This is usually good enough to cut perfectly.
The advantage is that there is no change in sync, because nothing in the file has changed, in a sense. It's also as fast as your hard drives. (hint - save to a different hard drive).
After this, just load in StreamClip and Convert to AVI, and your done! For full quality you can also Convert to MPEG2, this saves a bit of space but makes it a more compatible file, an mpeg2 (instead of transport stream).

[FreQi]

I can't offer perspective on a speed comparison, but it sounds like the same thing that HDTVtoMPEG2 (forum) would do. ProjectX also cuts on GOP, but it also corrects errors in the recorded stream, so I count it as a bonus. It'll also generate a .D2V for you, but it doesn't use the fancy format that DGIndex will make, so you have some options there. You can also use pjx to process to a new .ts instead of demuxing if you really wanted to.

[FreQi]

So after a lot of encoding, trying different apps and writing scripts and apps like theeo and tons of trial and error, I've managed to come up with what I consider to be a really simple and straight forward way of pre-processing HDTV Transport Streams to the point where you'll have an AVISynth script that you can feed into whatever encoder you want.

How To get Transport Streams into VirtualDubMod via AVISynth using ProjectX and DGIndex

What I have done is created a system that uses ProjectX as the starting point, then uses DGIndex to generate the .d2v project and the AVISynth script. Then, just to take it one step further, I made it open the .avs in VirtualDubMod so I could easily set up the encode jobs. The funny thing is, it's all bundled together by a windows batch script. Here's how...

First, I made a directory structure like this:

C:\Program Files\Encoding
C:\Program Files\Encoding\DGIndex
C:\Program Files\Encoding\VirtualDubMod
C:\Program Files\Encoding\ProjectX

In the DGIndex directory, I put DGindex 1.4.8 (found here), and two AVISynth Templates which I've listed below (modify them to your needs, but these are what I use and the resolutions I like to encode my XviD's to).

720p-to-FILM.avs

LoadPlugin("C:\Program Files\Encoding\DGIndex\DGDecode.dll")
MPEG2Source("__vid__")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Decomb.dll")
SelectEven().Decimate(cycle=5).Crop(4,4,-4,-4).Lanczos4Resize(960,528)

1080i-to-FILM.avs

LoadPlugin("C:\Program Files\Encoding\DGIndex\DGDecode.dll")
MPEG2Source("__vid__")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Decomb.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Uncomb.dll")
Uncomb().Decimate(cycle=5).Crop(4,4,-4,-8).Lanczos4Resize(960,528)

In the VirtualDubMod directory, I put VirtualDubMod 1.5.10.2 build 2542 (found here).

In the ProjectX directory, I have a couple things. First, I extracted the "launch.exe" program dvd.matt linked me to (found here). I also created two batch files that reside there.

MakeD2V-720p.cmd and
MakeD2V-1080i.cmd

@echo off
echo.

IF [%1]==[] GOTO NoArgs
set STRIPTIN=%~dp0

:: this is the template DGIndex will use to generate your AVISynth Script.
set AVSTEMPLATE=720p-to-FILM.avs

::the only thing you can pass is a filename, so if there are
::more "args" then assume it's a filename with spaces.
set INFILE=%1
:getname
shift
IF "%1"=="" GOTO gotname
set INFILE=%INFILE% %1
GOTO getname
:gotname
::launch.exe seems to put quotes around the filename
::so trim the double quotes off both ends
set INFILE=%INFILE:~1,-1%
echo. Source File: %INFILE%

:: trim the file extension off the end
set OUTFILE=%INFILE:~0,-4%
echo. Output Base: %OUTFILE%

set DGINDEX="%STRIPTIN%..\DGIndex\DGIndex.exe"
set ARGS=-IF=[%INFILE%] -YR=1 -FO=0 -OM=2 -AT=[%STRIPTIN%..\DGIndex\%AVSTEMPLATE%]-OF=[%OUTFILE%] -exit

echo. Launching DGIndex to make .d2v project file...
echo %DGINDEX% %ARGS%
echo.
%DGINDEX% %ARGS%
echo.

IF NOT EXIST "%OUTFILE%.avs" GOTO NoAVS
IF NOT EXIST "%STRIPTIN%..\VirtualDubMod\VirtualDubMod.exe" GOTO NoVDMod
echo.
echo. AVISynth Script found, opening in VirtualDubMod...
IF EXIST "%OUTFILE%.avs" "%STRIPTIN%..\VirtualDubMod\VirtualDubMod.exe" "%OUTFILE%.avs"
GOTO end

:NoArgs
echo.
echo This script is meant to be run by launch.exe demuxing with ProjectX.  To use
echo it, open the PreSettings and on the postprocess tab and put something like...
echo.
echo. "C:\<pathTo>\launch.exe" open "C:\<pathToThisScript>\MakeD2V.cmd" ?1
echo.
echo in the demux line.  Also, on the ProcessWindow (click "prepare>>"),
echo make sure you have "enable post processing" checked under the Edit menu.
echo.
pause
GOTO END

:NoAVS
:NoVDMod
:end
::exit

The only difference between the two scripts is the value of AVSTEMPLATE. 720p uses the 720p template, 1080i uses the 1080i template (duh?). The ProjectX directory also contains two copies of ProjectX 0.90.04.00 (found here) that are in their own subdirectories cleverly named \720p and \1080i (see footnote1).

So, I extracted one copy of ProjectX to ProjectX\720p and set it up the way I like (see the "First Time Configuration" section of my guide for details). In addition to that, I put the following command in the "demux" box under the postprocess tab of the PreSettings:

"C:\Program Files\Encoding\ProjectX\launch.exe" open "C:\Program Files\Encoding\ProjectX\MakeD2V-720p.cmd" ?1

When I was done, I copied the 720p directory to the 1080i directory, but I opened up ProjectX and changed the demux line under the postprocess tab to use the MakeD2V-1080i.cmd script instead (make sense?).

To finish the setup, I just created shortcuts to the ProjectX binary and put them in my quicklaunch for easy access.

So, now when I have a show I want to encode that's been recorded from a 720p source, I just open that flavor of ProjectX, drag-n-drop the source files on it, cut the commercials (as outlined in my guide) and click the "Quickstart" button. After a couple minutes of processing, DGIndex opens up, makes the d2v and avs script then opens it up in VirtualDubMod for me to double check and setup the encoding job (see footnote2).

Done. Wha'd'ya think?



footnote1
Why two copies of ProjectX? Because Interlaced video needs to be processed differently than Progressive video, and to make that happen, i needed ProjectX to lauch a different batch script when it's done demuxing. The only way to do that is to either

(a) change the Post Processing command every time I switch from I to P or back,
(b) create different projectX .ini files and load them from command line or in the target properties of a shortcut to the jar (ie: C:\WINDOWS\system32\java.exe -jar "C:\Program Files\Encoding\ProjectX\ProjectX-v0.90.04.00-20060330.jar" -ini "C:\Program Files\Encoding\ProjectX\720p.ini" -gui)
or
(c) you can just make two copies, one with the settings for 720p material and one for 1080i.

It seemed like less work, so I opt'd for c.

footnote2
Instead of opening VirtualDubMod, there is a ton of other things you could make the cmd script do. You could launch x264.exe to encode it, or feed the avs to avs2avi instead. Go nuts.

[Livesms]

I have HDTV (1080p) video (Gladiator) with 2 audio streams (eng and rus) - 21Gb. PID 011 fo video, 014 for rus audio and 015 for eng.
I want to reencode it with x264 to fit 2DVD.
I tried to put it to Avisynth using d2v generated by DGMPGDec 1.4.9 Final but it always says smth like filed order was incorect (generating d2v) and then eng audio stream is shorter then rus (2:34:58 vs. 2:33:51). d2v files generated by DGMPGDec 1.4.9 Final is differs when i generate it using Streams 011, 014 and when i do the same task with 011, 015.

How can I reencode correctly?

[FreQi]

I don't think you need to worry about the runtime for the audio being different by one minute. There's probably just no audio at the end durring the credits or something.

I also think you might want to take your question to a thread about DGIndex. This thread is more about ATSC HDTV Broacast Transport Streams, not HDDVD/Bluray (although you should be able to use much of the info here).

[Livesms]

Originally Posted by FreQi

I don't think you need to worry about the runtime for the audio being different by one minute. There's probably just no audio at the end durring the credits or something.

I also think you might want to take your question to a thread about DGIndex. This thread is more about ATSC HDTV Broacast Transport Streams, not HDDVD/Bluray (although you should be able to use much of the info here).

I checked audio in original ts for both PID - both in eng and rus audio run till the end of the video and ts has length of 2:34:58

And seems like AC3 audio generated be DGindex is not the same audio i hear with original TS file.
I just put the slider into random position (for ex. 0:24:40) and TS audio is far differ (by time I mean) from AC3.

When I use HDTVtoMPEG for converting TS to MPEG i have the same problem - video is trancating to 2:33:51 (from 2:34:58).

[FreQi]

Well, like I said, you'll probably want to take this to a DGIndex thread...