HD streams to DivX (*.ts --> *.avi) -- NEWBIE GUIDE!

[vision_sans_expertise]

In case anyone's interested, I've written a guide for transcoding .ts streams to AVI files.

Basically, I was looking for an easy way to create my own 350MB DivX/XviD rips of hour-long HD content without commercials (so, 45 minutes of actual content). Found a lot of help in various places, thought it would be helpful for the community if I consolidated into one newbie's guide.

Let me know if you have any feedback or suggestions.

Submitted humbly...

http://www.0s1s.com/wp-content/How_..._HD_Streams.pdf

http://www.0s1s.com/?page_id=21

[stax76]

Are you sure this solves a pain than cannot be solved with free tools better?

[vision_sans_expertise]

Like the footnote says, VideoReDo is my tool of choice, but HDTV2MPG or MPGVCR have worked for others. VideoReDo is the only tool that has produced successful results for me for this task. I would bet that the rest of the guide holds true no matter what your commercial editor of choice is. All other software listed is free.

If you can get something other than VideoReDo to work, more power to ya. :)

Huzzah for freedom of choice,

vision_sans_expertise

[vision_sans_expertise]

p.s. I'm assuming that everyone knows that Dr. DivX has gone open source / freeware, I didn't mention that in the guide, that will go in next revision. :)

[stax76]

Freeware cutters I know:

ProjectX
cuttermaran
mpeg2schnitt
AviSynth (as GUI supporting cutting there is StaxRip, AVSEdit and soon MeGUI)
VirtualDubMod

There are probably more free cutters, I don't know though how well they handle HDTV.

[Pookie]

Pretty good, but you need to address several issues:

Deinterlace vs IVTC vs Decimate

Selecting the correct PIDs on Transport Streams

How to do this for free - eg

ProjectX --> Mpeg2Cut2 --> Demux --> Encode -->Remux

How to deal with corrupted transport streams -->Mpeg2Repair


Still, a good start. I encourage you to expound/expand on your guide.

[GaveUp]

I've only done a quick once over, but so far I have a suggestion and a correction.

First, the switch between 2.0 and 5.1 AC3 in an AVI will not cause audio sync issues. The change that some stations make that does cause audio sync issues is a bitrate change (Fox).

Also, a suggestion would be to cover the pros/cons of GOP level cutting and frame accurate cutting.

[vision_sans_expertise]

Pretty good, but you need to address several issues:

Deinterlace vs IVTC vs Decimate

Selecting the correct PIDs on Transport Streams

How to do this for free - eg

ProjectX --> Mpeg2Cut2 --> Demux --> Encode -->Remux

How to deal with corrupted transport streams -->Mpeg2Repair


Still, a good start. I encourage you to expound/expand on your guide.

Thank you for the encouragement! I'm collecting feedback from various forums before I go to version 2.0. :) One thing to note: I'm really trying hard to keep this guide directed at complete idiots like myself. The fewer the steps, the better. I've received a lot of feedback that VideoReDo isn't a good recommendation due to its cost, but for me, the cost was justified in terms of the simplicity. All of this is to say that I'm not sure how much deeper into detail I'm going to get, because I don't want to cross the line from "QuickGuide" into "Complete Manual," you know? That said...

Re: Deinterlace vs IVTC vs Decimate. Could you please expound, or perhaps direct me to where I could learn more? I know this was something I wondered a lot about particularly when messing with AutoGK, and to be honest, with various trials I never noticed a lot of difference, which is why the guide recommends little on the topic. What would be most helpful perhaps is to hear what settings others use, particularly in Dr. DivX 2.0 OSS, specifically for HD, and why.

Selecting the Correct PIDS on Transport Streams: I thought this was covered, in the part about using VideoReDo's Select Stream option. Is there more worth documenting/going over in a QuickGuide?

How to do all this for free: The only cost incurred using this guide is the 50 bones for VideoReDo. Free alternatives are listed--see also above note about why I stand by my choice, particularly for a newbie reference guide, of VideoReDo.

How to deal with corrupted transport streams -->Mpeg2Repair I'm excited about this. I'd never heard of Mpeg2Repair, so methinks I've got another cool tool coming to the arsenal. Thank you!

First, the switch between 2.0 and 5.1 AC3 in an AVI will not cause audio sync issues. The change that some stations make that does cause audio sync issues is a bitrate change (Fox).

Good to know. I'd read otherwise elsewhere. This explains why the tighter cuts yielded better results as well. Nice.

Also, a suggestion would be to cover the pros/cons of GOP level cutting and frame accurate cutting.

Where's a good place to start on this, or, what are your recommendations on the matter?

Thanks again everyone...your help towards documentation for the layman is greatly appreciated.

Cheers,

vision_sans_expertise

[Pookie]

GaveUp - "I did not know that " (said in a Johnny Carson voice). So Bitrate is the culprit, huh? No wonder I've had so many problems with FOX.

[GaveUp]

Yes, bitrate is the problem. That's why I wrote a tool to fix it (thread for it in the forum, somewhere).

When it comes to the different ways of cutting. If you do the cutting in the TS, with say HDTV2MPEG2, then you will be cutting at the GOP level. By doing this you prevent the artifacting that happens from partial GOPs. To see this take two mpgs and join them (TMPGEnc will do the trick). At the join point you will probably see nasty artifacting ... this is the cause. There are a few tools like VideoReDo that are capable of doing frame accurate editing to an MPEG2 file by reencoding the affected GOPs. I've never messed with it so I don't know if this is the default or not (judging by the wording in your tutorial I'd wager not). I've also heard of some problems with VideoReDo to do frame accurate editing but I don't know if it was user error or what ... you'd have to test that yourself.

On the other side of the spectrum you have frame accurate editting which is done with virtualdub (avs source) and avisynth. Actually any application can do this so long as every frame in the source is a keyframe.

The difference between the two is that frame accurate editting yields 'smoother' cuts a lot of the times because the GOP going into a commercial will not end at the best possible place. For some the smooth cuts may not be an issue, but for me I find it jarring and it removes me from the show ... I keep thinking damn I should have cut that better so I choose to do frame accurate cuts.

A few other comments. Mpeg2repair. It does do a decent job on those 'problem' transports where all other tools fail (rare), but in my opinion some of its fixes are worse than the problem. For example, if there's a bad video frame it will take the bad macroblocks and replace them with black blocks. Personally, the contrast in color bothers me more than leaving the problem there. ProjectX, again in my opinion, does a much better job. For the rare times that it will fail (had it fail maybe 2 or 3 times in the last year due to bugs that have been fixed) using the chain of DGIndex->AC3 Sync Tool->AVS works great.

Maybe an option to keep this guide for the newbies but also cover different tool chains and more advanced topics would be to have links to other guides, or 'tool tips' of sorts. For me going into something I have two goals. Typically, the first thing is I want to get it working. After that I want to understand, that way when problems arise (and they will with HDTV) I know how to deal with them. I think it's entirely possible to add the advanced topics and still keep the guide newbie friendly.

Finally, good work, overall, on the guide. I had intentions of doing a newbie guide for years since the SmC guide is outdated. I just have no ambition to actually write the information up (webpage design is on my top 10 list of evil things).

[laserfan]

My own simple guide & process:

1. Edit .ts with VRD+
2. Convert to Xvid with AGK

Works great--never a video glitch or audio sync issue to-date. Sometimes I convert to SD (for a 44min show 4.5Gb to 340Mb 640x368 and about 1100Mbps), sometimes to HD (4.5Gb to 1.2Gb 1280x720 at about 3800Mbps), and sometimes I make one of each.

The "bad news": 6 hours to convert to HD, 3 hours to convert to SD.

[tomos]

personally, i run the TS through dgindex and use a script kinda like the following:

Loadplugin("C:\Program Files\AviSynth 2.5\plugins\MPEG2Dec3.dll")
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\Decomb521.dll")
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\UnDot.dll")
MPEG2Source("e:\blah.d2v")
cropbottom( 8)
Telecide(order= 1)
UnDot()

obviously i dont resize and keep the original res.
some of the above would obviously be changed every now and then, but tbh its very rare i have to do anything. mosly its the ac3 thats goosed and has to be tinkered with

then open the avs in vdub and encode to either divx@insane, or x.264 (depending on what bitrate would have to be used) - 2 passes of course (takes bloody ages, even on an x2. thank god i have 2 of them :D )

i do this with tv shows too. 3x45 min eps on each dvd5 works great too. 1 8case for each season when done (encode as 1 solid block and split afterwards to make sure the bitrate is spanned across the 3 eps as needed)

always does me well - did joan of ark to 1 dvd5 with x.264 and it looked sweeeeet :)

[Karl Beem]

Well, it's pretty obvious that you people haven't been dealing with live broadcasts of "real" video (as opposed to film), especially sports. I have a Moto 6412 DVR from Comcast and can capture the transport streams with Windows XP. In general, they are very glitchy.

I always run them thru mpeg2repair, they may not play well until I do. I demux with the latest DGIndex and prepare a .avs file. If the .avs doesn't play in sync, I use Videoredo -> .mpg (with possibly some editing) and demux the .mpg. In one case, I didn't get in sync play until all the commercials were cut out). If it's still not in sync, I try ProjectX. Lately it hasn't been of much use, giving AC3 files which are a fraction of the real length.

I use AC3 cutter for repair and editing of the AC3 file. I use VirtualdubMod for editing of the .avs file or a "raw" file produced by a relatively quick encode to 720x480 using PicVideo MJPEG at a setting of 10. This is so much easier to edit. In a few instances, I couldn't get an in sync .avs file, so I just edited with VideoRedo an produced an .mp4 file with Nero Recode 2. Even this was out of sync with the recent Duke NCSU game.

I encode with Virtualdub and xVid 1.1.0 (unconstrained) at 720p or 1080i. The Duke Georgetown (93 minutes) took 18 hours for the two passes. I usually shoot for a single DVD-r ( a bit rate of ~6000) unless the video is too long.

I use Zoom Player, BSPlayer and VLC. The latter is easiest on the CPU and is the one to check for audio sync. ZP especially pegs the cpu of my AMD64 3500+ with full 1080i or 720p encodes.

[Pookie]

Another demuxer which seems to work when ProjectX doesn't is Mpeg2cut2. http://www.geocities.com/rocketjet4/
You do need to convert your TS file to regular MPEG first, however.

Also, Comskip has recently been modified to work with HD resolution material. Might be worth looking into.

[hkazemi]

Yes, bitrate is the problem. That's why I wrote a tool to fix it (thread for it in the forum, somewhere).

When it comes to the different ways of cutting. If you do the cutting in the TS, with say HDTV2MPEG2, then you will be cutting at the GOP level. By doing this you prevent the artifacting that happens from partial GOPs. To see this take two mpgs and join them (TMPGEnc will do the trick). At the join point you will probably see nasty artifacting ... this is the cause. There are a few tools like VideoReDo that are capable of doing frame accurate editing to an MPEG2 file by reencoding the affected GOPs. I've never messed with it so I don't know if this is the default or not (judging by the wording in your tutorial I'd wager not). I've also heard of some problems with VideoReDo to do frame accurate editing but I don't know if it was user error or what ... you'd have to test that yourself.

On the other side of the spectrum you have frame accurate editting which is done with virtualdub (avs source) and avisynth. Actually any application can do this so long as every frame in the source is a keyframe.

The difference between the two is that frame accurate editting yields 'smoother' cuts a lot of the times because the GOP going into a commercial will not end at the best possible place. For some the smooth cuts may not be an issue, but for me I find it jarring and it removes me from the show ... I keep thinking damn I should have cut that better so I choose to do frame accurate cuts.

A few other comments. Mpeg2repair. It does do a decent job on those 'problem' transports where all other tools fail (rare), but in my opinion some of its fixes are worse than the problem. For example, if there's a bad video frame it will take the bad macroblocks and replace them with black blocks. Personally, the contrast in color bothers me more than leaving the problem there. ProjectX, again in my opinion, does a much better job. For the rare times that it will fail (had it fail maybe 2 or 3 times in the last year due to bugs that have been fixed) using the chain of DGIndex->AC3 Sync Tool->AVS works great.

Maybe an option to keep this guide for the newbies but also cover different tool chains and more advanced topics would be to have links to other guides, or 'tool tips' of sorts. For me going into something I have two goals. Typically, the first thing is I want to get it working. After that I want to understand, that way when problems arise (and they will with HDTV) I know how to deal with them. I think it's entirely possible to add the advanced topics and still keep the guide newbie friendly.

Finally, good work, overall, on the guide. I had intentions of doing a newbie guide for years since the SmC guide is outdated. I just have no ambition to actually write the information up (webpage design is on my top 10 list of evil things).

1.) the links to the guide don't work anymore...
2.) regarding mpeg2repair:
from the readme.txt included with mpeg2repair:
"FixMacroblockErrors=1" - changing this to '0' will cause the program to not repair corrupted pixels. The program will still perform all the other tasks listed below. This may be useful for people that just want to use this tool to remove sub-channels, remap PIDs, strip null packets, etc. but want to preserve the original video.

program's author (Wizziwig) says:
If you set "FixMacroblockErrors=0" it will not replace bad pixels with black blocks. Depending on your player, it might work better or worse by leaving the errors in the file unrepaired. If you're going to replace the bad frames from another recording anyway, then it should not matter if it's repaired or not."

I've used the latest version of mpeg2repair 1.0.1.5 in this way and it leaves the damaged video frames alone. I too prefer to leave that damage in place (I let mpeg2repair correct other low level errors and then let VideoRedo Plus Quickstream Fix have a go at it). I hope an Avisynth filter can be derived to repair damaged frames in the future by DGDecode passing damaged frame hints to filters the use spatial and temporal interpolation to regenerate lost content (maybe use logic from mcBob). More on this idea:
http://neuron2.net/board/viewtopic.php?t=629&postdays=0&postorder=asc&start=50

[chillin8]

This is a good start...I've tried just about every app..mentioned here..and I am happy with VideoRedo..I mostly capture Mpeg2 1080i and 720p trp.files straight to my external hard drive and VideoRedo reads it just fine...However i haven't found a app. that can read Mpeg4 files....but for now VideoRedo does the job....

[phædrus]

Just a note about MPEG2Repair vs videoredo, I tried VideoRedo and it seems like its default is to remove a lot of problem frames rather than fix them. Look I am a near complete idiot, so I am not an expert on program options -- maybe that can be adjusted. All I'm saying is that if you are recording a cop show or a comedy, a few dropped frames because of corrupt audio or video is not bothersome to me. But I record a lot of music specials, and if frames get dropped, then the rhythm of the song is affected. You lose a bar or half a bar of music, it's quite annoying (to me). MPEG2Repair will usually fix the stream enough for encoding without losing sync, whereas the same file with VideoReDo would have lost frames (though still in sync at the end). I don't like musical hiccups. In a TV drama it usually isn't a big deal. If the stream is really bad, you might do better with redo, and just accept the loss of frames. I usually have a pretty clean recording, I am about 5 miles from all the transmitters. So MPEG2Repair is a good tool for me.

[hkazemi]

My current preference is to run mpeg2repair first (set to not replace corrupt blocks with black), and then run Videoredo quickstreamfix (QSF).

The default in Videoredo is to cut audio frames to retain sync...but this can be changed to insert extra video frames instead. It's not a foolproof change, as Videoredo (vrdo) will still cut audio frames under certain conditions but its not as likely and not as many audio frames will be cut. If in the future Videoredo gets an option to insert video & audio frames to maintain sync, I'll certainly be using that option.

[rebkell]

This is a fascinating discussion, I record tons of shows and I am always having audio sync issues. Unfortunately I have nothing to add to the discussion, but I really appreciate all the input.

Another question about mpeg2repair, will it only work on transport streams? I am recording nearly everything in Vista Media Center(it's been absolutely rock solid as far as not missing any recordings), but it always puts the shows in the dvr-ms wrapper and the only thing I know of to extract the mpg is video redo and it works, but I do still have a few sync issues when I encode.

[luckyhell]

I am happy with VideoRedo..I mostly capture Mpeg2 1080i and 720p trp.files straight to my external hard drive and VideoRedo reads it just fine

???
Videoredo don't read .trp files !!!

[rebkell]

???
Videoredo don't read .trp files !!!

It does if they're mpeg2, trp and ts files are basically the same thing with a different extension(as far as most HD(ota and qam) captures in the US) Most capture card software allows you to select the extension you want to use on your captured files. the normal selections in my experience have been tp, trp and ts.