Cutting commercials from HD Transport Streams

[FreQi]

I recently discovered that my local FOX affiliate amp'd up their signal, so now I can get their 720p HD signal. My ABC affiliate also uses 720p, however they are using PID's 0x11 and 0x14. FOX is using 0x21 and 0x24. Now, I don't necessarily want to pin that as the problem, but I can't seem to find any other excuses.

I first use DVD2AVI2 v1.83.5 (http://forum.doom9.org/showthread.php?threadid=49866) to read the .ts's and demux the 5.1ch .ac3 audio and create the .d2v files. Next I make an AVISynth script to open the .d2v using the MPEG2Dec3 plugin by MarcFD (http://ziquash.chez.tiscali.fr/). I open that script using the old v1.4.13 copy of VirtualDubMod. With that I set the audio to Direct Stream Copy, select the .ac3 and set it's delay. Finally I can cut the commercials out and demux the audio.

This system has worked almost flawlessly on my ABC, NBC and CBS affliates, however with FOX, when I demux the audio I only get a very about 50% of what I was expecting (it's a lot shorter than the final runtime), and the commercials and other parts I cut out are still there. It's as if it just demuxes the audio from the beginning, ignoring the Trim's completly.

A friend and myself have been wrestling with this for several weeks, trying to encode The O.C. in preperation for the next season of 24. Is there some known problem with this version of DVD2AVI and transport streams with PID's of 0x21 and 0x24 ?

What do you other people use to cut commercials out of recordings? I've tried using ProjectX to demux and make the .d2v but the same problem comes up. I'd be more than happy to make samples of my transport streams available if that would help...

[Emp3r0r]

@FreQi: I've too been searching for a reliable tool to cut commercials for some time now. The major tools I've tried are mpeg2vcr, projectX, pvastramento+mpeg2schnitt, and VideoRedo. I also tried your method. None have worked perfectly since I have imperfect reception (dropouts) they always get out of sync due to the corruption.

Recently, I retried using VideoRedo, and it worked! It can now open HDTV files without crashing. It doesn't choke on corrupted streams and it re-encodes the frames at the beginning of each cut to be frame accurate (very nice). The good news, VideoRedo has erased all sync issues I was having. The bad news, the current release doesn't work with transport streams.

A new release of VideoRedo is expected soon that should work with transport streams. Until then you could try converting the file to a program stream by using a program like hdtv2mpeg.

Good luck.

[FreQi]

@Emp3r0r
Thanks for the suggestion, I downloaded the trial and gave it a shot this afternoon. It seems like a really useful and user friendly app for cutting commercials, but for some reason the mpg it generated didn't include any audio. that's not a limitation of the trial version is it? I didn't see anything on the limitations...

I first made an .mpg from the .ts's using HDTV2Mpeg2 v1.11.0 beta (build 037) which plays just fine in InterVideo WinDVD 6, but there's no audio after trimming it with VideoReDo (http://www.drdsystems.com/VideoReDo/).

Any other suggestions?

[FreQi]

I started playing around with ProjectX some more because it's so verbose with it's logging and analysis of the streams, and I discovered something that seems ... unusual. It looks like my FOX affil is flipping between 2ch 192k audio for commercials and 5.1ch 448k audio on the episode.

What ended up happening is the first portion of the recording was a commercial, so DVD2AVI would create the .ac3 with a filename something like "series AC3 T01 2_0ch 192Kbps DELAY -418ms.ac3" which I would open up in VirtualDubMod to do my trims. VDMod, in turn, would open the file, say it is only 192k stereo and processes my trims as such. This causes the cuts to occur much earlier than expected, making the audio completly out of sync and much shorter.

I found that if I set the first file in DVD2AVI to be a small sample of the 5.1ch 448k recording, then add all the file like normal, it makes the file "series AC3 T01 3_2ch 448Kbps DELAY -430ms.ac3". When this file is opened in VDMod, it recognizes it as a 5.1ch 448k ac3 and it looks like my cuts stay in sync!

I don't quite understand why it works, but it does so I can't complain too much. If VDMod is just now assuming the audio clip is 448k instead of 192k, it seems strange that the trim's aren't still messed up. I mean before it cut everything short, it would only seem logical that now it would cut everything long.

[unixfs]

Avidemux can cut commercials and save as mpeg-ps (and other formats).

http://developer.berlios.de/project/showfiles.php?group_id=1402
(link edited: this is the official one)

I didn't try this windows version, but the linux one works well.

[eb]

I started playing around with ProjectX some more because it's so verbose with it's logging and analysis of the streams, and I discovered something that seems ... unusual. It looks like my FOX affil is flipping between 2ch 192k audio for commercials and 5.1ch 448k audio on the episode.
You can use this phenomena to automaticly cut to separate files using PVAStrumento, then use ProjectX to join the files of the show only

eb

[Emp3r0r]

for some reason the mpg it generated didn't include any audio. that's not a limitation of the trial version is it? nope... did the mpg hdtv2mpeg generate have audio?
You can use this phenomena to automaticly cut to separate files using PVAStrumento, then use ProjectX to join the files of the show only This phenomenon is nice indeed. It would be nice if they added detection of this phenomenon to comskip.

For those of you who haven't heard of comskip, it is a command-line tool that can automatically detect commercials in mpg files. It will generate a cutlist for you.

http://www.sage-community.org/index.cgi/wiki/ComSkip

[eb]

PVAStrumento http://www.offeryn.de/pvas_2_1.htm is already cutting the files at the points where audio type is changing.

eb

[link993]

I have used Mpg2cut2 available at http://www.geocities.com/rocketjet4
the major advantage to this program is it is freeware. I find this program extremely fast, also. The disadvantage is that it does not support transport streams, but neither does the $50 video redo, so I'd thought I'd mention this program.

[FreQi]

It seems to me that ProjectX is the true answer for me, at least as far as my recordings from FOX are concerned. On the audio tab in PX, there are a couple options to (1) do CRC checks, (2) fill gaps with previous frames, (3) add frames, (4) replace all non-3/2 with 3/2lfe and to (5) patch the first AC3 header to 3/2ch. I enable all five of those options and the .ac3 it makes behaves just like I want in VirtualDubMod.

@eb
I tried PVAStrumento to see how it dealt with auto splitting at the audio format change, and while it works great, it ends up keeping more than I wanted, like the network feed of the promo's for the night. So I'll still want to manually make the commercial cuts.

@Emp3r0r
Yes, HDTV2Mpeg2 did make an mpg with audio. It must have been something to do with the 2ch/6ch switching that threw VideoReDo for a loop. But with a $50 required price tag, I've already dropped it from my arsenal. It is a great app tho...

@link993 & unixfs
I haven't tried Mpg2cut2 or Avidemux yet, although from their descriptions I'm interested. They sound like they might make for possible alternatives to using VDubMod for cutting commercials, but I'm so comfortable with my current method I'm not sure I'd want to switch it up at this point. Plus the "helper" app I wrote (theeo) depends on the .vcf's that vdmod uses in order to find chapter points.

Now I just need to touch up theeo a bit so it plays nicer with ProjectX. It currently discovers the .ac3 fles by looking for a file named the way DVD2AVI or DGIndex creates, and PX doesn't do that... PX also doesn't take "batch file lists" for it's sources, except fro the one patched by jrmann1999 (http://forum.doom9.org/showthread.php?&threadid=85083), but that's version 0.81.8.01 and 0.81.9.00 is out... fun fun fun.

[hms]

Did you try womble's Mpeg-VCR?
I have no trouble editing HD .ts video streams.
There is a full version download available free for a limited period.
http://www.womble.com

[minolta]

I had sync problems before, but solved it via the following. Since, I've done about 10 episodes of my fav show w/o any issues.

-cut commercials with HdtvToMpeg2 (see doom9 download page for latest version with cutting tools). Output a TS file (great if not re-encoding).

-if re-encoding, sync audio with projectx by demuxing to mpv/ac3. This inserts blank audio into ac3 (usually commercial breaks). Only necessary because dvd2avi (and other programs) ignore blank audio and ignore sync points in TS file.

-load mpv into dvd2avi. encode video to whatever (e.g. xvid). encode ac3 file to whatever (e.g. mp3). mux together...


this method creates a lot of huge temp files but is VERY fast and worry-free. be great if TS files in dgindex (dvd2avi) did not have sync problems, but as you already know...

[eswrite]

I read several comments here regarding VideoReDo's lack of support for Transport streams. Not quite sure I understand this comment: in the trial version I tried last night, I found that it shows .ts in the Open Dialog box, which filters out .tp files (generated with MyHD on my HTPC). When I select all files (*.*), I can click on my .tp files, and VideoReDo opens them right up and processes them (with commercial cuts, etc.) without fuss or muss. My understanding is that these .tp files are HD transport streams, so at least in my case, I'd have to say VideoReDo does support these. Their features list (http://www.drdsystems.com/VideoReDo/FeaturesOverview.htm) says as much.

BTW, I am trying VideoReDo to solve audio/video sync problems when generating DVDs with ULead MovieFactory 4 from MPEGs generated by HDTVtoMPEG2. I left my home PC churning away, so I should have some test results by tonight.

[FreQi]

My understanding is a .tp and a .ts are exactly the same thing. One program names Transport stream a .tp (MyHD) another calls the Transport Stream a .ts (HiPix). Just like a .avs is the same as a .txt really.

Rather than using HDTV2Mpeg2, I'd really suggest you use DGIndex to generate a .d2v and demux the .ac3 file. I'm not familiar with MovieFactory, but I'd expect you should be able to feed it a .avs for the video source and process the audio seperatly.

Maybe I should give you a copy of theeo if you're making dvd's from hdtv. lmk if you're interested in testing it out.

[GaveUp]

You've met the dreaded Fox problem! The problem is the tools that cut AC3 make assumptions that the bitrate will stay the same for all packets, however, fox changes the bitrate from 448 to 192 for commercials. Not to toot my own horn (well ok...*toot*), but in another thread in this forum "AC3 Cutter" I've linked to a utility I wrote specifically to deal with the audio problems on fox. Basically, run the transport stream through dgindex. Then take the audio dgindex spits out at you and run it through my utility. From there you can process normally as the audio will be just like any other AC3, essentially.

The HDTV2MPEG2 option minolta mentions also work but cutting out completely the problem audio frames from the start. The option works great, but in the event that you can't make exact cuts with HDTV2MPEG2 to get rid of all the audio with out cutting into what you want to keep you'd be pretty much stuck.

[FreQi]

For the lazy (myself included), here (http://forum.doom9.org/showthread.php?threadid=92325) is the thread eluded to above.

@GaveUp
That sounds like a very useful tool. It can even read the cuts from a .vcf eh? I like that. I'll have to see if I can use that instead of AC3Fix and BeSplit. I'll try to play with it tonight and let you know what happens.

[GaveUp]

It will be able to do vcf reading, it can't as of yet. Honestly, ac3fix is a really ugly program. All it does is either pad ac3 frames or lop off data if it is too short or long. My program replaces the frame with a silent frame so you don't get the pops and squeels ac3fix gives you.

[FreQi]

I never noticed audible artifacting after running anything through AC3Fix...

I did notice that if I run a file through ac3fix, it'll fix a bunch of frames, then I can run the fixed file through besplit and it'll fix some more. So I don't know what ac3fix is really fixing. It just seems to make the audio stay in sync, were besplit would desync it. I wish I knew more about what each app really did to the audio stream.

Does ac3fix patch in some missing flags like the start and end of ac3 frames? Does BeSplit insert/delete missing bytes on short frames? I'm not sure. What does AC3 Cutter do?

I was looking at the documentation and command line args for AC3 Cutter... can you not specify the frames to cut on yet? I only see options for input and output ac3's. Also, would the frame numbers correspond to the video frames? Or are the AC3 frames different...?

[GaveUp]

Cutting is not implemented yet (see the TODO for what isn't complete). My memory is a bit rusty on exactly what ac3fix did but if I recall correctly it would find a syncword. Then read the samplerate and bitrate (so it knew the size of the frame). Then it would read ahead until it found the next syncword. If it came to soon it would bad the frame, if it came too late it would lop off the end of the frame. Often times it won't produce a noticable audio glitch, but on occasion it will. The reason it helps keep it in sync is that the ac3 frame sizes are now 'proper' lengths. It doesn't mean the data in the frame is valid, beyond the fact it has a syncword and the frame length corresponds correctly to the samplerate/bitrate.

As for the cutting in my program, it's the 2nd thing on my list of things to do with it. The first is to find and fix a known bug in it. Some frames are written slightly improperly (doesn't affect playback) and that's the first thing I want to fix. After that I'll add in the ability to manually speicfy frame ranges to cut/save and once that's working I'll add in support for vcf and avs files.

[FreQi]

@GaveUp

I was just encoding a couple shows here and was wondering if you had a chance to make any progress over the weekend. :)

[GaveUp]

Yes and no. Some bugs fixed, more found. Also encountered some bugs this utility would just never be able to handle so I'm focusing my attention on a different way of doing things. For now, though, I've found a work around that works reliably even on the worst captures. First run it through projectx to fix the corruption and such. Then run it through this utility to patch to the correct bitrate. I have yet to encounter an ac3 that when done this way didn't work with my utility.

[FreQi]

Still planning on being able to cut the ac3 tho? I currently use VDubMod 1.4.13 to cut and if it existed, it would be nice to be able to do that using a command line tool that could do the bitrate patching you're talking about all in one shot. I use DGIndex to make my source ac3 and the d2v. I'm not real keen on PX just yet... Maybe I need to become more familiar with it.

[GaveUp]

Yes, but the cutting is a back burner seeing as there is a way to do it already, it's just not as convenient.