Filter to remove non-letterbox frames?

[skinny]

I'm considering writing a filter to remove frames that are not letterboxed (no black bands on top and bottom of frame). The intent is to remove commercials from recorded TV shows (e.g. Enterprise).

I have read "Writing Avisynth plugins" and have a basic understanding of the process but I'm looking for insight into this particular problem.

How do you programmatically determine if a pixel or set of pixels is "black"? Is there a reasonable algorithm for color detection?

I would think that this mechanism would require frame buffering to allow for occasional black spots in the non-letterbox frames. Is there example plugin code that does similar buffering?

Is this a reasonable filter to write or is there some other method (using Avisynth) to do that same thing?

Thanks.

[Guest]

Sure it's reasonable and it should be a very simple filter to write.

[wmansir]

You will also have to account for the logo, and those annoying promo popups. A false positive would really mess up an encode.

[Krajensky]

The easiest way to get an idea of how to write plugins is to take an existing plugin and walk through it in your head. The way YUY2 data is stored is (for two pixels) [luma-1] [chroma-u] [luma-2] [chroma-v]. You can test against the luma values without considering the color (if the capture is analog, the chroma will contain junk when the luma is very nearly black).

In Avisynth, you don't really "buffer" frames - you pretest against them. Basically once the plugin is initialized, its function GetFrame(x) is called. This function can then request any frames it needs (calling parent.GetFrame(x), I can't remember what that object is called atm) to work with (x, x+1, x+50...anything you want). You can save your choices either to a global variable within your plugin class, or to some sort of file (but that's adding to the complexity).

The function then returns a frame...either one of the buffers acquired by parent.GetFrame(x), or an output framebuffer created in the function. I have no idea how to drop a block of frames in Avisynth though, but surely it is in the Decomb code.

One surefire method for finding commercials would be test for perfectly black frames with very quiet audio preceeded by louder audio that are followed by a non-letterbox scene...it always does that audio thing when it goes to commercial (and my station shows on average, 35 black frames before the commercial starts).

If you need any help from another plugin newbie, feel free to ask.

Krajensky

[SILICON]

Originally posted by skinny
I'm considering writing a filter to remove frames that are not letterboxed (no black bands on top and bottom of frame). The intent is to remove commercials from recorded TV shows

It´s a wonderfull idea.

How do you programmatically determine if a pixel or set of pixels is "black"? Is there a reasonable algorithm for color detection?

1- The color of one pixel in variable in VCR and TV. It´s best look only for luma component (Y). It is stable much.

2- The value of a black pixel usually is 16, not zero

3- it watches single in tercion midle of the line. On this way you avoid most of the noise and logos.

4- Check only the top of film. On this way don´t mistake with subtitels.

5 - Don´t check the very top lines. You can see the teletext info or all time black lines.

I think: Add all pixel values in tercion midle of the line, check if this value is hihq than treasure. If this high, you have a 4/3 frame. In example:

// Resolution: 720x480
int wide = 730;
int height = 480;
int speed = 1; //higest = more speed but less precise
threshold = 18; //the suitable value, approximately

int count;
int sum_value1=0;
int sum_value2=0;
int sum_value3=0;
int sum_value4=0;
int sum_value5=0;
int letterboxed=0;
int Pixels_Checked=0;

// only check the midle pixels of the line.
// more speed, less compared pixels
for(count=height/3; count<height*2/3; count+=speed)
{
sumvalue1 += LumaPixelValue(10, count); //check the 10 line
sumvalue2 += LumaPixelValue(11, count); //check the 11 line
sumvalue3 += LumaPixelValue(12, count); //check the 12 line
sumvalue4 += LumaPixelValue(13, count); //check the 13 line
sumvalue5 += LumaPixelValue(14, count); //check the 14 line
Pixels_Checked++;
}

// compare the very high lines
if (sumvalue1/Pixels_Checked) < threshold) letterboxed+=1;
if (sumvalue2/Pixels_Checked) < threshold) letterboxed+=1;
if (sumvalue3/Pixels_Checked) < threshold) letterboxed+=1;
if (sumvalue4/Pixels_Checked) < threshold) letterboxed+=1;
if (sumvalue5/Pixels_Checked) < threshold) letterboxed+=1;

if (letterboxed > 3) return(0); // 3 lines are black. Is leterboxed
return(1); // less of 3 lines are black. It's 4/3 o very noise film.

Sorry, but I don´t know how coded the LumaPixelValue funtion.

I hope that it serves to you as aid. :-)

[sh0dan]

Doing what you suggest is not easy, and even if you succed you'll probably get many 'false hits'. Also you cannot modify the length of the clip once the filter has been instantiated.

It would probably be easier just to use trim, and find the values using Vdub.
A script to make it easier could be like:

video=avisource("file")
video = removecommercial(video, 90000, 1000) # Removes video from 90000 to 91000
video = removecommercial(video, 80000, 1000) # Removes video from 80000 to 81000
return video

function removecommercial(clip c, int startframe, int endframe) {
start = Trim(c,0,startframe)
end = Trim(c,endframe,0)
return start+end
}

Beware - it's not tested, but it should work.

[edit] Forgot to add. Remove the sections BACKWARDS (last section first), otherwise your frames will get wrong after the first trim.

[SILICON]

Originally posted by sh0dan
Doing what you suggest is not easy, and even if you succed you'll probably get many 'false hits'. Also you cannot modify the length of the clip once the filter has been instantiated.

sh0dan, it´s true.

This can't be a plugin. Can be a program thar read avisynth and make this. The program can make the TRM scrips)

You will get some 'false hits' when a comercial have all black frame. But is easy skip the false hits. All commercial are contigous (and very long:-) ) You can ignore the 'false comercial frames' and 'false film frames' if are less of one second (30 frames).

I think that this metod will be stone solid.

[ronnylov]

Another way to detect the commercials that would work here in Sweden could be a "no logo detector". Because everytime they show the commercials they remove the channel logo in the upper right corner.

So if the filter could remove all frames without the logo the commercials would have been removed. But it sounds complicated to me. It's easier to just use trim as shodan suggested.

[wmansir]

@sh0dan

video = removecommercial(video, 90000, 1000) # Removes video from 90000 to 91000
did you mean "removecommercials (video, 90000, 91000)"? I think that's what your function requires, and based on your varible name (EndFrame, not something like Length). Anyway, requiring the length as an argument would add more work, because you would need to calculate it instead of just reading the endframe from Vdub.

[Guest]

Originally posted by SILICON
sh0dan, it´s true.

This can't be a plugin.You're both wrong. You are failing to think creatively. :)

[sh0dan]

Originally posted by wmansir
@sh0dan
Did you mean "removecommercials (video, 90000, 91000)"? I think that's what your function requires, and based on your varible name (EndFrame, not something like Length). Anyway, requiring the length as an argument would add more work, because you would need to calculate it instead of just reading the endframe from Vdub.

Yes - I intended it to be 91000 - sorry for the confusion!

[sh0dan]

Originally posted by neuron2
You're both wrong. You are failing to think creatively. :)

Please enlighten me :)

I agree, I haven't given it much thought, but now I'm thinking if it.... :confused: ... nope .. nothing! :rolleyes:

I hope you're not thinking of Twopass/logfile?

[int 21h]

Originally posted by sh0dan
Doing what you suggest is not easy, and even if you succed you'll probably get many 'false hits'. Also you cannot modify the length of the clip once the filter has been instantiated.


If this were true, it would make Decimate() pretty useless :(

So anyways, all you do is make one filter called DetectCommercials() run it and dump all the frame numbers that are commercials into an array of some sort, then make a filter called RemoveCommercials() that removes those frames.

P.S. - You can look towards Decomb.dll for an example of how to do this because it implements some of the same sort of ideas, afaik.

http://shelob.mordor.net/dgraft/decomb391src.zip

[sh0dan]

As far as I can tell from decomb, it already knows how many frames it will end up with, before it starts with the first frame.

Decimate::Decimate(...) Adjusts the number of frames. GetFrame doesn't change that.

[edit] smilies
[edit2] - just think about it - how would the dropped frames look in vdub - black frames at the end?

[Suzahara]

This type of filter would be very hard/impossible to make. First of all, it's hard to eliminate commercials by just letterboxing. Some commercials have letterboxing and some have black at the bottom, and you'd have to judge it just right for it to detect each frame correctly. And then it also has the UPN logo problem since most logos go across onto the black border. I would be amazed to see this filter written. And it's uses would be limited as not all that many shows are in letterbox yet.

[int 21h]

Originally posted by Suzahara
This type of filter would be very hard/impossible to make. First of all, it's hard to eliminate commercials by just letterboxing. Some commercials have letterboxing and some have black at the bottom, and you'd have to judge it just right for it to detect each frame correctly. And then it also has the UPN logo problem since most logos go across onto the black border. I would be amazed to see this filter written. And it's uses would be limited as not all that many shows are in letterbox yet.

This filter would obviously be useful for its author, and with all due respect, he didn't ask you how useful it would be, he asked if it would be possible to implement and for advice on how to implement it.

Implementing this filter is very possible with the specifications he supplied. Instead of discouraging him, we should all be encouraging him. He's already 5 steps ahead when his post didn't start out with: "Could somebody write this for me?"

[int 21h]

Originally posted by sh0dan
As far as I can tell from decomb, it already knows how many frames it will end up with, before it starts with the first frame.

Decimate::Decimate(...) Adjusts the number of frames. GetFrame doesn't change that.

[edit] smilies
[edit2] - just think about it - how would the dropped frames look in vdub - black frames at the end?

You aren't considering my example. RemoveCommercials() would know exactly how many frames to adjust and it would know exactly how many frames it would end up with, thanks to DetectCommercials().

[Kurosu]

My ideas:
- Checking for letter box should check pixel on corners or, more precisely, on lines/columns starting from the corners like:
+--·········--+
|···············|
·················
·················
|···············|
+--·········--+
(ASCII art anyone :D This "one" was a pain)
Then check if luma < 16 and if there are enough pixels to consider the frame letterboxed
- it would be a lot easier to force the user to a linear edit
I even guess that an none linear edit would crash the viewing application. An idea would be to check if previous requested frame = current frame - 1, and if not, disable the "filtering"
- then, how it would proceed:
during "instantiation", set count to 0;
in function GetFrame:

RemCom()
return tempframe;

in fucntion RemCom:

tempframe = child->GetFrame(n+count, env);
//Verify in tempframe if letterboxed
if (LetterBoxed)
{
count++;
RemCom();
}
return

- I don't check here possible infinite loop and the so...

Maybe I could have written it myself but... err... I have lots of avisynth dll projects that are plebiscited. Yeah, let's say it that way :rolleyes:

[EDIT] Even more with all those edits :D

Anyway, that filter has another problem other that the frame length of the video returned: modyfying its time length, you have to also remove audio... I haven't even have a look at that problem...

[int 21h]

Originally posted by Kurosu
Anyway, that filter has another problem other that the frame length of the video returned: modyfying its time length, you have to also remove audio... I haven't even have a look at that problem...

After the DetectCommercials() function you could dump a log file of where commercials were detected so that you could cut the audio manually in a WAV (WavLab, CoolEdit, etc) program (starting from the end of course).

Example:

28:55.212 - 25:21.216
15:34.100 - 10:43.021

etc.

[Didée]

Just a few informations how commercials are done here in Germany:

- no black frames at all between the show and commercial break! They do a hard cut all the time, perhaps a blend of 1-2 frames ...

- more an more, they come to take away the logo some seconds before the show ends, and display the logo quite some seconds after the show has re-started!
-> impossible to rely on logo detection alone.

Just FYI

Didée

[wmansir]

I know there are some Tivo-like devices which detect and automaticlly skip commercials. I'm sure a little investigation into how they detect commercials would be worth it. I heard one way was to look for a group of black frames and then check exactly 30 and 60 seconds later for another group of black frames.

I wouldn't be suprised if they also check the audio, commercial audio is usually highly compressed. I don't know enough about teletext, but perhaps there is something in that info that could signal a commercial.

[Kurosu]

What I've seen so far concerning commercials:
- some blue pictures are sometimes broadcasted before teh commercial
- the sound volume suddenly jumps
- more important, and it seems it's used by systems automatically starting recording: a sort of jingle tells the tape recorder that the movie starts/ends. More exactly a sharp sound in high frequencies...

Anyway, i think a trim() is much more efficient :)

[skinny]

I would like to thank everyone for there input. From what I can tell from the comments this filter would be easy to impossible to write. So, I imagine there is some ground in between where I could get something to work.

I will be spending the early part of this week putting together a new machine that I am going to be using primarily for video editing. After I've got it up and running I will look into writing the filter to remove non-letterbox frames or a filter to just detect-non-letterbox.

Thanks for the different suggestions of algorthims for letterbox detection. Right now I'm going to concentrate on the video frame method (absence of black on the top portion of the frame).

I'm sure I'll have further questions. I'll keep you posted.

[Belgabor]

As much as I would like such a filter, I think its impossible to do it if you mean to save time. I think you would need more time to tweak the filter and check for detection errors than you would usually need to skip through the movie to find the commercials and remove them by Trim.

Just my 5 euro cents
Belgabor