Commercial Removal Software

[KG6ZVP]

Hi all,

I sat down yesterday and wrote out a rough program for doing commercial removal. I have a working binary of it, but I thought I'd just post the source (the source is newer than the binary).

A little bit about this program: It doesn't remove any commercials yet. As of the present, all it does is take a file with a list of black frames retrieved via avisynth script (also below) and generate a new avs with different sections of video labeled. I will continue working on this to come up with a complete solution (ie: I'd like to add in the ffmpeg libraries so that it could work directly on a broadcast video recording and generate the list of black frames itself as well as use the audio to help with the analysis)

Anyway, start by placing whatever file you want to analyze in this format:

LoadPlugin("GScript.dll")
Video=mpeg2source("video.d2v", info=3)
Audio=NicAC3Source("audio 3_2ch 384Kbps DELAY -541ms.ac3")
film = AudioDub(Video,Audio).DelayAudio(-.541)
film

filename = "Black_frames.txt"
current_frame=0
last_black=0
GScript("
	while(current_frame < framecount+1){
		if(AverageLuma(film) < 17){
			WriteFileIf(film, filename, "AverageLuma(film) < 17 && last_black<(current_frame-1)", "current_frame")
			last_black = current_frame
		}
	}
")

Open that video and let it play and it will generate a file with a list of black frames in it.

Format a script to "edit" into sections. Use the example below or write your own.
Example:

Video=mpeg2source("video.d2v", info=3)
Audio=NicAC3Source("audio 3_2ch 384Kbps DELAY -541ms.ac3")
film = AudioDub(Video,Audio).DelayAudio(-.541)

Then, use my program to write a script (note: at this stage, my program assumes that whatever avisynth script is being edited calls it's output by the name "film.")

Currently the program can be run in a few ways:
Usage:
commercialremover [input list of frames] [avs script to add to] [append pattern part 1] [append pattern part 2]
or
Usage:
commercialremover [input list of frames]

or
Usage:
commercialremover [input list of frames] [avs script to add to]
or
Usage:
commercialremover [input list of frames] [avs script to add to] A
note:^ the A means to append a subtitle to each section of video displaying the section number

Example:
commercialremover.exe Black_frames.txt t
est.avs A

The above command produced this file:

Video=mpeg2source("video.d2v", info=3)
Audio=NicAC3Source("audio 3_2ch 384Kbps DELAY -541ms.ac3")
film = AudioDub(Video,Audio).DelayAudio(-.541)

film.trim(0,94).subtitle("Section 0",align=5,size=256)
+ film.trim(99,603).subtitle("Section 1",align=5,size=256)
+ film.trim(604,717).subtitle("Section 2",align=5,size=256)
+ film.trim(722,1278).subtitle("Section 3",align=5,size=256)
+ film.trim(1278,1507).subtitle("Section 4",align=5,size=256)
+ film.trim(1533,11363).subtitle("Section 5",align=5,size=256)

I will edit this post and add my working binary as soon as it is uploaded.

Please comment and bring up some different usage scenarios. I would like to eventually add keyframe and audio silence detection so that it could detect scene changes as well as just commercial breaks.

By the way, I compiled this binary on an arch linux x64 dev server using this command line:
i486-mingw32-g++ -static-libgcc -static-libstdc++ main.cpp -o commercialremover.exe

Here is the source code as it stands now (binary was compiled yesterday so they are slightly divergent)

/* 
 * File:   main.cpp
 * Author: sam (kg6zvp@gmail)
 *
 * Created on 31. Oktober 2013, 04:29
 */

#include <cstdlib>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <string>

using namespace std;

class Section{
public:
	int a;
	int z;
	bool isContent;
	Section(int beginning, int end, bool isContent){
		a = beginning;
		z = end;
		this->isContent = isContent;
	}
	operator char*(){
		char* output = new char[30];
		sprintf(output,"film.trim(%d,%d)",a,z);
		return output;
	}
	vector<int>* getPreviewFrames(){
		int length = z-a;
		vector<int>* output = new vector<int>();
		//Switch method
		/*switch(length){
			
		}*/
		//average method
		int segment = length / 5;
		int remainder = length % 5;
		int segments[5]; //set up array of segments
		segments[0] = 0;
		for(int i=1; i < 4; i++){//populate array
			segments[i] = segments[i-1]+segment;
		}
		segments[4] = segments[3]+remainder;
		output->push_back( a+segments[0] );
		//Iterate through array and don't add duplicates
		for(int i=1; i < 4; i++){
			if(segments[i-1]!=segments[i]){
				output->push_back( a+segments[i] );
			}
		}
		return output;
	}
};

bool useSTDout;
bool append;
bool preview;
bool middleFrame;
string appendA, appendB;
/*
 * 
 */
int main(int argc, char* argv[]){
	switch(argc){
	case 1:
		cerr << "please pass the file to read from as an argument!\n";
		cerr << "Usage:\n\tcommercialremover [input list of frames] [avs script to add to] [append pattern part 1] [append pattern part 2]\n";
		cerr << "\t\tor\nUsage:\n\tcommercialremover [input list of frames]";
		cerr << "\t\tor\nUsage:\n\tcommercialremover [input list of frames] [avs script to add to]";
		cerr << "\t\tor\nUsage:\n\tcommercialremover [input list of frames] [avs script to add to] A\nnote:^^the A means to append a subtitle to each section of video displaying the section number";
		return(1);
		break;
	case 2:
		useSTDout = true;
		break;
	case 3:
		useSTDout = false;
		cout << "Outputting to: " << argv[2] << "\n";
		break;
	case 4:
		if(argv[3][0]=='A' && argv[3][1] ==0){
			append=true;
			preview=false;
			middleFrame=false;
			appendA.assign(".subtitle(\"Section ");
			appendB.assign("\",align=5,size=256)");
		}else if(argv[3][0]=='P' && argv[3][1] ==0){
			append=false;
			preview=true;
			middleFrame=false;
		}else if(argv[3][0]=='M' && argv[3][1] ==0){
			append=false;
			preview=false;
			middleFrame=true;
		}
		break;
	case 5:
		appendA.assign(argv[4]);
		appendB.assign(argv[5]);
		cout << "Will append " << appendA << "[INT]" << appendB << "\n";
		break;
	default:
		cerr << "Too many argumeuts!\n";
		return(1);
		break;
	}
	fstream input;
	input.open(argv[1],fstream::in); //open input file for reading
	if(!input.is_open()){ //if input file couldn't be opened, print error and exit
		cerr << "Error opening file: " << argv[1] << "\n";
		return(1);
	}
	/******************************
	 * Read all integers into map *
         ******************************/
	vector<int> allIntegers;
	int tInt = 0; //temporary integer variable for storing sscanf output before writing to vector
	int i=0;
	string line;
	while(getline(input,line)){
		tInt = -1;
		sscanf(line.c_str(),"%d",&tInt);
		if(tInt != -1){ //if sscanf got useful input
			allIntegers.push_back(tInt);
			//allIntegers[i] = tInt;
			i++;
		}
	}
	/**************************
	 *  Process all integers  *
	 **************************/
	int sectionBegin=0;
	int sectionEnd=0;
	int prevBlackFrame = 0; //the integer of the last black frame (ie: not the end frame, but the previous)
	int currBlackFrame = 0;
	int nextBlackFrame = 0; //the integer of the next black frame
	vector<Section> sections;
	int sectionIndex = 0; //this keeps track of how many sections have been added to the map
	for(vector<int>::iterator it = allIntegers.begin(); it != allIntegers.end(); ++it){
		if(it == allIntegers.begin()){ //if this is the first black frame in the video
			if( (*it) > 0 ){ //if the first black frame isn't the first frame of the video
				sectionEnd = (*it)-1; //end the first section at the end of the last colorful frame
				sections.push_back(Section(sectionBegin, sectionEnd, true));
				sectionBegin = (*it); //mark the next section's beginning (a section with black frames)
			}
		}else{ //if this isn't the first black frame in the video
			currBlackFrame = *it;
			nextBlackFrame = *(it+1);
			if ( prevBlackFrame!=(currBlackFrame-1) && (currBlackFrame+1)==nextBlackFrame) { //if the black frame is at the beginning of a series of black frames
				sectionEnd = currBlackFrame - 1; //set the end of the last section containing content
				sections.push_back(Section(sectionBegin, sectionEnd, true)); //pass in true, because this section contains content
				sectionBegin = currBlackFrame; //set the beginning of the next section (containing all black frames) to the current black frame
			} else if ( prevBlackFrame==(currBlackFrame-1) && (currBlackFrame+1)!=nextBlackFrame ) { //if the black frame is at the end of a series of black frames
				sectionEnd = currBlackFrame; //set the sectionEnd equal to the current frame
				sections.push_back(Section(sectionBegin, sectionEnd, false)); //pass in false, because this section contains black frames
				sectionBegin = currBlackFrame + 1; //set the begining of the next section equal to the next framee (which contains content)
			} else if ( prevBlackFrame!=(currBlackFrame-1) && (currBlackFrame+1)!=nextBlackFrame ) { //if the black frame is alone between colored frames
				sectionEnd = currBlackFrame - 1; //set the end of the last section equal to the last frame with content
				sections.push_back(Section(sectionBegin, sectionEnd, true)); //pass in true, because this section contains content
				sectionBegin = currBlackFrame; //set the beginning of the next section (containing all black frames) to the current black frame
				sectionEnd = currBlackFrame; //set the end of this section to the same frame (that way any audio in that section can be added to or from the correct place)
				sections.push_back(Section(sectionBegin, sectionEnd, false)); //pass in false, because this section does not contain content
			}
		}
		prevBlackFrame = *it;
	}
	
	int itIndex=0;
	cout << "Writing output file: " << argv[2] << "\n";
	ofstream output;
	output.open(argv[2], ofstream::app);
	output.write("\n\n",6);
	if(preview){
		/***********************
		 *  Generate  Preview  *
		 *  and write to file  *
		 ***********************/
		string s;
		s.assign("null!");
		string itIndexS;
		itIndexS.assign("null!");
		vector<int>* previewFrames;
		int currentFrame=0;
		int numToWrite=0;
		for(vector<Section>::iterator it = sections.begin(); it !=sections.end(); ++it){
			if(it != sections.begin())
				output.write("+ ", 2);
			previewFrames = it->getPreviewFrames();
			for(vector<int>::iterator frame = previewFrames->begin(); frame != previewFrames->end(); ++frame){//loop through frames and add them
				if(frame != previewFrames->begin())
					output.write("+ ", 2);
				currentFrame = (*frame);
				char* buffer = new char[600];
				//printf("film.trim(%d , -1).subtitle(\"Section %d (%d)\", align=5, size=256)", currentFrame, itIndex, currentFrame);
				numToWrite = sprintf(buffer, "film.trim(%d , -1).subtitle(\"Section %d(%d)\", align=5, size=128)", currentFrame, itIndex, currentFrame);
				output.write(buffer, numToWrite);
			}
			
			output.write(" ", 1);
			itIndex++;
		}
		return 0;
	}else if(middleFrame){
		/***********************
		 *  Generate  Preview  *
		 *  and write to file  *
		 ***********************/
		string s;
		s.assign("null!");
		string itIndexS;
		itIndexS.assign("null!");
		vector<int>* previewFrames;
		int currentFrame=0;
		int numToWrite=0;
		for(vector<Section>::iterator it = sections.begin(); it !=sections.end(); ++it){
			if(it != sections.begin())
				output.write("+ ", 2);
			currentFrame = (*(it->getPreviewFrames()->begin()));
			char* buffer = new char[600];
			//printf("film.trim(%d , -1).subtitle(\"Section %d (%d)\", align=5, size=256)", currentFrame, itIndex, currentFrame);
			numToWrite = sprintf(buffer, "film.trim(%d , -1).subtitle(\"Section %d(%d)\", align=5, size=128)", currentFrame, itIndex, currentFrame);
			output.write(buffer, numToWrite);
			}
			
			output.write(" ", 1);
			itIndex++;
		}
		return 0;
	}
	
	/*******************************
	 *  Print all sections to ???  *
	 *******************************/
	if(useSTDout){
		cout << "\n\n";
		for(vector<Section>::iterator it = sections.begin(); it !=sections.end(); it++){
			if(it != sections.begin())
				cout << "+ ";
			cout << (char*)(*it);
			if(append){
				cout << appendA.c_str() << itIndex << appendB.c_str();
				if( (*it).isContent ){
					cout << "/*content*/";
				}else{
					cout << "/*BLACK*/";
				}
			}
			cout << " ";
			itIndex++;
		}
	}else{
		string s;
		s.assign("null!");
		string itIndexS;
		itIndexS.assign("null!");
		for(vector<Section>::iterator it = sections.begin(); it !=sections.end(); ++it){
			s.assign((char*)(*it));
			if(it != sections.begin())
				output.write("+ ", 2);
			output.write(s.c_str(),s.size());
			if(append){
				output.write(appendA.c_str(),appendA.size());
				stringstream ss;
				ss << itIndex;
				itIndexS.assign(ss.str());
				output.write(itIndexS.c_str(),itIndexS.size());
				output.write(appendB.c_str(),appendB.size());
			}
			output.write(" ", 1);
			itIndex++;
		}
	}
	return 0;
}

[turbojet]

It's a very ambitious project and very difficult to achieve based on comskip's years of development and still does bad detects often. Some common scenarios where the current and planned features won't detect correctly:
1. No black frames between ads and show
2. Black frames in between ads
3. No audio silence between ads and show
4. Audio silence in between ads

Keyframe detection could possibly resolve issues 1 and 3 but not sure how 2 and 4 would be handled. Logo detection is another common way but there's issues with that too, for example: show segment starts with random black frames between scenes or artistic effect before the logo shows up, which in some cases can be 15-20 seconds. Another issue with auto detectors is they take a long time to process.

Not to try to shift you from your current work but if you are up to it, I do think VideoRedo is begging for competition. There's other cutters like avidemux but nothing else is frame accurate. A tool that's frame accurate (requires short encodes) that has options to demux audio and convert cc to srt could grab a lot of videoredo users' attentions. No need to have all the encoding output settings like videoredo, that devs spend so much time working on.

[KG6ZVP]

I agree with you on all of those points, turbojet, and I'm glad I'm not the only one who thought of keyframe detection. The idea of this project is to generate a series of output files based on the detected scene changes so that people can join the files end to end afterwards or delete some of the files and just make a TV episode output to it's own folder with 001.mkv, 002.mkv, etc. so they can at least get rid of most of the commercials.

At the moment, I am really just cutting my teeth on this since it is something I wanted anyway (there are too many useful windows-only tools right now like MeGUI's avs cutter, etc.)

If you have any tips for me, I would really appreciate it as I am currently trying to understand the libav documentation well enough to eliminate avisynth from the picture for detecting the black frames.