Neural Network Processing

[vcmohan]

I have serached the forum for Neural network processing but could not find any. I thought atleast the forum maintained by neuron2(name suggests it must have) may have some but am disappointed.
I do not know how far neural network processing will be practical in video image processing, but I hope it will be. I therefore did some preliminary work and could get some results which I present here.
My input for this is a greyscale image which I used in the thread "recoloring".
http://img161.imageshack.us/img161/9071/birthday10hi.jpg (http://imageshack.us)
I first did an edge detection with a plugin of mine and its output is below:

http://img396.imageshack.us/img396/5229/edgeorig0000016cp.jpg (http://imageshack.us)
A small window of 100X150 located in lower middle of this edge map is used for training. The window used is below:

http://img356.imageshack.us/img356/9391/edgewin0000018hb.jpg (http://imageshack.us)
I iterated 1600 times for optimising layer weights using steepest descent method. Unlike my edge map which has values of either 0 or 255, the output of neural processing I found were in a range of 52 to 77 percent ( values of 132 to 196) but with a noticeable steep step across 66 to 67% (or around a value of 169).
Output image (neural processing done on full frame) with 168 as threshold is below:
http://img356.imageshack.us/img356/8510/edge0000663be.jpg (http://imageshack.us)
Output image with 170 as threshold is below:
http://img356.imageshack.us/img356/1548/edge0000679sy.jpg (http://imageshack.us)
While ideally a noise free image should be used for training my input is with noise. Possibly I should use a cartoon image for training. Or the part with balloons may have less noise.

Other details: Input edge detection was with a 3X3 grid. Neural network input was also 3X3 grid. Hidden layer has 9 nodes and output layer 3 nodes.

I would appreciate suggestions on any further work, input models etc; . My plugin is in initial stages and need to do some more work.
Currently my function handles one plane only. So color can not be handled at this stage. The range of weights to be used for hidden and output layers as well as number of iterations can be specified but I would like to do some automation in this area. While the improvement with each iteration can be gauged with an error parameter, with Avisynth I can not get this value interactively.

[Guest]

Neural networks come into their own when there are actually instantiated units to permit massively parallel processing. Simulating that serially is pointless. The algorithms just implement known things like gradient descent, etc. There's nothing magical about a "neural network algorithm".

[mg262]

Perhaps it's worth trying some other edge marks for comparison?

There's a whole bundle in a Photoshop plug-in here:
http://members.ozemail.com.au/~binaryfx/PSTV_convolcorner.html

The one I prefer for most purposes is Prewitt, which I made an YV12 AVISynth version of:
http://people.pwf.cam.ac.uk/mg262/posts/Prewitt_08Aug05.dll

Prewitt(bool thin = false)

The brightness of output will correspond to the strength of an edge. It will all be relatively dark, so you might want to follow it up with a Levels call (with coring off).

Have a look through and see what you think... .

[vcmohan]

What I am looking for is not an edge detection algorithm. There are many. I am exploring Neural networks and just took in edge detection as an excercise. There may be other applications which I want to explore. It may be wasteful of time, but I have some time with me. I want suggestions as to which other applications I can try.

[Guest]

Put "neural network applications" into Google.

[Zarxrax]

Would neural networks be useful for noise removal?

[Guest]

Would neural networks be useful for noise removal? I suppose you didn't get my point in my first post. A "neural network algorithm" is one designed to run on a network of parallel units called neurons. If you don't actually *have* a network of parallel units, but instead simulate it serially, as you would on a PC, you are doing a pointless thing, unless you are just trying to *develop* parallel neural algorithms. The algorithms designed and optimized for serial processors will generally perform better than those designed for parallel architectures but then simulated serially. As I said, there is nothing magical about a "neural net algorithm". It's just an algorithm designed and optimized to run on a parallel network of "brain-like" units called neurons. I hope that is clear now.

Now if you are asking can some algorithm like simulated annealing, which is typically targeted at parallel architectures, be useful for video processing, that is a more interesting question. I haven't seen such processing of video on PCs, presumably because it is dreadfully slow.

Of course, no one has yet figured out how to implement human intelligence on a network of simulated neurons. Even if that were achieved, running it on a serial processor would probably be way too slow.

[krieger2005]

lik 1 (http://www.it.lth.se/users/lambert/imaging%5CKESS03.pdf)

I would say that NN are good when it is difficult to set a right threshold, for example segmentation, scene-change-detection, deflicker... But in general for all those things for which there is no good algorithm available (for scene-change-detection there are some).

I searched a littlebit though the net and searched for "image processing" "neural net". Most of the stuff ist scientific or a link to a book.

[vcmohan]

If you don't actually *have* a network of parallel units, but instead simulate it serially, as you would on a PC, you are doing a pointless thing, unless you are just trying to *develop* parallel neural algorithms.
I am not too sure about this. ANN processing is done extensively in share market predictions using serial input. For instance in the 3x3 grid used in edge detection is input as 9 nodes in ANN and works same except the weights are determined in a different fashion and some times more aptly.


I haven't seen such processing of video on PCs, presumably because it is dreadfully slow.........
Even if that were achieved, running it on a serial processor would probably be way too slow.

The edge detection training took a couple of minutes but actual processing was as fast as the usual edge detection algorithms.

As for Zarxarax,s suggestion of noise elimination, I have been thinking using my Fan Filter as a training material. Right now I am not sure as to how many output nodes I need. I am also to understand in more detail the requirements of number of nodes, their starting weights, the number of layers etc.

I am aware that all my efforts may not yield any practical or usable process and has to be discarded. However at this stage I am not put off by those considerations. On the otherhand I am optimistic about the outcome.

A google search for ANN applications indicates mostly stock market predictions, and some medical image Analysis. I am sure some part of this analysis will be applicable to video images also. May be similar to stock prediction, motion can be predicted?

[MfA]

You could train a NN for determining image dependent filters (ie train a NN to provide weights for a linear filter depending on the input).

Doesnt make much sense to train a NN with input/output of an existing filter though, unless the filter is very slow and a NN could approximate it faster. VQ is actually better for this though.

Usually you train with known correct reference images.

[vcmohan]

Doesnt make much sense to train a NN with input/output of an existing filter though, unless the filter is very slow and a NN could approximate it faster. VQ is actually better for this though.Usually you train with known correct reference images.
Yes. However in absence of such images with me I used edge detection just to check that my code is working correctly. I need to use noise free image as reference output and add synthetic noise to simulate noisy image probably. But that as I understand just makes NN solution for that particular type of noise only.

Being not a proffessional I did not understand what a VQ is.

[Guest]

Being not a professional I did not understand what a VQ is. http://www.geocities.com/mohamedqasem/vectorquantization/vq.html

[vcmohan]

Thanks for the reference. I had a quick glimpse at VQ page. It certainly put me off with all those mathematical symbols. However I felt that the regions and codewords appeared to be close to what one gets in Watershed segmentation. That may have an application in compression. But right now I am concentrating on Neural Networks.

[vcmohan]

I have posted the plugin NeuralNet (http://www.avisynth.org/warpenterprises/#NeuralNet) and its documentation. This is classic 3 layer categorisation implementation.I have tested it with an edge detection task and could get very encouraging and satisfying results. However as commented by Neuron above its application for video image processing/editing is difficult to imagine.

I am now contemplating to expand the network to a regression type, provided I understand the literature on that aspect.

I will appreciate very much any cooments, bugs or suggestions on this.

[Guest]

Source code?

[vcmohan]

Source code?
Yes. That will follow soon, after I put in some more comments etc. Meanwhile if I can get some usage problems, I may need to modify the source.

[vcmohan]

A linear filter function NeuralNetLN is now added to NeuralNet plugin. This function can process images where a linear relation from input to output exists.
http://img498.imageshack.us/img498/7081/neuralfan10hl.jpg (http://imageshack.us)
Center is input, on right is processed by VFan and a small part of it used for training and on left is processed by NeuralNetLN. The images at this scale appear identical but at full scale there were differences. Most noticeable was in the rectangle inside the inverted triangle to the right of the subjects head. Some faint details seen in VFan filtered image were not seen in NeuralNetLN processed image.
I have also tested with GaussBlur (part of ColorIt plugin) with good result.
The script used is

f=imagereader("D:\TransPlugins\images\msintrf0.jpg",0,1,25,false).converttoyuy2()
i=imagereader("D:\TransPlugins\images\fanmsint0.jpeg",0,1,25,false).converttoyuy2()

ln=NeuralNetLN(f,i,tlx=250,trx=380,tty=0,tby=400,xpts=9,ypts=1,iter=400,wh=0.01)
stackhorizontal(ln,f,i)
reduceby2()


I find that the NeuralNet provides good results if the starting weights are small and near the expected values. Using High values as starting weights not only were noisy but also the image y values were exceeding 0 to 255 and needed clipping, and in some cases were getting washed out appearance.

I experimented trying to deblur by inputting blurred image and using original as training material. For small starting weights and grid size it appeared to work but in general not. There is no linear relation for the reverse and so may not work.

Zarxrax has asked whether Neural Networks will be useful for noise filtering. Blur can be done but Blur directly is far more faster. Many noise reduction schemes are not entirely linear and so will be difficult.