colorization code used as denoiser?

[Hero--]

can the method used here be used to 'normalize' larger areas
of 'noisy' color section be used to reduce noise by enforcing
uniform color areas?

http://www.cs.huji.ac.il/~yweiss/Colorization/


(also isn't it time there was a gui interface (with selecting masks etc) for avisynth? ie when will there be a 'mother' application where you can mark/select areas, easily split screen to view effects of different filters etc)

thanks,


/H

[E-Male]

interesting idea

also this colorizing algorithm alone looks pretty nice

i daubt i can port it, because i never used matlab
but i'll have a look


EDIT:
i had a look at it and daubt it'll be much use as a chroma-denoise
but CNRF uses a similar idea and should be the way to go in avisynth

but it would be really cool if this paper/matlab-code could be ported to avisynth

[HighInBC]

I would love to have an AVISynth filter that can do this! The 'recoloring' especially...

I am already pictureing strange alien landscapes!

[E-Male]

would be cool
but seems like it'll take a real pro to port this (if it's possible at all)

[HighInBC]

If instead of using the matlab code, one could use the math in the paper they released... The actual formula is rather simple. The paper describes the proccess in plain english and math.

The ability to selectively color, or recolorize would really boost the functionality of avisynth. I am afraid it is a bit beyond my programming abilities.

[E-Male]

rather simple??
the formula maybe is, but the whole process...

implementing it in c++ and making it run efficiently seems quite complex to me


EDIT1:
Now given a set of locations ri where the colors are specied
by the user u(ri) = ui; v(ri) = vi we minimize J(U); J(V) subject
to these constraints. Since the cost functions are quadratic and
the constraints are linear, this optimization problem yields a large,
sparse system of linear equations, which may be solved using a
number of standard methods.


EDIT2:
gues i spoke to soon
http://www.ece.northwestern.edu/~mya671/Colorization.htm
nice and small c++ colorizing tool


i've e-mailed the authors of both projects to see if they will publish the c++ code we need


EDIT3:
maybe one of our pros (if any reads this) can use this code already:
http://www.ece.northwestern.edu/~mya671/files/FinalReport.pdf

[E-Male]

we won't get any code or help for the method in the first post
the university is going to use it comercially
the people who wrote it can't do anything about it

[HighInBC]

Hmm.. Here is hoping... I would not care about speed for something like this, if it take 5 days to colorize a scene I don't mind, as I am making my own films and only have to do this sort of thing rarely.

Hmmm remember that green chick from original star trek? What if I made Picard green in all the TNG ep's ahaha.

[Mug Funky]

the university is going to use it comercially
the people who wrote it can't do anything about it

dammit, that annoys me.

a lot of universities (ie the one i went to and probably all of them, truth be told) make you sign an IP waiver before you are allowed to start the course - anything you submit can be used by the uni in a manner of their choosing.

and with the slump in government funding for education, more and more universities are having to pay their way - that means stealing their students' hard work and making money off it.

[edit]
a colouriser would be very cool though. i'm thinking fixing chroma dropouts in composite/component analog captures. with the last good frame combined with avisynth's existing motion-compensation functionality (MVtools), plus this handy colourisation stuff, there'd be no percievable difference from the original.

[E-Male]

well, there is those 2nd project (partially based on the other) :
http://www.ece.northwestern.edu/~mya671/Colorization.htm

nice small c++ program is there for dl :
http://www.ece.northwestern.edu/~mya671/files/colorization.rar

and some core-source-code in the paper, too (for dl) :
http://www.ece.northwestern.edu/~mya671/files/FinalReport.pdf

and i'm still waiting for an e-mail reply from the author

if he doesn't reply we might be able to do it with the code from the paper (experienced programmers, i'm looking at you guys)

that OpenCV library is news to me, so it's somewhat cryptic to me



here is the code from the paper:

#define MAX_BLEND_COLOR 3 //Max number of color to be blended
#define MAX_DISTANCE 65535 //Max intrinsic distance of pixels
///////////////////////////////////////////////////////////////////////////////////////
// Pixels representation for blending colorization algorithm
///////////////////////////////////////////////////////////////////////////////////////
class CBlendPixel
{
int m_x; // x-coordinate of the pixel
int m_y; // y-coordinate of the pixel
COLORREF m_c[MAX_BLEND_COLOR]; // maintained chrominance
float m_d[MAX_BLEND_COLOR]; // maintained intrinsic distance
};
typedef set<CBlendPixel*> BlendPixelSet;

///////////////////////////////////////////////////////////////////////////////////////
// Colorization Algorithm
// Blending the color with the intrinsic distance as the weights
///////////////////////////////////////////////////////////////////////////////////////
class CBlendColorize : public CColorize
{
public:
BOOL Colorize(IplImage* pColorImage,const IplImage* pGreyImage, const IplImage* pLayerImage);
CBlendColorize();
virtual ~CBlendColorize();
protected:
void BlendColor();
void PropagateColor();
void GenerateColoredSet();
BOOL ModifyPixel(CBlendPixel& testPixel,const CBlendPixel & curPixel);
BOOL Internal8(int x,int y);
inline double WeightDistance(double d);
void AllocateArray();
void ReleaseArray();
int m_iColored;
BlendPixelSet m_ActiveSet;
CBlendPixel*** m_PixelArray;
};

// Iteratively propagate the active colored pixels to their adjacent pixels until the equilibrium
void CBlendColorize::PropagateColor()
{
CString str;
BlendPixelSet::iterator it =m_ActiveSet.begin();
BlendPixelSet::iterator it_temp;
while (!m_ActiveSet.empty())
{
it =m_ActiveSet.begin();
while (it!=m_ActiveSet.end())
{
CBlendPixel * pPixel = *it;
for(int y=pPixel->m_y-1;y<=pPixel->m_y+1;y++)
for(int x=pPixel->m_x-1;x<=pPixel->m_x+1;x++)
{
if (x==pPixel->m_x||y==pPixel->m_y)
if (x>=0&&x<=m_width-1&&y>=0&&y<=m_height-1)
{
CBlendPixel * testPixel = m_PixelArray[y][x];
if (ModifyPixel(*testPixel,*pPixel))
m_ActiveSet.insert(testPixel);
}
}
it_temp = it; it++;
m_ActiveSet.erase(it_temp);
}
}
}

//Modify the color list of curPixel according to testPixel
BOOL CBlendColorize::ModifyPixel(CBlendPixel& testPixel,const CBlendPixel & curPixel)
{
if (cvGetReal2D(m_pY,testPixel.m_y,testPixel.m_x)==1)
return FALSE;
double y1 = cvGetReal2D(m_pGreyImage,curPixel.m_y,curPixel.m_x);
double y2 = cvGetReal2D(m_pGreyImage,testPixel.m_y,testPixel.m_x);
double d = abs(y1-y2);
BOOL result = FALSE;
for(int i=0;i<MAX_BLEND_COLOR;i++)
{
if (curPixel.m_d[i]<MAX_DISTANCE)
{
int j=0;
while (((curPixel.m_d[i]+d)>=testPixel.m_d[j])&&(j<=MAX_BLEND_COLOR))
{
if (curPixel.m_c[i]==testPixel.m_c[j])
{
j=MAX_BLEND_COLOR;
break;
}
j++;
}
if (j<MAX_BLEND_COLOR)
{
if (curPixel.m_c[i]!=testPixel.m_c[j])
for(int k=MAX_BLEND_COLOR-1;k>j;k--)
{
if (testPixel.m_c[k-1]!=curPixel.m_c[i])
{
testPixel.m_c[k]=testPixel.m_c[k-1];
testPixel.m_d[k]=testPixel.m_d[k-1];
}
}
testPixel.m_c[j]=curPixel.m_c[i];
testPixel.m_d[j]=curPixel.m_d[i]+d;
result = TRUE;
}
else
break;
}
}
return result;
}

///////////////////////////////////////////////////////////////////////////////////
// Command Pattern to support Undo and Redo functions
///////////////////////////////////////////////////////////////////////////////////
class CCommand
{
public:
virtual BOOL Execute(IplImage *pImage,WPARAM wParam, LPARAM lParam);
virtual BOOL UnExecute(IplImage *pImage);
virtual BOOL ReExecute(IplImage* pImage);
CCommand();
virtual ~CCommand();
protected:
BOOL m_bExecuted;
};



samples:

http://www.ece.northwestern.edu/~mya671/files/color/Shirley_t.jpghttp://www.ece.northwestern.edu/~mya671/files/color/Shirley_n.jpg
http://www.ece.northwestern.edu/~mya671/files/color/lilies_t.jpghttp://www.ece.northwestern.edu/~mya671/files/color/lilies_n.jpg
http://www.ece.northwestern.edu/~mya671/files/color/Sunset_t.jpghttp://www.ece.northwestern.edu/~mya671/files/color/Sunset_n.jpg
http://www.ece.northwestern.edu/~mya671/files/color/Arch_t.jpghttp://www.ece.northwestern.edu/~mya671/files/color/Arch_n.jpg

[guada 2]

Very interesting, if one wishes to restore old photographs.

But somebody has it to try to remove the color of an image (VDM+BMP+Flitre MSU OLD CINEMA) to treat it with this process?

It would be interesting to make the comparison of the 2 images:
the original with its own colors, and the image treated with absence of color.

[E-Male]

>>Very interesting, if one wishes to restore old photographs.

-try the sample program :)


>>But somebody has it to try to remove the color of an image (VDM+BMP+Flitre MSU OLD CINEMA) to treat it with this process?

-no, at least not in the example program
i guess in most implementations the colors of the picture would just be ignored


>>It would be interesting to make the comparison of the 2 images:
the original with its own colors, and the image treated with absence of color.

- http://www.ece.northwestern.edu/~mya671/Colorization.htm
(i just didn't link those to keep the post from getting to wide)

[guada 2]

Thank you, E-Male. :)
However, you will excuse me I had not opened your first link.

[HighInBC]

ok, here is a side effect that I just realized.

If you colorize only a bouncing ball in a movie with blue, then took that blue and turned it to white and everything else the black then you would have automatic mask detection for moving objects... If anyone has ever done any mask related special effects they will surely see the advantage of this.

The program seems to be able to track objects that are moving, while the camera is moving, ontop of a moving background.

This is very hard to mask normally, and ussually must be done by hand frame by frame.

[E-Male]

so you colorize the ball frame per frame and use for masking
doesn't seem to help much IMO
(in case my irony is bad as always: this method for now is for stills only, we'll have to add video automation ourselves, maybe include mvtools funcionality, but let's do still colorizing first)

I think Hero-- or a mod should change the thread titls, in order to have the coding pros look in here

else i'll have to try to make sense of that code, which won't be pretty

[HighInBC]

Originally posted by E-Male
so you colorize the ball frame per frame and use for masking
doesn't seem to help much IMO
(in case my irony is bad as always: this method for now is for stills only, we'll have to add video automation ourselves, maybe include mvtools funcionality, but let's do still colorizing first)

I think Hero-- or a mod should change the thread titls, in order to have the coding pros look in here

else i'll have to try to make sense of that code, which won't be pretty

If you look lower on the page(where it says 'Video Clip Colorization Examples'): http://www.cs.huji.ac.il/~yweiss/Colorization/

You will see that the formula does traverse space and time, needing only occasional updates as it loses the shape.

[E-Male]

if you look higher in this thread you will see that we can't use that algo, or at least won't get any help doing it

EDIT: sorry for my tone, i was in a bad mood when i replied (just got up)

i think we should first focus on getting code for still-colorisation working, when we got that we can expand in any direction we want

[insanedesio]

Sounds interesting to me. I wouldn't mind coding this, especially since I've been looking for something to code. As it stands, though, I'm new to AVS development, or any kind of coding that has actual applications, for that matter :-\. Given that I'm swamped in finals right now I don't have the patience to think about that paper, but after finals hopefully I should have more thought to give it... I read it through and thought about it a bit and can see how to implement most of it.. just missing the main part I think. If I get a chance I'll ask some people that I know who might be able to help.

[E-Male]

nice

i've been doing some more research on the topic and found some more papers and algos
i'll post a summary with all links here after i'm sure which paper is based on which others and after i've mailed all authors and waited some days

then i'll also make a new thread to attract some more experienced programmers from this board
together we should be able to make open-source colorization possible

[E-Male]

got e-mail reply no. 2 (of 3)
again university patenting
so only one left to put hopes on
if it's an other 'no' we have to do the coding from scratch with the avaible information

[insanedesio]

The one called FinalReport.pdf or whatever seemed a bit useful, I managed to sift through it up to the ModifyPixel function, and then got confused. I'll have to take a closer look when I get a chance.

[E-Male]

GOOD NEWS:
Ming Yang, author of "Still Image Colorization (http://www.ece.northwestern.edu/~mya671/Colorization.htm)" has replied to my mail
He plans to release the source!
I still have to talk to him a bit about GPL and stuff, but things look very good
I'm in a hurry right now, just wanted to tell the good news

[tritical]

This could be way off, but just thought I'd throw in what I quickly gathered from the two papers.

The finalreport.pdf method has quite a few steps but is pretty easy to follow as far as I understand from the webpage and code. Basically, you start with the original image with a given set of pixels (say, set A) that have had their colors manually defined. Inside this set you define groups (say i, ii, iii, ...), each group contains all of the pixels of matching color that are connected (i.e. one group for each manual color marking). Next, for every pixel in the image that is not in set A, you find the top three colors or groups for it. The ranking is done by smallest luma difference. To find the luma difference to each group from a single pixel, you find the smallest luma difference of any pixel in the group to that pixel. Once you have these rankings defined there is a propagation step that swaps the ranked colors between each pixel not in set A and its surrounding 8 neighbors. This swapping is based on local luma differences between each pixel and its 8 neighbors. The swapping routine continues until it converges (i.e. no more changes occur). After all of the above, you end up with 3 color values for each pixel. The last step is you average these 3 colors together based on the weighting function they give which is a function of luma difference. This could possibly be improved by taking distance into account as well.

The other method seems as though it could implemented as an iterative minimization process using the local average function they define in the paper. This would slowly propagate the colors throughout the image without colors propagating across major boundaries. The process would be similar to iterative smoothing processes like anisotropic diffusion. Also, while it does use motion vector information, it is only to define neighbors in the surrounding frames and not to propagate colors in time. So one could ignore that part initially and make a single frame version. My main concern would be it could take quite a while (many iterations) for such a method to converge or fully propagate colors on large images. The paper says it takes something like 15 seconds per image and the images they used are relatively small.

[E-Male]

Originally posted by tritical

The finalreport.pdf method has quite a few steps but is pretty easy to follow as far as I understand...we'll have the complete source of that soon
plus people already looking into it

so we might really have a colorisation plug-in in the near future

i already got some idea of combining this with mvtools and maybe my pixeledit plug-in to get a video-version (without 'scribbeling' each frame)

exiting to have another expensive thing going open-source

[guada 2]

" so we might really have a colorisation plug-in in the near future.
i already got some idea of combining this with mvtools and maybe my pixeledit plug-in to get a video-version (without 'scribbeling' each frame)
exiting to have another expensive thing going open-source "


Very interesting in the content and of the form.
I hope that that will be enough for you to lead to "artistic work" conclusive.

Good luck E-Male :)

[insanedesio]

I'm working on a smoother based loosely on the stuff in the original paper. I'm doing it not so much to make a good smoother, because I don't know how well that algorithm will work out for smoothing purposes. I'm doing it more as a first step in this process. Once I get that done, and after my final on Saturday, I'll start looking at putting actual colourization code into it, although the way I'm trying to implement it, I might not have to do much... but the paper itself isn't that informative, so it's hard to tell what exactly will happen. Plus, I don't really know what I'm doing :D.

After that, I'll hopefully take a look at MVTools to add in temporal neighbour detection.

EDIT:
Oh, yeah, and thanks to tritical who's helped me a lot in getting the smoother started.

[E-Male]

here is the full source (needs OpenCV):
http://e-rels.dyndns.org/downloads/ColorSource.rar
it's GPL, the author asks for his homepage url to be kept in (also in all code based on this), which seems fair to me

i couldn't get it too compile,yet
if someone knows how fix that, please post
i get these errors:

MainFrm.cpp
u:\colorize\MainFrm.cpp(39) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CMainFrame::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type
ColorView.cpp
u:\colorize\ColorView.cpp(40) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CColorView::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type
u:\colorize\ColorView.cpp(41) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CColorView::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type
u:\colorize\ColorView.cpp(42) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CColorView::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type
u:\colorize\ColorView.cpp(43) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CColorView::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type
u:\colorize\ColorView.cpp(44) : error C2440: 'static_cast' : cannot convert from 'void (__thiscall CColorView::* )(WPARAM,LPARAM)' to 'LRESULT (__thiscall CWnd::* )(WPARAM,LPARAM)'
None of the functions with this name in scope match the target type

[E-Male]

while i can't compile the stand-alone tool i had a look at the code and at the opencv library
and it seems like putting the colorozation algorithm inside an avisynth plug-in won't be hard
there also is room for improvement:
it needs Y,Cb,Cr which is similar to YUV and can be fed from avisynth directly (no need to go over rgb conversation, also i'll do that in my first test)
the way opencv stores images in memory is about the same as avisynth uses, so many thing will work without copying, just pass pointers (for the non-programmers: there shouldn't be a big slowdown caused by avisynth<->openal conversions)

[E-Male]

i got a bit further and it seems that a full avisynth port (opencv-free) will be possible

i'm not sure yet what's the best (most pracial & userfriendly) way to feed the needed material (gray clip, colors and mask)
but i think i'll soon have a test version ready

[HighInBC]

Single frame clips would be good, or perhaps a built in image loader.

If it is going to have a temporal aspect aswell, then you will need to add color touchups every so often, a good interface with this would be to just name the file like so:

color00.bmp
color07.bmp
color18.bmp
color97.bmp
...

then the plugin will know the framenumbers from the filename. I hope this makes sense, it is only 6am for me.

[E-Male]

nice idea, i'll keep that in mind when advancing to the video version

[insanedesio]

Finally got the smoother finished. There are a couple things left that I want to add to this before I do the actual colourizer. A lot of the code for the colourizer will be copied verbatim from the smoother.

Here's the smoother:
ColourizeSmooth v0.7.4.0 (http://www.ualberta.ca/~ab31/ColourizeSmooth.rar)

EDIT: v0.7.4.0 up with YV12 support. Same link above. Probably the last change for a couple of days, and then I'll start looking at temporal neighbour detection.

[E-Male]

had a quick look, but can't test in the next view days

since we're both working on colorizing plug-ins it might be best that we join our efforts

[insanedesio]

Yeah, that might work, but you're working yours based off the code from that other guy... I'm trying to get something off of the original algorithm. Besides, I'm still working on the smoother, couple things I want to add (temporal neighbour detection being the most important). I also want to add YV12 support but it seems like a pain in the ass to upsample it to YUV or downsample YUV to YV12, so I might drop that because I'm just that lazy.

[guada 2]

insanedesio,

It is a beautiful initiative, good continuation :)

[E-Male]

got Ming Yangs code ported, working on the interfaces now

[HighInBC]

You rock, thanks!

[jimlong]

Hi E-Mail,

Ay progress with this project? I saw this post late and it seems it's been one year since last post. However, I'm still very interested about this great idea. I tried to download ColourizeSmooth v0.7.4.0 and colorsource.rar from your site but failed. Is http://e-rels.dyndns.org still on?

Thanks,

Jim

[anton_foy]

Hmmm this is very cool indeed.

I have always been very concerned about my lame blue skies in my footage. Maybe this would get my deep blue look im searching for :p

[Hero--]

I found this thread:
http://forum.doom9.org/showthread.php?t=93990

[Petetheking]

Hello ALL


Plz can anybody send me this rar file (complete file of colorization using optimizing cause link doesnt work)

Please im also very interested

THANKS a lot

Greets Peter

[jimlong]

I found this thread:
http://forum.doom9.org/showthread.php?t=93990

Thanks, Hero. Mohan's work is really great. I also posted a message in that thread to see if anyone knows where to find e-male's work as well.

[E-Male]

hi guys & sorry for disappearing

I'll try to get a dyndns working again so you can get the files you want

I might also do some more video related coding soon (after doing Java for some time)

EDIT: for now:
http://rapidshare.com/files/10846379/ColorSource.rar.html
should(!) be Ming Yang's original code, not modded by myself

[jimlong]

Thanks, E-Male. File downloaded. Will take a look and compare with Mohan's work soon.

[Petetheking]

Hi @ll!!!

How must i implement Visual c++ 5.0 with OpenCV to Compile his code property??
#
Can anyone give me a short tutorial ??

Plz cause first time with visual c++ and OpenCV!!! Normally i Made it in C++ Builder or Delphi


Thanks A LOT

Greets Peter

[Zarxrax]

Hmmm, this seems very interesting. It seems like this would be extremely useful for the process of rotoscoping--that is, isolating an object from the rest of the frame. This is typically a very laborious process, as anyone who has ever used photoshop to remove the background from behind an object should know (just imagine doing that on every frame of a video clip!). The way this colorization process is able to detect the edges of objects, it seems that this could almost automate the process with very little user input.
Could anyone comment on whether this code would indeed be suitable for such a process?

[chr2000]

hi guys & sorry for disappearing

I'll try to get a dyndns working again so you can get the files you want

I might also do some more video related coding soon (after doing Java for some time)

EDIT: for now:
http://rapidshare.com/files/10846379/ColorSource.rar.html
should(!) be Ming Yang's original code, not modded by myself
Hi, is there anybody post the code again? I can not download the code. thanks

[Fizick]

I should have it somewhere at harddrive (original, not E-male plugin).

What do you want to do with a code?

[chr2000]

I should have it somewhere at harddrive (original, not E-male plugin).

What do you want to do with a code?

I just want to know how to color one image correctly, and does this code works?
I had done some work about tranfering color to gray image serveral years ago. I know that it is very difficult to adapt the algorithm to all kinds of image.

[Fizick]

i found the archive (in my "Imaging" foder, not video-related). where to put it, to preserve it from disappearing again? It is 1.62 Mbytes.