Automatic Colorization - Tutorial (Windows 10 edition)

[color]

LAST UPDATED: 2019-12-11 - Updated the install script and to the newer version on Ubuntu Shell in Windows and added new images. Step 2 has a little bit more to do than before. If you get error try:
$ sudo apt --fix-broken install python-pycurl python-apt

EDIT: There might be different between different countries(?) wierdly. Stick with the "Ubuntu" version (not the 18.04 or 16.04)

Only works with 64-bit version on Windows 10.

This works great on scenes that are filmed outdoors, some indoors works great as well mostly skin colors but some backgrounds indoors may be colorized a bit wrong. Cartoons does sadly not work at all, it has problem to regonize drawings. If you want to colorize cartoons i recomend Chromajig (https://forum.doom9.org/showthread.php?t=173364&highlight=chromajig), CFCF2 (https://forum.doom9.org/showthread.php?p=1785308#post1785308) or HaldCUT (https://forum.doom9.org/showthread.php?p=1828774#post1828774)

1.) First off all, see that your computer has the latest updates, then go in to Control Panel > Programs > Turn Windows Features On Or Off. Enable the"Windows Subsystem for Linux" and click "OK". Then restart your computer (important step)

https://image.ibb.co/nOOUMz/01_1.png

Then go in to Microsoft Store and install Ubuntu (https://www.microsoft.com/en-us/store/p/ubuntu/9nblggh4msv6). Then open Ubuntu in the start menu. You will get the step to write a nickname and password

https://i.ibb.co/4PGVP3q/Sk-rmbild-3.png

https://i.ibb.co/B6YqMgt/Sk-rmbild-4.png

2.)Before installing, you need to update ubuntu to the latest package.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade
$ grep -r lucid /etc/apt/sources.list*

3.) INSTALLING TORCH

$ git clone https://github.com/torch/distro.git ~/torch --recursive
$ cd ~/torch; bash install-deps;
$ ./install.sh
$ source ~/.bashrc
$ ./clean.sh
$ luarocks install image
$ luarocks list
$ sudo apt install torch-repl

(It will take a while to install, and you will need to write your password a few times and answer "yes" for some installed)

then test it:
$ th

It should look like this:
https://image.ibb.co/n07mZK/skarmbild_2_1.png

4.) INSTALL LET Let There Be Color! (http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en/)
Download here (https://github.com/satoshiiizuka/siggraph2016_colorization). Extract the files in a folder called "color" in the download folder.

$ cd /mnt/c/Users/(YOUR USERNAME)/Downloads/color
$ ./download_model.sh

change to your username "(YOUR USERNAME)". It is sensitive with capital letters.

Now this is done its time to get the files to colorize.

5.) Open a video in virtualdub, change the resolution so the width is 220px or 226px and export it as image sequence (put in the "Minimum number of digits in" so many numbers that is in the movie, if its 3400 frames you need to write 4, this is so you easy can transfer the colorized images if you need to stop and start another time). in the color folder you extracted the files in. Also see so its PNG, that will colorize the images faster.

6.) SCRIPT TO CONVERT ALL IMAGES

$ cd /mnt/c/Users/(YOUR USERNAME)/Downloads/color
$ for file in * ; do th colorize.lua "$file" "${file/./-color.}" ; done

It takes hours and hours. but it will be done.

Here is one before and after image:
https://image.ibb.co/ePxLuK/000_1.pnghttps://image.ibb.co/nDwDEK/000_color_1.png

As you see here there is an error, its no problem, that is couse its trying to colorize all files in the folder, and it can only colorize images files not anything else.
https://image.ibb.co/d8QOEK/skarmbild_5_1.png

7.) When it has colorized the images you need to do a avisynthscript to merge the color (overlay) on the high quality greyscale image.

I don't get a good script for how to use the image directly (like with ImageSource()), so I import the first image "*-colorized.png", change the framerate so it matches the original (important) and export it.

Now its time to merge the colorized video over the original grayscale video. Use the script that is best for you, I use:

B=DirectShowSource("original_video.mp4")
C=DirectShowSource("colorized_video.avi")
Overlay(B, C.BicubicResize(B.Width, B.Height), mode="chroma")

If you want to do a bigger quality 1080p or higher, I suggest you to upscale the colorized photo to 720p (use Lanczose3) and use warpsharp (Depth in about 114 is good, do not change the Blur passes), otherwise there will be some color bleeding.

A sample, klick on the image to watch it on youtube. (I don't know how to post so you can see the video directly here)

https://preview.ibb.co/kmW6Gz/Namnl_st_2.jpg (https://www.youtube.com/watch?v=eXQu3MNuPY4)

https://preview.ibb.co/b9rw3e/gypmzl16.jpg (https://youtu.be/EKeV9Edhq6k)

I'm don't know realy how to write the script with the imagesource. import the images in avisynth and export it with the right framerate. But if you know how please help me with the script:

B=DirectShowSource("high quality movie.mkv")
C=ImageSource("00000.jpg")
Overlay(B, C.BicubicResize(B.Width, B.Height), mode="chroma")

[color]

I will update with more images soon. :)

[StainlessS]

I'm don't know realy how to write the script with the imagesource. import the images in avisynth and export it with the right framerate. But if you know how please help me with the script:

Not sure if something like this is all that you want. (untested)

B=DirectShowSource("high quality movie.mkv")
FC=B.FrameCount # Assuming same number of frames
C=ImageSource("00000.jpg",End=FC-1)
Overlay(B, C.BicubicResize(B.Width, B.Height), mode="chroma")
AssumeFPS(B.FrameRate,True)
# ReSampleAudio(B.audioRate)

# Export in VirtualDubfilterMod


PS, DirectShowSource is rather frowned upon as being Frame Inaccurate.

EDIT: Added in blue.

[color]

I will try it out tomorrow, thank you for the script. What is better to use instead of directshowsource? I want to make the best tutorial.

Btw, i know there is some spelling error. I will fix it as soon as possible. :)

[Mounir]

Lsmash is best imo

[johnmeyer]

If I want to do this, can I just boot from an Ubuntu CD/DVD? That's how I've been running Ubuntu up until now.

[color]

Mounir, I will check that out.

johnmeyer. Yes it work with booting from Ubuntu CD/DVD or USB, but you need to do a "sudo apt-get update" so you get the latest packages in terminal first. It does take a bit longer in "pure install" of Ubuntu wierdly. I will do a tutorial in ubuntu also. But its almost the same, just start from the 3.) and that you are in the root folder when you install torch.

I did have some problem in Ubuntu, so after I installed torch i did:

./clean.sh #before the code "th" to see that its correctly installed. (on Ubuntu, no need in Windows Ubuntu install)

[johnmeyer]

I will be very interested to see what people can do with this. I don't have time to play with it right now. StainlessS and others spent a lot of time on the goofy colorization code that was posted here a couple of years ago. It was tantalizing, but if one was honest, it really didn't work very well.

I just looked at the samples on the download page you linked to, and they look incredibly good. This would be a big deal for me if I could actually colorize some of the work I do for clients.

[poisondeathray]

Has anyone used larger than 224px width images successfully ? If so can you post some more examples? The "thumbnails" look decent, but I'm curious if it's actually usable for decent image sizes


While the model works on any size image, we trained it on 224x224 pixel images and thus it works best on small images. Note that you can process a small imageto obtain the chrominance map and then rescale it and combine it with the original grayscale image for higher quality.

[color]

poisondeathray, yes you can do bigger than 224px, but it takes longer time to let it colorize the photo. For be the smaler one takes 6 seconds per image in that small size. I used 1280px it tooked more than a minute. The best way is to use 220px or 224 px that it is trained for. I do merge the color only over the original high quality picture and it look great, I'm colorizing a 1080p movie and it tooks great.

I will post some examples soon. I have not been home so mutch lately. I think it will be up under this day or latest tomorrow. :)

[color]

I will be very interested to see what people can do with this. I don't have time to play with it right now. StainlessS and others spent a lot of time on the goofy colorization code that was posted here a couple of years ago. It was tantalizing, but if one was honest, it really didn't work very well.

I just looked at the samples on the download page you linked to, and they look incredibly good. This would be a big deal for me if I could actually colorize some of the work I do for clients.


Yes it does an amazing work. There is one problem tho', It works better with outdoor images. Inside does work but it has sometimes need to adjust the brightness and contrast.

Also there might be some flickering, I do use CNR2 in virtualdub and a deflicker filter, I haven't tried the CNR-filter for avisynth yet. But I guess that might be faster and work a bit better. It does give you more controll.


EDIT: Just added a link to a sample, the original video is found here and is legal to use, change and publish without any permissions (https://pixabay.com/sv/videos/solnedgång-landskap-idyll-kor-10737/).

[color]

I do want to put out the limitations, as you can see when the cows get blurry, the script recognize them as gras so they get a little bit green. But CNR2 (http://avisynth.nl/index.php/Cnr2)will fix that

[Gser]

What's this whole install linux on windows business?

[Yanak]

Nice tutorial, i might give it a try with a live CD as not having win10, never tried to play a bit with linux but might be the occasion to try when i'll have a lot of free time to burn.

This said i had used this in a form of online service some years ago, had forgot about this until seeing this thread : https://colorize.dev.kaisou.misosi.ru/ ( pics resized to fit 800px max on the largest side )

examples where it does a nice job :
https://colorize.dev.kaisou.misosi.ru/result/f1b97f77-00c0-4e35-b9f1-691bb96ddfdb
https://colorize.dev.kaisou.misosi.ru/result/8b780679-a164-46df-b96c-12ffbe23b23d
https://colorize.dev.kaisou.misosi.ru/result/4fc2a347-a02b-4232-b41c-80764b2a52e0


For some footage it's really impressive, outdoors where it was trained especially, for some others it used to give some things not really better than the colorize script i saw a few times in the avisynth section of the forum.

Thanks to "color" for reminding me this exists, one could only wish it was easier and directly available on win without the linux steps :)

[color]

Gser, its needed for the colorization. The "program" is made for linux.

Yanak, thank you. What windows version do you have? Its a good page, but they have made a bit approvment after that page you liked to, they have an officiell online colorization here (http://hi.cs.waseda.ac.jp:8082/), it is mutch better than colorize.dev.kaisou that uses the older trained file. Yes I wish it was easier, but to install the ubuntu terminal in windows isn't that hard and works great. It does work with cygwin also (works with older windows versions), but there is some other scripts like "sudo apt-get" does not work its something else for that command, its kind of hard to install. There is win-bash also, but I have not tried it. I don't know about Windows 7, it might have the option also with latest update to in "Windows Subsystem for Linux" I don't use 7, 8 or 8.1 so I can not check.

[StainlessS]

You can also get a live image linux on USB flash stick with added persistence, which can be used to store settings/updates (Read/Write partition).

EDIT: on Google:- https://www.google.co.uk/search?source=hp&ei=ATulWrLoMIadkgXhsbiIBw&q=live+linux+persistence&oq=live+linux+persistence&gs_l=psy-ab.3..0j0i22i30k1l8.1773.6866.0.7139.22.22.0.0.0.0.224.2811.2j17j2.21.0....0...1.1.64.psy-ab..1.21.2806...0i131k1j0i131i46k1j46i131k1j0i10k1j0i22i10i30k1.0.FwIvKMcaA6Y

[Yanak]

I'm on win7 and there isn't anything like this sadly, so a usb stick or live CD are probably the only options, something i will need to try and explore one of those days.

Thanks for the link on them page, had not used this one before, but looks like they resize more compared to the other allowing 800px, will need ot compare both results and see if it is really improved, especially when there isn't much landscape where it seems to be quite good..

Well i'm not into linux stuff at all but will have to try this one of those days, what StainlessS said seems like a good option, especially curious about higher resolutions and compare original grayscale input with the tool does on output.

Thank again.

[color]

Outdoor it looks the same. But there is better with humans in photos indoors. It gives more different colors than all similar as the other one. But it does also sometimes help to adjust the brightness and contrast. But yes it has some limitations.

[StainlessS]

Further to links for live linux USB persistence, (from one of the links on posted google results), and those wanting persistence greater than usual
limitation of 4GB, see here (I found it of particular interest wanting to install on 64GB USB3 with full drive usage):- https://askubuntu.com/questions/397481/how-to-make-a-persistent-live-ubuntu-usb-with-more-than-4gb

[smok3]

Quick test (using web service) on some cgi (1. orig, 2. gimp grayscaled, 3. ai colorized)

https://cdn.scrot.moe/images/2018/03/12/rocketAIcolorized.jpg

p.s. Getting this error in Debian 9
th colorize.lua ~/Downloads/vintageRocket11d_gimp2.png
/home/b/source/torch/install/bin/luajit: cannot open <colornet.t7> in mode r at /home/b/source/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7f7ea2f4a3a0
[C]: in function 'DiskFile'
/home/b/source/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
colorize.lua:24: in main chunk
[C]: in function 'dofile'
...urce/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x5621a1c52610

[color]

Did thorch work after installation? Are the color-files in the same folder?

[smok3]

Did thorch work after installation?
Yes.

Are the color-files in the same folder?
Thorch folder or some other folder?

[color]

Thorch folder or some other folder?

the colorizing files should be in the same folder as the images that are supposed to get colorized. Don't forget to download the model "./download_model.sh", that has to be in the same folder as well, cause its the trained file on how it should colorize.

EDIT:
You need to run the script in the same folder as the coloring files are.

[smok3]

@color: thanks, I have missed more than one step, downloading model now ...

[color]

It's easy to miss. :) Hope it works. Otherwise feel free to ask.

[trondmm]

This looks really promising. I'll have to play around with it this easter.

I'm wondering - if I have parts of the clip in color, can I use that to help the colorizer figure out the correct colors?

[amayra]

i hope someone port this as AVS/VS Plugin somedays

[color]

Trondmm, sadly no, thats not really how it works, the file is trained by images they used who created the "coloring AI". There should be a way to teach it but I don't know how and didn't get answer when asking them.

Amayra, I hope someone will port it, there is a port off another system but its not free and they take pay for every image you colorize and they are not as good (in my eyes) http://demos.algorithmia.com/

[color]

I also want to add that using CNR2 you can get better color flow. (if you know what I mean)

[rsd2015]

I will try it out tomorrow, thank you for the script. What is better to use instead of directshowsource? I want to make the best tutorial.


color, your tutorial worked fine for me on Win 10 x64, I've got the original and colorized images and .mp4s, but I'm not sure how to combine them in an avisynth script.

[color]

rsd2015, I did import the images, change the framerate to the same as the original and resized the image, using warp sharp to not get color bleading when mergin. Then I used:

B=DirectShowSource("original_video.mp4")
C=DirectShowSource("colorized_video.avi")
Overlay(B, C.BicubicResize(B.Width, B.Height), mode="chroma")

I know people don't like DirectShowSource, but I don't know whats better and I have a lot of problem with imagesource and similar to get the framerate right. This works for me. But if someone know better for mp4 that works great just give me a hint. :)

PS. I'm logging in once a week, so you guys know I'm not gone. :)

[color]

A little update. There were an update not long ago on "Ubuntu on Windows". It seems like its faster. One movie that tooked 36 minutes before, now took 28 minutes. :) Good news, it might be faster on some movies to colorize. :)

[wonkey_monkey]

Any chance you could post a link to an output video? A scene with lots of cuts and different angles?

[color]

There is on the first page one clip (to YouTube) if you click the image.

Link (https://youtu.be/eXQu3MNuPY4)

I will try finding a cc0 movie that I can show more on indoor and outdoor scenes.

[color]

Here you go, just klick on the image to watch it on youtube there is two scenes filmed indoors:

https://preview.ibb.co/b9rw3e/gypmzl16.jpg (https://youtu.be/EKeV9Edhq6k)

[color]

New step added cause Ubuntu does not update automatic anymore, well not in 18.04. The lates one is a bit faster also to colorize, so it's a great idea to update.

[color]

I have changed my e-mail and notised that my youtube channel was connected to it so the video here is deleted.

I have now installed windows 10 may update (64-bit home). But I do have some small problems getting the install script to install. I will try to get it to work and figure out whats the problem.

[RocketJet]

When executing the script "./install.sh"
I get the error message "cmake: command not found."

Please help, I know nothing about Unix.

[StainlessS]

RocketJet:- https://www.google.co.uk/search?sxsrf=ALeKk036wZebvMojdqEYxW-2_5SSb093xg%3A1592133949120&source=hp&ei=PQnmXpHeBIzUUdX4peAC&q=install+cmake+ubuntu&oq=install+cmake+ubuntu&gs_lcp=CgZwc3ktYWIQA1CmmAJYvqUCYOW2A2gAcAB4AYABAIgBAJIBAJgBBqABAaoBB2d3cy13aXo&sclient=psy-ab&ved=0ahUKEwjR8OCTmYHqAhUMahQKHVV8CSwQ4dUDCAg&uact=5

EDIT: To below: Sorry cant help further.

[RocketJet]

Thanks StanlessS.

Sorry, but I have another install problem.

In the command: bash install-deps
I got the following error message:

Package python-software-properties is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
software-properties-common

E: Package 'python-software-properties' has no installation candidate

I tried replacing python-software-properties with
software-properties-common in install-deps,
but this made no difference.

I found python-software-properties on the Ubuntu site,
but there is no documentation on how or where to install it.

Can anybody help?

[color]

RocketJet, did you get it to work? Are you i Ubuntu terminal or CMD?

If something did not install correctly you can do the installation again.

[color]

This is a bit old and needs a new trained file.

But i recommend DeOldify that is online in Colab. Its slow but it works great and mutch better. Also they updates the script now and then.

https://github.com/jantic/DeOldify

[Forteen88]

Did anyone of you try to colorize classic Kurozawa-movies, like the great "Seven Samurai" or "Yojimbo"? I'd like to see screenshots of that.