AviSynth for CinePaint / Photoshop / PaintShop Pro??

[Karyudo]

Is there any such thing as an AviSynth-style frameserver or other helper app for Photoshop-style paint programs?

I'm thinkin' an effective frame server or helper app would be able to supply "virtual" PNGs to a paint app, and be able to keep a running log file of actual frames it writes out, for later re-integration into an AVS script.

I know this is asking a lot, but think of the possibilities: a way to make manual (not just mathematical, procedural) edits to a video, while still retaining all the benefits of AviSynth!

For example, in a lot of cases it would be much faster to actually do a quick manual touch-up to some problem frames (e.g. dust, dropouts, etc.) and be able to load the manually-fixed frames with Image() than it would be to find some magical combination of AviSynth filters to have the script weed out such flaws. But right now it's incredibly clunky to do so.

After all, if major motion picture companies do dust and scratch removal pretty much by hand, there must be a good reason. Throwing math at the problem must not be the best solution (for whatever reasons: cost? time? accuracy?)!

I'm guessing there is no such animal, which means my next question is, "Is anyone interested in writing a frame server / helper app for external painting?" or, "Can AviSynth add support for Photoshop, etc.?"

Or even, "Can someone help out Robin Rowe on CinePaint?"

CinePaint (formerly known as "Film GIMP", where GIMP stands for "GNU Image Manipulation Program") is an open-source, cross-platform image-manipulation program that basically can do Photoshop-like stuff (yawn), but is also designed to allow Photoshop-like stuff to huge series of frames from motion pictures (OK, I'm awake now!). Cool stuff, eh?

However, a couple of the problems with CinePaint are that it's not very robust on a Windows platform, and it's not being developed particularly quickly. Although it's on the list of things to come, CinePaint doesn't seem to be able to open raw video files, meaning that if you want to manually edit a random frame here and there (e.g. to remove dust specks), you've got to do a bunch of pre-processing (ImageWrite, for example) in order to get a bunch of individual frames to edit. That starts to take up a lot of disk space, especially since in most cases there aren't actually all that many frames that need touching up (still need to write out all frames).

[My ideal app would sorta work like ACDSee 2.43: scroll wheel to move through video quickly, and maybe a few brushes and tools to be able to either paint from a previous or successive frame onto the current frame, or from one part of the current frame to another part of the frame. Nothing major...]

For anyone who's still with me, I'm sorry if this comes across as not very AviSynth-related. I love AviSynth, and wish I could use it better. I also wish I could use it for more stuff! But one criticism I have is that a lot of people spend a lot of time writing complex algorithmic filters with switches and parameters that take a lot of time to tweak, just to get an approximation of better corrections that could be made much more quickly by hand, if the tools existed to make the process more intuitive.

Just two cents from an enthusiastic user (but non-coder)...

[Mug Funky]

any kind of manual touch-up sounds good to me. there are some things too hard for a computer to do, where you have to give up and throw a brain and a paintbrush at them.

painting in mattes manually for use in Overlay or MaskedMerge would be very cool indeed. clone brush and previous-frame-brush would also be awesome.

hmmm... i'm wondering if it's possible to do this with current stuff?

it'd be too cumbersome a process for photoshop, but this cineGimp thing sounds interesting if the interface can be made quick enough (ever use "Autodesk Animator" in the old 320x200 DOS days? i haven't seen much that can beat that interface for actually getting in and painting on frames.)

[edit]

just d/l'd the latest win32 binary of cinePaint i could find (a year old, sigh.)

veeeeeeeerry interesting. good to see there's HDR support in it, because HDRshop is a pain in the buttocks for anything but loading and saving 32 bit float images.

one crit - it crashes whenever i try to use the curves function.

[Fizick]

I also wonder why Cinepaint is not developed anymore.
But some times ago i try to use similar rotoscoping software (commercial) for my 8 mm film restoration. After about one month of handy works (almost every evening) i have got only 2 minute clip. :(
After that I learned C++ and I am Avisynth filter-writer now. :)
(DeSpot , DeScratch plugins...)

But probably some part of your wishes may be realized as Avisynth plugin:
1. Invoke ImageWrite function for every frame, save frame as BMP.
2. Call external graphic application such as Paint.exe, with image filename in command line. It may be done similar way as in Call plugin by Ddogg.
3. Draw in Paint over image file, save changed file, close Paint.exe.
4. Plugin get info about paint window closing.
5. Invoke ImageRead function to read changed file.
6. return frame to avisynth.
7. delete any unneeded BMP (may be it must be done first)
8. get next frame ...

May be this method will not work.
it is first what i have in my head.

[Karyudo]

Originally posted by Fizick
After about one month of [work rotoscoping 8mm film] i have got only 2 minute clip. :(

I can imagine that's about the toughest stuff to work from: pretty lame original material. Fizick, I think your plugins are great! Maybe especially for your difficult sources. But for sources that are quite clean, I could imagine that a part math, part manual labour solution would be the ideal.

1. Invoke ImageWrite function for every frame, save frame as BMP.

Hmm... that's the step I'd most like to avoid! Having a hard drive full of BMPs is not my idea of a good time.

I'd like to be able to load a frame into Photoshop (or something similar) using something like an AviSynth script. So when I open the "file", the frame I want is served to the application. When I save it, I don't mind that it's an actual BMP (or, even better, a PNG) on my hard drive, but until I save something, I don't want to have to mess with long lists of BMPs at the outset.

The rest of your listed steps for a proposed workflow seem pretty reasonable, Fizick, even if they don't seem all that fast.

Another idea would maybe be to be able to load a series of frames as a bunch of Photoshop layers, and be able to copy/draw/clone/etc. between layers?

I assume people are familiar with ACDSee? That app -- especially earlier versions, like the 2.43 Classic version I have -- is tiny, and FAST. It will almost "play" a series of still photos, it (pre-)loads them so quickly. It's super-easy to scroll backwards and forwards through a directory of image files. That's the sort of ease I'm hoping for. Plus, of course, I'd want to be able to stop on any given frame and do some simple touchups. I don't expect to be able to do wire removal, etc. That's CinePaint's territory, and more than I think most people (like most of us hobbyists) need. But a few simple brushes and clone tool options would be awesome.

I'm glad to see there is at least some discussion of this among names that I recognize as heavy hitters in the AviSynth plugin realm! At no point do I want to come across as demanding anything of the incredibly talented and altruistic individuals who have made AviSynth the super-powerful tool it is today. Things are happening quite nicely enough without my new line of thinking. I only offer up my ideas as "wouldn't it be cool?" stuff that maybe somebody could get inspired by. (BTW, I'm free for lots of alpha or beta testing!)

[Mug Funky]

i think this would need a dedicated program... maybe avisynth and cinepaint could be merged somehow.

the big problems i see are dealing with colourspace, and what to do with the files you're working on.. avisynth doesn't support layers, and doesn't do 32 bit float, etc.

ideally there'd be a way of loading the "avi" (or whatever video type), and saving the changes done to a temporary folder (rather like photoshop's history), with a final render-out at the end. this way you'd get to work through the files quickly, have minimal hard drive clutter, and possibly the option to frameserve straight out into an encoding application.

perhaps avisynth's scripting language could be extended to the point where manual retouching could be recorded and performed through script.

dunno... just dreaming now, but it certainly would be cool.

[hanfrunz]

my two cents:
a while ago i wrote a example filter using a GUI. So someone could write a filter with a GUI. There you have a button "copy to clipboard" or "save and open with photoshop" or maybe you set an in- and out- point and do a "export range to folder"
the filter can write a .ini file which stores the information what frames come from a external picture. example:

#frame nr filename
234 "c:\rotoscopedpics\00234.bmp"
323 "c:\rotoscopedpics\00323.bmp"

The filter just scans the frame nr and loads a pic if necessary.
So an examplescript could look like this:

avisource("myold8mmmaterial.avi")
converttorgb32()
GUIexportframesfilter("myold8mmmaterial.ini")

that should be easy to code, the only problem i see is that photoshop and other programms do only use RGB or is there a YUY2-paint app??

mmh i maybe start to code this filter today, should not be too hard to do...


hanfrunz

[Karyudo]

Originally posted by hanfrunz
[S]omeone could write a filter with a GUI. There you have a button "copy to clipboard" or "save and open with photoshop" or maybe you set an in- and out- point and do a "export range to folder"

Sounds cool! Sign me up for beta-testing.

A few UI touches that I think would be nice (don't know how possible they might be):

1. Once Photoshop (or whatever) is open, keep it open. So if you load one frame, PShop stays open when you close that frame and return to AviSynth. This sort of stuff won't be fast if you're always waiting for Photoshop to open on every frame.

2. Allow users to choose which photo-editing program they like. Photoshop is the granddaddy of them all, but it's also way overkill for most things. And in a lot of cases, I prefer Paintshop Pro.

3. It would be awesome to be able to have the option to copy certain frames to a paint app as different layers in the same file, rather than only as separate files. Possible?

the filter can write a .ini file which stores the information what frames come from a external picture. example:

#frame nr filename
234 "c:\rotoscopedpics\00234.bmp"
323 "c:\rotoscopedpics\00323.bmp"

The filter just scans the frame nr and loads a pic if necessary.
So an examplescript could look like this:

avisource("myold8mmmaterial.avi")
converttorgb32()
GUIexportframesfilter("myold8mmmaterial.ini")

Again, I like it! I like the idea of an .ini file, because you could conceivably make your own in something like Excel or Notepad.

I'd hope that you'd have the option of choosing which file format to export to (I like PNG; TIFF is nice for RGBA, etc.). Can't be too difficult, since ImageWriter already supports lots of options.


[T]he only problem i see is that photoshop and other programs only use RGB -- or is there a YUY2-paint app??

Hmm... Photoshop (among other programs, perhaps?) can do "Lab" colour, which is based on the CIE colour model, which I'm pretty sure means it is YUV planar. There's a "lightness" channel, and then "a" and "b" colour channels (hence the name "Lab"...), just like YUV.

If you go to save a Lab-mode picture, then the options given are various Photoshop-specific formats (e.g. PSD), Encapsulated PostScript (EPS), and TIFF. I'm thinking TIFF might be the best?? But I really don't know. I'd like to think I'm very good at finding out how to do things in a given app, but I'm not good at all at getting under the hood to know how to code/decode file formats.

Maybe it's only a quick, relatively-lossless conversion from YUY2 or YV12 to Lab?

Looking forward to your first efforts, hanfrunz!

[hanfrunz]

Sounds cool! Sign me up for beta-testing.

okay, done :)

1. Once Photoshop (or whatever) is open, keep it open. So if you load one frame, PShop stays open when you close that frame and return to AviSynth. This sort of stuff won't be fast if you're always waiting for Photoshop to open on every frame.

this should be possible if you open a new .bmp in photoshop it stays open and just opens a new window.


2. Allow users to choose which photo-editing program they like. Photoshop is the granddaddy of them all, but it's also way overkill for most things. And in a lot of cases, I prefer Paintshop Pro.

okay this could be set up in the .ini file
first line:
"c:\adobe\ps8\photoshop.exe"


3. It would be awesome to be able to have the option to copy certain frames to a paint app as different layers in the same file, rather than only as separate files. Possible?

mmmh maybe later... someday...


I'd hope that you'd have the option of choosing which file format to export to (I like PNG; TIFF is nice for RGBA, etc.). Can't be too difficult, since ImageWriter already supports lots of options.

i'm starting with bmp using the devil.dll (just try to understand imagereader() ) later versions should be able to chosse


Hmm... Photoshop (among other programs, perhaps?) can do "Lab" colour, which is based on the CIE colour model, which I'm pretty sure means it is YUV planar. There's a "lightness" channel, and then "a" and "b" colour channels (hence the name "Lab"...), just like YUV.

If you go to save a Lab-mode picture, then the options given are various Photoshop-specific formats (e.g. PSD), Encapsulated PostScript (EPS), and TIFF. I'm thinking TIFF might be the best?? But I really don't know. I'd like to think I'm very good at finding out how to do things in a given app, but I'm not good at all at getting under the hood to know how to code/decode file formats.

Maybe it's only a quick, relatively-lossless conversion from YUY2 or YV12 to Lab?

mmmh no i won't do that. LAB is not the right choice, because we have the same chromasubsampling problems. We would need a real YUV2 paint app, where you can edit each component seperately (Y,U and V)
I think RGB is the way to go! or maybe a RGB picture with YUV content... mmmh that could be an idea R=Y, G=U, B=V and then you edit the seperate channels in PS...


the GUI is nearly ready, i will hopefully add the import/export tomorrow...

hanfrunz

[Karyudo]

Originally posted by hanfrunz
LAB is not the right choice, because we have the same chroma subsampling problems. We would need a real YUV2 paint app, where you can edit each component seperately (Y,U and V).

Maybe I don't know quite what I'm talking about, but I'm pretty sure Photoshop can edit each of the channels separately in LAB! I can certainly see the three channels in the Channels viewer. Pretty confident you can draw in any of them, too. Including the main, composite channel, which would look pretty nutso if it were Y=R, U=B, V=G.


the GUI is nearly ready, i will hopefully add the import/export tomorrow...

Wow. That's quick. Nice work!

[Karyudo]

Did a little searching, and found this:

LeadTools Raster Imaging / Raster Imaging Pro (http://www.pds-site.com/LEAD/Whatnew/archive/LTRI13.htm#conversion)

Does that help in the slightest?? Maybe not...

[hanfrunz]

here it is:
version 0.1beta1 of GUIFrameSubstitutor

download here (http://www.geocities.com/hanfrunz/GFSubstitutor01b1.zip)

I will post the source code after my holiday!

Have fun with it.

hanfrunz

[Karyudo]

Some first test notes:

- must create a dummy .ini file first, else the GUI crashes: sorta flashes up some boxes ultra-fast, then VDubMod disappears.

- if you don't have the path to the photo editor exactly right, VDubMod sort of loads the frame size, then hangs. (Guess how I found out...)

- Photoshop opens correctly, and stays open when a photo is closed -- good!

- getting the right frame into Photoshop seems to be virtually impossible, because:

a) finding a frame and then pressing Export/Open Paint Program does nothing, until you go back to VDub and move to another frame, at which point the "another frame" is dumped to Photoshop!

b) once you've scrolled past a frame, VDub can't re-display it if you want to move back to that frame. The counter looks right, but the picture is frozen.

- not clear how to get edited frames back into video (not done yet, maybe?)

I realize this is all very early days yet, so my comments are purely to help debugging, etc. This technique is going to rock when it's all worked out!

[Mug Funky]

FWIW i think it's possible to synchronize photoshop with other applications through plugins. i know in the 3d world theres something (that i can't afford) that will allow you to change a texture in photoshop and the change will immediately appear in 3ds max. this was a couple years ago, so i'm a little misty on the details beyond that...

the gui thing sounds cool. my computer's busy atm, but i'll try it out once i get through a ripping session :)

btw, about the colour space thing... perhaps cinepaint could be modified to support 4:2:0 or 4:2:2 colourspaces? it doesn't seem outside the realms of possibility, but i imagine there'd be a LOT of coding to do to allow it... brushes and filters could be calculated from RGB to YV12 on the fly. that'd definitely be cool, even if a simple clone brush were allowed (like how photoshop does jack-all in 16 bit colour mode, but it's worth it for what it can do - clone, healing tool, patch tool and selections.)

of course, i'd imagine most stuff we care about enough to do manual adjustment would probably be in RGB to begin with (like cineon or radiance formats)

[esby]

an old post...


http://forum.doom9.org/showthread.php?s=&postid=356628#post356628

it saves to bmp actually,

but I think I could add the png extension if you need it.

the procedure is simple:

Parse your video frame by frame (or with the slider)

drop the frames you want to edit with enter

then edit these with any application,
overwriting each files.

and generate the avs script that will fuse the results...

You can substitute adjacent frames by using + and - keys too.

esby

[Karyudo]

First of all, let me say that LBKiller is great (even though it has a strange name...)! I played around with it this evening, and got excellent results in not much time. Thanks for pointing it out!

Originally posted by esby
[LBKiller] saves to BMP actually, but I think I could add the PNG extension if you need it.

PNG is nice because it is usually much smaller than a BMP, but still lossless. So, yes, please! If you're willing to add PNG, that would be great.

If you're also willing to consider some other changes, here are my comments:

1. A "touches.Clavier.English" file would be very helpful! My French isn't all that bad (for a Western Canadian) and I can read most of what's necessary, but I think there are still things I'm missing -- partly because I don't understand all the issues that prompted you to write LBKiller in the first place. I can help clean up any English translation.

2. Rather than make an AVS file directly, it would be cool to have an .ini file (or something like that) that can be changed after further editing, without having to change the whole AVS script again. This is the approach hanfrunz has taken, and in my opinion that seems like the way to go. It allows going back and fixing more frames, without having to rework the base AVS script at all. [EDIT: I think I now understand: you're doing the same thing by telling the user to use Import(Name_Of_Output_AVS_File) in another script.] By the way, your approach of loading in files is great! In LBKiller, it's super-cool to be able to see the results immediately.

3. Could something like Shift+Enter be added, that would dump a given frame to a BMP (PNG), but not replace it in the clip? This would be useful for dumping a clean frame that could be used to fix up a really messed-up one: cut out a nice little "patch" from the good frame, and then place it on the messed-up part of the replaced one.

4. Could something like Ctrl+Enter be added, that would dump and replace the current frame, and just dump the frames just before and just after the target frame? I find that I often use little patches from those two frames quite frequently to fix particularly troublesome flaws.

5. A few more leading digits in the numbered BMP file format would be helpful. Right now you've got five (0000x.bmp), but a full-length movie could run to several hundred thousand frames. How about making it seven digits (000000x.bmp)? Almost 10 million frames should be enough...

6. Would a before-and-after StackVertical-type display be possible? After fixing a frame, it's sort of nice to be able to see what it looked like before, just to make sure you've done all that's necessary.

7. Can you undo a frame replacement in LBKiller? That might be one of the nuances I don't understand from touches.Clavier.

8. It would be incredibly handy to be able to dump the frames to BMP/PNG, and have them also immediately open in a specified paint program for immediate editing. I found that I dumped a frame, then immediately went to Explorer to drag'n'drop the file into Photoshop (already running), where 5 seconds of editing fixed the problem. In fact, opening the file and then saving the fixed version generally took longer than actually fixing the frame. So any little tweak that can reduce this overhead would be great.

9. A radical idea: Would it be possible to somehow arrange for editing frames in situ? Pick a frame, hit Enter, it's replaced by a BMP/PNG that's editable with a pop-up set of tools inside LBKiller? Maybe a zoom tool (useful), a clone tool (very useful) and a few other things like that, in a Photoshop-style floating tool palette? I can't imagine anything better...

Right now, I see that there are three things that three people are working on: GUIFrameSubstitutor by hanfrunz, LBKiller by esby, and some serious interest in improving CinePaint by Mug Funky. Each has things that are exactly what I think you'd agree is needed by the eventual killer app: GUIFS has .ini files and opens the dumped BMP directly in a paint app; LBK has an awesome front end -- one that immediately replaces fixed frames; and CinePaint has some great tools for series of still frames that would be great (better than Photoshop, or Paintshop Pro) for using the frames before and after the messed-up frame to help fix the problems. If these three efforts are the three points on a triangle that surrounds the "perfect" app, then I hope we can eventually make the triangle smaller and smaller, converging on the best possible solution. Throw in some native YUV support at the paint level, and you've got yourself one hell of a kick-ass video retouching suite.

I wish I could contribute some coding effort instead of just trying out other peoples' hard work and making comments, but since I haven't already spent several years getting good at coding, I can't. I do want you all to know that none of my comments is meant as any sort of negative criticism; just constructive criticism. I'm sincerely impressed by all the effort so far, and I'm interested in helping test and suggesting improvements for the tools for as long as any of you are interested in doing the coding.

[Karyudo]

@esby:

I've used LBKiller for a while now, and have a couple of observations/problems.

1. I've opened a HuffYUV AVI that's almost 8GB. No problem. But LBKiller won't display any frames past Frame 11100. That's 7:24.0 into the 28:18.360 AVI. Any reason why not?

2. When I went to StackVertical the LBKilled version with the raw version, I got an error stating that the LBK version didn't have audio. What am I doing wrong? (To be honest, I haven't spent a lot of time yet trying to troubleshoot this myself).

I've fixed about 287 frames, and as BMPs that takes up 208MB. I'm about one-quarter of the way through the video. Which is itself just the first of three LD sides! This is why I think PNG would help me out...

Of those 287 frames, perhaps a dozen or more are frames that didn't need editing; they were just raw material that allowed me to fix frames around them. I'd like to be able to revert some of those frames -- or, better still, not have to include them in the AVS script in the first place. Possible? Or is that something for v1.7 or beyond??

This tool is great! I hope you and other people stay interested in making this idea go even further... (subtle bump...)

[esby]

@Karyudo

I'll look on the problem as soon I can. (my main computer being brand new,without any os or software right now)

For (I), as far i can remember I've worked by the past with 25 min eps of anime without any problem, except I used avs to load the huff file.
So far, I don't think it's a frame problem, maybe a size one, since it is relying on vfw mainly.

For (II), if you can be more precise on the nature of the problem, I'll probably be able to help. If the fixed version fails to load because one of your sources have audio and some not (bmp files can't have any indeed :roll:), just add a killaudio() before calling lbkiller() on it, and it should work. To keep the audio, you have two solutions:

* save the audio to a wav and mux later or when you want, using audiodub() & wavsource()
* make a 'pipe';
src = mysrc.killaudio().lbkiller()
src = audiodub(src,mysrc) # using the filtered video and the unfiltered audio)

I'll add the png option as soon I can, that should not be a problem, only the time to reinstall Windows & delphi6.

esby

[edit] I just saw your first thread, I'll reply later when I can,
for deleting substitution, whether it has been + ou -; entered, just use del one time or two times. (one time will be 'spotted'; second time , delete from the specific frame changes - 'spotted' is the state which scanning for scene change put the scene into.)

[hanfrunz]

Hello everyone,
i am back from holiday.
- getting the right frame into Photoshop seems to be virtually impossible, because: a) finding a frame and then pressing Export/Open Paint Program does nothing, until you go back to VDub and move to another frame, at which point the "another frame" is dumped to Photoshop! b) once you've scrolled past a frame, VDub can't re-display it if you want to move back to that frame. The counter looks right, but the picture is frozen. - not clear how to get edited frames back into video (not done yet, maybe?)

a) yes this is my main problem, because the gui cannot acces the videodata directly, only the GETFRAME method can do this :mad:
And i don't know if there is way that to threads can have acces to avisynth...

b) have you used SetMemoryMax(0) in your script? This will disable caching. The edited frames will be loaded automatically if the number is in the ini file and the bmp exists. If the file does not exist the "normal" frame is used.

The only solution i see is to code an external viewer/editor like esby's LBKiller. Or is there a player outside which can use plugins?
And then this app opens an ini-file and an avs-script with my filter inside, which substitutes the frames from the ini file (so both programms use one ini-file!)
So you always the the substitued frames if you play the avs-file with any player.

The GUI-filter could get a preview window, a ON/OFF switch etc.

I think i start writing a new little player...

hanfrunz

[Karyudo]

Originally posted by hanfrunz
The only solution i see is to code an external viewer/editor like esby's LBKiller.

Yeah, that's a pretty elegant solution esby's got!

Since esby has released LBKiller under the GPL, shouldn't there be some source code to look at? There should, shouldn't there, esby?? I don't know if that will be cool with any/everyone, but I'd rather see two talented programmers working on parts of the same app (a la AviSynth development itself) than on two separate programs.

In any case, I'm still up for some testing, whatever you guys come up with!

[esby]

source code uploaded, but beware it might not be very clean on some part,

esby

[hanfrunz]

mmh the source is not c++, but delphi. Esby could you add support for my ini-file, yourself? I will upload the source of my filter today. And i think i start my own little player, too. It should not be very difficult if i use directshow.

hanfrunz

[esby]

Hanfrunz:
I'll look over it, but probably not before the next week.
I'm supposed to do some important work for university before 1st september. -.

esby

[esby]

LBkiller version 17 upped.

- timestamp option ( more than 5 digits)
- png and jpg option added.

For the rest, that s gonan be next week probably.

esby

[esby]

mmm about support for your ini file...

What kind of support you want exactly?

* conversion and loading of ini file into lbkiller?
* exportation of lbkiller settings as ini files?
* seamless integration of the two application?
* another idea?

For what I tested of the filter,
It would not work with avs 2.55
it crashed vdub several times with avs 2.54.

I was loading from a div3 source
and exporting the image to paint shop pro.

I guess the crash could be coming from psp7

but I ain't sure.

esby

[Karyudo]

Given the choices, I vote for "seamless integration of the two applications"! Since both you guys (esby and hanfrunz) are basically working on a similar problem (although it sounds like you're working in two different languages), as an end user I'd like to see all the various advantages turn up in one application, with all the trouble spots mitigated as much as possible. I know -- I'm a bit of a dreamer. But I feel that should be the goal. Sort of like how all the various disparate VirtualDub offshoots (e.g. Nandub) sort of ended up being replaced by VDubMod, which does almost everything any one of the other offshoots ever did.

Maybe a good first step would be to start building in some cross-compatibility? Both LBK and GUIFS use simple text files to output more or less the same sort of information (primarily the input file, the frame number, and the image file that should replace the frame of the input file). Could you maybe agree between the two of you to use a common format for the .ini/.lbk file? Maybe there would need to be a bit more prefatory material (setting up parameters like number of leading zeros, if variable; some comments as to where the file was generated first; etc.), but basically a common plaintext file could be used for either LBK or GUIFS. Or am I out of my mind??

As for how LBK/GUIFS work together with AviSynth/VDubMod, how about this:

- LBKiller outputs an .ini file. The .lbk file is just about perfect, actually. (It's not clear that it's a text file until you open it as a text file! I also don't understand all the various columns, besides the obvious frame number and filename.)

- A new AviSynth filter is written, called RotoINI (or something cleverer or simpler) that is of the form:

RotoINI(string "lbk_filename.lbk/.ini", [other switches as necessary])

I would imagine that RotoINI() would be written to do behind the scenes all the stuff that the LBKiller-output AVS would do now, but transparently in the background.

As a user, all you'd have to do is:

1. Run LBKiller, fix a frame (or many frames), close.
2. Write an .avs along the lines of:

AVISource("my_video.avi")
RotoINI("my_video_lbkfixes.ini")

3. Open the .avs in VDubMod, as usual.

Now, when you make any changes to the .lbk/.ini file, they are automatically used the next time you open the .avs file. No muss, no fuss, no stray .avs files to be updated and imported with Import(). Just the user's .avs, and LBKiller's .lbk/.ini file.

That's my opinion, anyway...

By the way, I should have thanked you already, esby, for updating LBK! I'll be trying out the increased leading zeros capacity and the PNG output shortly.

I solved my AVI-opening problem, by writing a simple AVS. However, before I tried that, I discovered that opening the AVI by itself in LBK 1.7 is now even worse than in 1.6: the image is now black and white, upside down, and seemingly split into two fields side-by-side! This is repeatable with my 720x352, HuffYUV YUY2 AVI. See the screenshot...

http://www3.telus.net/11811/doom9/LBK_weirdness.jpg

Other than that, there are just a few UI things that to me seem non-standard, and which I'll note sometime soon for your consideration (or not).

I'm quite chuffed that there is now a reasonably-elegant way of rotoscoping changes to selected individual frames, using open source tools that are AviSynth-friendly! Kicks the ass of Photoshop's "filmstrip" mode, that's for sure...

[esby]

@Karyudo:

I'm not sure, but your gray lines might be coming from your huff codec configuration
check there http://esby.free.fr/tmpPic/huff.png

if you set the number below your the actual video resolution, the video will be decompressed as interlaced...

Which might happening here.

You could play with convertToRGB() too...
(and note that I'll be tempted to say use ConvertToRGB32() instead of ..RGB24())

On Another note:

for my vision of operation inside of lbkiller...

I did it because i was against the 'automatic filtering approach'.
I explain:
When you have problems in a video:
you have usually two choices:
- you get another source.
- you try to filter, and clean it with more or less success.

If you can get a better source, the problem is almost solved.

But if you need to do filtering,
there are question to the demarch you are going to use:
1- Are you going to use the same set of filter for the whole video?
2- Are you going to use different set of filter for different part?
3- Are you going crazy group of frame by group of frame, applying specific treatment to each of them?
4- Are you even crazier, checking almost each frames of the video?
5- you prefer to rely on an automatic approach of point 2 to 4. (maybe 5, ( like mf :D ? ) )

I did lbkiller to do 4 actually, and maybe 3, and maybe 5(3) 5(4)
depending on how you consider the question.

If you choose another solution, for example 5(*) using conditionnalFilter(), there are three inconvenients in my opinion:
- first, you rely on an automatic process, which can do errors.
- secondly, you might redo all the analysis each time you re-encode the video.
Of course, the second point might not be true, if you store the value externally, and check for their existence before you run the process, but i'm not sure of that for all filter.

For me, point I here is not acceptable. I prefers passing 6 hours to check 25 min of video beside to rely on an algo that will freezeframe each scene changes... because it has a cost on the animation especially if you freeze when it's not needed, and when you don't freeze when it was needed (example: a crappy frame on a static scene... undetected... encoded ... etc.)

Now about lbk.

My goal is to clean the source the maximum I can before encoding a release candidate.
Of course I do errors when checking, so I keep my source to fix the errors I could (and of course, this happends all the time) missed the first time.

If i see an error, all i have to do is to note the approximative timing on a notepad when playing it.
After, I go back with vdub & lbkiller (one instance of each), and check frame by frame to locate the exact frames causing problems,
and I try to replace them with the previous or next ones (using + or - in lbkiller).
If i am really forced, I'll drop them to bmp and edit with an image editor after.
Why after? because you are faster when you only have one work to do, not two.
Determining a list of frames, then editing these, is faster than determining a frame, editing it, determining another editing etc, because this work recquires a 'different concentration state'.

But I'll probably implement the launching of an external application with the image name as a parameter.

Generating ini files working with hanfrunz filter might be possible, without problem, but supposing they support the features used, and vice versa. (eg: + & - use freezeframe() and no intermediates images,
which I think hanzfrunz filter don't do actually.)
Of course adding the features might not be a problem.

esby

[Karyudo]

Originally posted by esby
I'm not sure, but your gray lines might be coming from your huff codec configuration

Hmm... I don't think so. I just upped that number to 480 (i.e. above my clip's height of 352) and lowered it to 288 (default), and it was the same both ways. The AVS route works fine. LBKiller 1.6 worked fine, too, until it hung up at a frame some 10 minutes or so into the clip.

No matter, really, as long as the AVS route is OK. But I thought it was odd that it was weird-looking now, because it used to work!

I [created LBKiller] because I was against the 'automatic filtering approach'.

I'm very happy to hear that, because I agree. There are many things that can be corrected well with automatic means (overall noise reduction, changing hue and saturation, cropping and scaling, etc.), but in many cases I think dust and scratch removal is not one of those things! I think Fizick and various other authors of dust removal filters have done a great job of providing the very best in automatic tools, but somehow I just can't trust that approach for a whole movie.

But if you need to do filtering,
there are question to the demarch you are going to use:

I had to look up "demarch", because if that is really an English word, it isn't very common at all. Looks like "demarche" is OK, having come from French (surprise, surprise) and meaning "course of action" or "approach". Useful!

4- Are you even crazier, checking almost each frame of the video?

Yes, and yes! I'm happy to hear you made LBKkiller to do 4, because that's how I want to use it, too.

For me, point 1 here is not acceptable. I prefer passing 6 hours to check 25 min of video than to rely on an algo [....]

Me, too! I guess we're both crazy.

My goal is to clean the source the maximum I can before encoding a release candidate.
Of course I do errors when checking, so I keep my source to fix the errors I could (and of course, this happens all the time) missed the first time.

Perfect! I like to do the same. That's why I'd like to load my raw AVI into LBKiller, do some fixing, and then write an AVS file that takes the source (.avi), the LBKiller list of fixed frames (.lbk or .ini), and the fixed frames (.png or .bmp), and then lets me manipulate the clean source as I would a perfect AVI. So I could load captures of each side of an LD separately into LBKiller, but then load them into an AVS independently:

side_1=AVISource("side_1.avi")
side_1=side_1.RotoINI("side_1_fixes.ini")side_1=side_1.<some_other_filters()>

side_2=AVISource("side_2.avi")
side_2=side_2.RotoINI("side_2_fixes.ini")side_2=side_2.<some_other_filters()>

side_3=AVISource("side_3.avi")
side_3=side_3.RotoINI("side_3_fixes.ini")side_3=side_3.<some_other_filters()>

whole=side_1+side_2+side_3

With this strategy, I could work on the clips separately, or on this script that compiles them. I could work on the <some_other_filters> part without being finished the LBKiller part, and I could still see what my original source was, just by commenting out the lines where the LBKiller .ini file is used. At any point I could open the raw AVI in LBK, make changes, and know that when I closed LBK and re-loaded my compilation file with this script, I'd have all the up-to-date LBK edits included.

If I am really forced, I'll drop [flawed frames] to bmp and edit [them] with an image editor after. Why after? Because you are faster when you only have one thing] to do, not two.

I understand your point about marking everything first, and then going back and doing the edits. That's certainly a time-efficient way of doing things, as you say, and for large flaws that works very well.

However (and this is a big consideration, I feel), some flaws are so subtle that only by comparison to frames before and after the frame in question does the flaw become apparent. For example, if I have only one frame of a sky full of stars, and the flaw is a tiny white speck that appears only for one frame, I'll never be able to remember which "star" in the frame saved out as an IMG file is actually a speck of dust. Sometimes it takes the shuttling back and forth through the video to be able to see the flaw, pinpoint it, and check that you've fixed the right thing. That's one of the reasons I'd like to be able to send frames directly to an image editing program.

(Also, sometimes I just want to see how well I'm doing, and feel "finished" on the fly!)

But I'll probably implement the launching of an external application with the image name as a parameter.

That would be excellent! Is it a lot of extra work to offer both behaviours (i.e. launching an external application immediately, or not)?

Generating ini files working with hanfrunz filter might be possible, without problem, but supposing they support the features used, and vice versa.

That's what I'd expect, too. If you have something in the .ini that you need, then hanfrunz's filter should either support that, or at the very least, ignore it and preserve it in the .ini. Likewise, if he adds something he needs, then LBK should either use it or ignore it, but not remove it.

It would be great if a common syntax could be worked out that supported all the major things both of you guys hope to support. You both use (and need!) the frame number, so maybe as long as each filter wrote the same stuff to the same line (indexed by frame number), you'd be fine. Like you'd need:

[frame number], [+ or -], [something else], [PNG filename]

and hanfrunz might need:

[frame number], [unique info], [PNG filename]

so you could agree that both programs would use the syntax:

[frame number], [+ or -], [unique info], [something else], [PNG filename]

If you used LBK first, it would write what is basically a CSV file that looked like:

[frame number], [+ or -], , [something else], [PNG filename]

hanfrunz's GUIFS would load this file without changing anything, and if you happened to modify one of the frames already changed with LBK, then GUIFS would write out:

[frame number], [+ or -], [unique info], [something else], [PNG filename]

If, on the other hand, you used hanfrunz's GUIFS first (or at least first for a given frame), it would write a CSV line that looked like:

[frame number], , [unique info], , [PNG filename]

...and likewise, LBK would preserve this info, too.

In that way, a user could use either (or both!) tools to help fix their clip, without having to keep the two tools from interfering with changes made by each other. That would be great progress! (At least until there is one killer application that can do everything from both apps...?)

I also like the fact that both of you are using plaintext files, so that even a non-programming idiot like I am can go in there with Notepad (or Excel, maybe) and edit things manually. For example, if I move all the files, or find I should have named them differently, I can use a batch file or search-and-replace or some Excel command to fix things how I like!

[Fizick]

Hi!
OI have one idea for discussion.
Recently one man ask me about implementing some restricted region filtering in Descratch plugin by some handy procession.
Later we concluded, that it is general problem.
As the man still not post here, I decided make post myself (to not forget).
I think, above problem must be considered as masking.
If we could interactively paint some mask on video frames,
it may be used with Masktools etc:
For example, for strong denoising (by Despot etc) in masked regions only, or on contrary, in non-masked. Or may be better with 2 masks (pure red - for despot (defect area), pure green - for not despot (protected area), and rest - as filter decides).
I think, i would be more simple and fast, than pure handy image restoration by painting (retouche).
Probably, it may be implemented in Lbkiller or Guifs. Or not?
Does anybody understand my night english?
:)

[morsa]

I understand you perfectly.
That is exactly what I was thinking about those times when I bother you with Restoreinpaint inpainting features....Do you remember :) ?
That is exactly the way it selects the areas to be inpainted...pure red (255)

[Fizick]

It may be and not red, but white ;)

[Fizick]

Addition.
This mask can be preliminary formed as a result of filter (Despot, Descratch) work.
This preliminary mask can be loaded to interactive plugin, and edited here. For example, we simply point to some pixel (change it) of spot to reject removing of whole this false (artifact) spot. Or use some other logic.
After editing, edited mask is returned to avisynth, and used for masking (or may be for second filtering).
It is only ideas for discussion.
:)

[Mug Funky]

using masks... seems like an efficient way of getting some good spot removal tools going with minimal fuss.

something like 4 brush modes:

-replace with last frame (with or without position offset)
-replace with offset position on current frame (clone brush)
-replace with next frame (with or without position offset)
-replace with filtered source of choice (maybe there could be several of these for different filter chains?)

with just these 4 modes i think you could handle almost any retouching job.

hmm... maybe another mode for rotoscoping which does an overlay of a different clip - like a manually drawn blue screen.

[morsa]

Mug,
Are you saying something simillar to this:

Given the mask, the filter then looks for the corresponding image portion for that Spot(?) on before and after frames, based on some image portions (blocks) around the Spot...

Is it something like that?

[Mug Funky]

i mean more a manual approach.

you paint a mask that is used for different things.. maybe in total 5 masks are made (perhaps denoted by shading a different colour, or simply selecting from a box), and they do different things.

you'll have a mask that replaces what's under it with the previous frame (rather like revealing a layer underneath in photoshop), one for the next, one that applies a clone brush (actually, just thinking about it now, it'd be a little more difficult because multiple spatial offsets would be involved with 1 clone-brushed frame), and one for masking between "filter of choice" and the source.

so a "cigarette burn" on film could be fixed by replacing it with the previous frame (manually), as could large spots. blemishes on characters' faces would be fixed with clone brush, and the rest would be filtered for badness, with any artefacts replaced by the unfiltered source. this could be very powerful, and faster than any 1 method used on it's own.

dammit, i wish i could program :)

[morsa]

Well, I was talking about the same :D
I manually paint the thing I want to remove, with the difference that instead of just copying the portion on same location from other place, I would let the filter find the correct one for me based on the sorroundings of what I chose..
Am I clearer now?

[Mug Funky]

AH! that's even better!

hehe.

we're talking the same language now, i think.

[Fizick]

Originally posted by Mug Funky
i mean more a manual approach.


I hate manual approach! :angry:

[esby]

I have more time now.

:)


@Karyudo:

I could not reproduce the bug you were talking of with 720*352 huff yuy2 output.(I used colorbars(720,352).convertToYuy2(), which I encoded to huffyuv...).

If you can upload somewhere one or two frames, maybe i could reproduce it, and thus, fix it.

@Mug funky: using masked replacement could be done.

If i ain't wrong, a possible implemenation would be:
1- select an area of the picture.
2- capture this area only and drop it to a bmp.
3- edit this area with an external tool.
4- load the area with overlay(), at the corresponding position.

I guess, that doing a masked replacement of previous or next frame, could be done too, (without edition, but with shifting, maybe scaling & zooming?).
But I need to think about the UI demarch for it.

And since these two approach mean that we can have several overlay on a same picture, that would mean more UI headaches probably, unless someone can think of something, simple & efficient in term of manipulation...


@Fizick:
If I understand you well,
If we define custom mask for each frames of the clip...
And allow to change it when it is needed, manually, by editing it, or by telling to use another one, via the UI.
the problems would be:
a- the size used by each different mask. (simple mask of two colors could be ok, but not complex one resulting of screenshot edition for example). Am I right about the subject? Are we talking of simple mask, or of another type of mask?
b- the global strategy to use, and how lbkiller could be of any use in it.
c- the possible interaction (or syntax , name of the given filter) to implement the features corresponding to c).

esby

[Karyudo]

@ esby:

PM sent with FTP details of sample clip!

[Fizick]

Esby,
Yes, I think about simple (binary) mask, for example produced initially by DeSpot filter, by subtract its output from from source.
Then we look at this mask and de-mark some of whole spots (artefact)by hand, for example by simple mouse click on spot.
Then some filter compare initial mask with returned mask, and conditionally restored (from source clip) regions of these false spots .
So we may use more strong initial setting for despot filter.
It is one of simplest semi-automatic method, of course.

[esby]

@Karyudo: I uploaded a new verson (17b).
In fact, I was able to reproduce it finally with my avs, I was just not opening the good file (the avi, not the avs used to load it. ^^)

Try with it, it should work now.
I fixed it, but since it has been ages now, I am not sure if it could not badly influence some other video loading, as far i tested, the answer is no, but I ain't sure. (Xvid loads correctly, div3 too, huff too etc. here).

@fizzick:
Could such binary mask you are talking of take advantages of the png compression?
Or do you have custom mask in mind, with custom procedure loading?

esby

[Karyudo]

Thanks, esby! It's cool you're making the time to do some development on LBKiller. I also noticed there's now an English 'Keyboard Shortcuts' file. Excellent...

Just so everybody knows, LBKiller can be downloaded from
esby's LBKiller page (http://esby.free.fr/prog/lbkiller/).

[esby]

I think I m going to add my www page on my signature, it will be just the simpler to do.

esby

PS: And it is not the LBkiller homepage, that is just a magic trick of my php script.

[Karyudo]

Originally posted by esby
I think I m going to add my www page on my signature...

Perfect! Thanks, esby. I had to go back to an earlier post in this thread that pointed to another thread where you listed the location of LBKiller. Took a little doing...

PS: And it is not the LBkiller homepage, that is just a magic trick of my php script.

I knew that -- I removed the '/lbkiller' part just to see where I ended up -- but I figured that subsection of your page was simpler to look at for those people interested specifically in LBKiller. Lots of other stuff there that is over my head!

[HighInBC]

There is a toolkit called ImageMagick, it is an api to do many things.

There are many interfaces to it, for C, and Perl, and even command line. It has a simple text interface and gives any information about the image you might want. All that would be needed would be an API for avisynth to apply a set of commands to a series of frames.

Then if someone was adventerous, they could design a paint program that worked through image magick and stored the commands as avisynth code snippets.

I currently do something similiar, I wrote a Perl script that does all the image magick routines and saves the png's and then gives me the avisynth code to use to get at it. But that makes for gigabytes of temporary files.

[esby]

@HighInBC:
Maybe that can help other people indeed :)
but I don't need that since I already have an API for opening avi files, extracting what I want etc.

Some good news now:

LBKiller v.18 out:
* GIMP support added ( http://www.gimp.org/ )
Gimp can now be used to edit captured frames.
* Menu parameter saving & loading added via an ini file.

I changed the User Interface a bit...
Now when using enter key, it can do (according to menu settings ):
- capture frame to an image file (png/jpg/bmp).
- edit a previously captured frame, supposed the file still exists.
- capture and edit.

two working mode can be used:
* Capturing frames... and editing them on the fly...
* Capturing all the needed frames, and editing these after, by using a new available menu.

esby

[Karyudo]

Originally posted by esby
LBKiller v.18 out:
* GIMP support added ( http://www.gimp.org/ )
Gimp can now be used to edit captured frames.


For those who don't use GIMP, you can put the path to your favourite image editing program in the LBKiller INI file instead. I replaced the default path to GIMP with 'C:\Program Files\Adobe\Photoshop CS\Photoshop.exe' and it works great!

Also, if your image editing program is already open (or kept open), the file will open in that instance, which saves a bunch of time.

* Menu parameter saving & loading added via an ini file.
- capture frame to an image file (png/jpg/bmp).


I meant to ask before, esby: What are the parameters for PNG and JPG compression? The defaults are '5' for PNG, and '80' for JPG. I was under the impression that PNG was lossless, so I'm sort of startled to see a compression setting there. I want the best picture quality, so how high (or low) can that parameter be?

Today is Canadian Thanksgiving, so one thing I'm thankful for is esby's excellent work in developing LBKiller -- especially according to some of the input users such as myself have made!

[esby]

PNG is lowless.
JPG is lossy.

Now the PNG indice is a compression factor,
that will affects time to compress and to decompress versus final size...
(a little like when you use different compression in zip archive...)

While the JPG one is a quality (or degradability) indice, meaning you will degrade the quality, and decrease the filesize.

So now if you find that you are png are decoding or encoding with too much time, reduce the compression factor...
If you use png, the quality will not be degraded... only the time to compress and decompress...

esby

PS: And... I added Gimp support, since I was able to get it working... people are free to use what ever program they have... as long they can parameter it correctly... the BeforeParamter & AfterParamter are here for that in fact, as I can't be sure than any program work the same...

[Karyudo]

Oh, good: I'm not wrong about PNG being lossless!

What's the range of values allowed for the PNG compression number? At least 1 to 5, I guess, but how high can it go?

[esby]

PNG range goes from 1 to 9
Usually letting it to 5 is fine on most computer...
After all it is just a user preference.

JPG range from 0 to 99 I think.

esby