Improving Foto-Camera "video"

[Sigma9]

Hi,

I've got a digital FOTO-Camera (HP850) which makes nice still fotos, however the "videos" it takes are, well, limited in quality (no complaint, however sometimes I am just too lazy to take my DV...)

Before trying myself to chain some avisysnth filters to work on that, I would like to know if someone else encountered that problem (shouldn't be sooo esoteric...) and already has a solution.

The camera takes NTSC-type videos (29.97 fr/sec, quarter NTSC-pixels, progressive) and stores in a motion-JPEG format.

What bothers me most is the noise in the video which I think derives from detector noise as well as a chroma sampling problem: I have the impression the camera takes the color planes one after another with significant delay - as a consequence, one gets color seams, especially during camera pans. (could post an example if needed...)

Any experience/hints out there?


/Sigma9

[Wilbert]

Maybe you can upload a short clip somewhere (25 frames)?

[Mug Funky]

hmmm. my camera does 320x240x15fps, at the stupid sample rate of 11024hz (4? 4???? way to be obtuse... i can't feed that into anything)

the chroma sampling is weird on mine, but i don't see a delay (and can't see where this would come from - CCD samples in RGB essentially, so it needs all colour info in order to make the luma plane)

i think the chroma sampling probs are from a too-hasty YV12 conversion in the processing circuitry.

btw, i have a Canon Powershot g3 (4MP).

[Sigma9]

The "delay" is just my guess - thats how it looks like. I will upload a short clip tomorrow - at the moment I am "creating" some web-space (3Mb needed)

As for the detector: yes, in normal foto condition RGB are sampled at the same time. However, in video mode that seems different - maybe due to the higher continuous data rate (btw - is there info on HOW the video is recorded in a foto machine?)

/Sigma9

[vidiot]

Originally posted by Sigma9

What bothers me most is the noise in the video which I think derives from detector noise as well as a chroma sampling problem: I have the impression the camera takes the color planes one after another with significant delay - as a consequence, one gets color seams, especially during camera pans. (could post an example if needed...)

Any experience/hints out there?

Sigma9,
i guess the noise is from the short exposure time.
Consider the size of your lens and the "Lichtstärke" (1:2,8 - 4,0?)
- sorry don´t know the english word for it.
So the CCD has to raise the electric power = noise (most at low light).
That is the reason for makin most photocameras only takin 10 to 20 fps.
My Canon V3 (S 230 in the US?) is takin movies at 15fps but with 640x480 pixels and i often leave my camcorder @home...

[Mug Funky]

CCD collects data (exposure), then at the end of exposure time it dumps it (serial!!) to the processing circuits.

video mode uses a far smaller number of cells on the CCD (otherwise you'd be waiting an awful long time for each frame), so processing times are negligible.

think of the speed a CCD must work in a DV cam (a 3-chip would be better as processing is far less complicated).

my G3 can burst-shoot at about 8fps if i push it... that's of course in bright light where shutter speed is fast enough. for 1600x1200 pics that is not bad (RAW CCD burst shots go at about 3fps, but they're 2272x1704 when decoded, so that's no surprise). this doesn't do compression on the fly though (it fills it's memory, then does the compression i think, but i'd be damned if the manual is going to tell me that)

CMOS cameras work a lot better than CCDs for speed though - each cell on a CMOS has it's own data collection dooby built onto the same chip. data dumping is done in parallel, and i've no idea where the processing goes (but the possibility exists for on-chip processing too). so in this case 320x240x15fps would be a snip.

[Richard Berg]

"Lichtstärke" -> "f-stop." The German word is much better :rolleyes:

[scharfis_brain]

f-stop = Lichtstaerke ?!?

can you explain why it is called f-stop instead of something more intuitive?

[Mug Funky]

i think the "f" denotes some variable for focal length.

it's not the sort of question easily entered into google though...

it's certainly proportional to focal length - low f-stop, low depth of field.

[Sigma9]

@vidiot:

I don't think the illumination time of the chip itself can be a problem: a normal still foto taken at the same (bright sunlight) conditions typically needs 1/100 s == 10ms. For 30fps, one has more than 30ms time per frame... I think the bottleneck is more the clock-out of the ccd pixel data (see below) and the time needed to software-treat and store the data (the latter is actually the reason for starting this thread: maybe the artefacts are sw-generated and thus there is a chance to fight them)

Another problem with the CCD is the following: the optical pathway in the camera is made to fill the whole chip of 2700 * 1200 pixel. So also for video you have to read out (serially!) the complete chip and only after that reduce the data. Again my feeling is that this reduction sacrifices quality for the sake of speed.

/Sigma9

[kassandro]

Originally posted by Sigma9
What bothers me most is the noise in the video which I think derives from detector noise as well as a chroma sampling problem: I have the impression the camera takes the color planes one after another with significant delay - as a consequence, one gets color seams, especially during camera pans. (could post an example if needed...)

I cannot believe this. All pixels are shot simultaneously and 1/3 of all pixels capture the red value, the next 1/3 of all pixels for green values and the rest for blue values. Thus it is not true that for each pixel the three basic color are captured sequentially. If there are color seams in video frames then there should also be color seams in photos. The quality deterioation is perhaps due to the very high frame rate. My Canon A70 makes 640x480x15 fps. The frames are fairly sharp, but the motion is often quite jerky. I haven't shot many such clips, because one really needs a tripod for good recrompression.

[scharfis_brain]

I haven't shot many such clips, because one really needs a tripod for good recrompression.

I have a Powershot A60, which makes crappy 320x240@15fps.

To polish those Videos, I am currently using deshaker to stabilize the clips, and motionperfect (or my home cooked genmotion-script) to interpolate the framerate up to 25fps.

[Sigma9]

@kassandro

Of course you are perfectly right for still fotos (1/3 of all pixels for each color plane). However, I am not so shure if this also holds for the video mode (for speed reasons). - OR - it is indeed a sw problem.




@Scharfi:
Deshaking and fps conversion - okay. But what about noise- or colour-artefacts: Do you have any? Do you have an idea/script helping against it?


Still waiting for my webspace to become active to upload my prob.


/Sigma9

[Mug Funky]

@ kassandro: i thought CCD matrices were not that simple... IIRC there's red and blue cells interleaved with green ones in a kind of tartan pattern. this is why processing is such a pain - it's got to sort out colour and luma with offset colours.

[Sigma9]

@Wilbert and others

okay here (http://perswww.kuleuven.ac.be/~u0010261/example.avi) is the video. It is 30fr == 1sec == 3.5Mb; huffyuv-AVI.

Please watch the almost horizontal structure above the ball, it shows my problem most clearly (in this case yellow/blue artefacts.

/Sigma9

[scharfis_brain]

your video has a true framerate of only 29.97/2 fps = 14.985fps

try to test this function:

loadplugin("c:\x\masktools.dll")

function repaircrappyvideo(clip i, bool show)
{
x=i.converttoyuy2()
y=x.greyscale()

edgeh=y.converttorgb32().generalconvolution(1,"0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 -1 -1 -1 -1 -1 0 0 0 0 0").converttoyv12()
edgeh=edgeh.binarize(128,upper=false,y=3,u=1,v=1).blur(0,1.58).blur(0,1.58).blur(0,1.58).blur(0,1.58).blur(0,1.58)
edgeh=edgeh.binarize(64,upper=false,y=3,u=1,v=1).converttoyuy2()

blurred=x.mergechroma(x.blur(0,1.58).blur(0,1.58).blur(0,1.58).temporalsoften(5,255,255))
repaired=overlay(x,blurred,mask=edgeh)

r=stackhorizontal(blurred .subtitle("smoothed chroma"), edgeh.subtitle("edgemask"))
s=stackhorizontal(repaired.subtitle("repaired clip"), x .subtitle("input clip"))

show ? stackvertical(r,s) : repaired
}


avisource("example.avi").duplicateframe(0).selectodd()
repaircrappyvideo(true)

repaircrappyvideo(true) will show you a before/after comparision
repaircrappyvideo(false) will only show the repaired video

I've seen this kind of (rainbowing) effect on normal video cams, and webcams, too!
It seems to come either from a low resoluted CCD, or from low resoluted readout of the CCD.
both things will end in something like "chroma subsampling without filtering"
(turn on Cleartype on your WinXP, take a screenshot and zoom in!)

[Sigma9]

@Scharfis Brain:

Tx for looking at it. How did you find out it has 14.985 fps? All my tools tell me it has 29.97..?

For the rest - I will test your suggestion tonight. I understand (as for now at least theoretically) most of your script. I assume the parameter values for BLUR are experience-optimized?

I'll be back.


/Sigma9

[scharfis_brain]

How did you find out it has 14.985 fps? All my tools tell me it has 29.97..?

Do never trust your Tools!
count the unique images yourself!
your video has actually 29.97 fps, but every unique frame is shown twice. thus it only haves 14.985 fps. that's why I used selectodd after loading the file.

I assume the parameter values for BLUR are experience-optimized?
not experience, just trail and error.

if you mean the value of 1.58, it is the max. value for blur()

[Mug Funky]

btw, there's no need for huffy samples - anybody who's got ffdshow will be able to decode the MJPEG that the camera uses.

[edit]
hmmm... that behaviour needs some more than blurring - it's asymmetric due to the lack of good processing in the camera.

yellow line = red cell and green cell show full brightness
cyan line = green cell and blue cell show full brightness

obviously there's been no attempt on the camera's part to work with it's own CCD matrix, to save time no doubt.

it looks as though luma wasn't extracted properly from a decoded RGB, but rather each cell was mapped to it's YUV values where they stood, leading to serious artefacting. i'm not sure if this can be easily fixed - most digital camera decoding software works with the raw CCD info, not a (badly) decoded form of it.

i wouldn't see it as a big problem though.

[Sigma9]

@Scharfi:

Tried out your script - works (after upgrading to AVS 2.54 ;=) )

And: it improves things quite a lot - color artefacts are much less (not completely gone, though). Right now I am tweaking parameters (which ones best?)

@mug funky:
tx - I thought so, too, wasnt completely sure, though. In order to play safe I made it huffyuv.


/Sigma9

[vidiot]

Originally posted by Sigma9

I don't think the illumination time of the chip itself can be a problem: a normal still foto taken at the same (bright sunlight) conditions typically needs 1/100 s == 10ms. For 30fps, one has more than 30ms time per frame...

I´m not sure about having 1/30 fraction of a second exposure time...
The pictures would be to blurry then.
My very personal comparison about cameras with a 15/fps mode and 30 fps(btw Mug Funky - your G3 is capable of 20 fps@ 320x240) comes to the conclusion that "real 30fps" photocameras introduce more noise.
(Compared Fuji 602 with my V3).

The post work on my videos is the same that scharfi does:
Deshaker (sometimes) Motionperfect (nearly everytime).

[scharfis_brain]

btw. I've tried fixing this chroma-problem (orange/cyan) on the pixel level, but this isn't possible anymore, because the Video has been converted to YV12. If it has been stayed in YUY2, a complete removal of this chroma problem would be possible.

But because it is YV12, you have to be satisfied with heavy 3D (spatial & temporal) smoothing on areas with horizontal detail.

btw. I dislike motionperfect sometimes, especially when extreme panning motion is present, or a static scene is influented by luma changes.

if motionperfect would have some motionmasking (like mocomp for morsa) has builin, it could deliver a rock solid instead of its swimming motion compensation.

[Sigma9]

@vidiot:
As Scharfi pointed out, my camera fakes 30fps by doubling each frame. Thus the real illumination time is 1/15 s which makes things more consistent with what you say.

@Scharfi:
I am already quite pleased with the improvement - none of the GUI progs I know of could even do a fraction of it. And - finally it's a foto camera, so don't expect the impossible (although I'd like to get close ;=) ) - I am still tweaking...

Tx so far


/Sigma9

[Mug Funky]

your G3 is capable of 20 fps@ 320x240

plz explain how? i've only ever got 15fps out of it, though i must admit i haven't used the firmware upgrade available for it. it was hard enough getting it to talk to my computer, and i never figured out what solved it in the end. i don't want to screw it all up with a firmware upgrade :O (and what's the deal with sound at 11024Hz? that's such a PITA)

hmm... if anybody wants perfect motion out of one of these babies, i suggest stop motion.

[Sigma9]

Firmware upgrade???

I did not even consider this possible - where would I look for my HP850?
Maybe this could also improve video quality...

/Sigma9

[Mug Funky]

don't count on it.

i found it through google, though it was on the canon site and i should have found it through there (google saves a buttload of time, i must say. you can even find obscure postal codes with it:))

the upgrades are normally bugfixes. i think the g3 one is.

[vidiot]

Originally posted by Mug Funky
plz explain how? i've only ever got 15fps out of it, though i must admit i haven't used the firmware upgrade available for it.
;)
I was pretty sure about 20fps, but maybe it was our A40 Modell?
Because in our company we have the G2 /G3 / G4 and all take MotionPictureMovies or whatever they call it @20 fps.
EDIT: Seemed that I dreamed this...look below...

The crazy thing is that their smallest cams take 640x480 pixel movies.
But as i mentioned before @15 fps.
So far (til last year)Fuji 602 which we also have in the company - was the only cam with 640x480 30fps and endless (Your CF Card is the border...)
Now with Samsung and many others this behavior is typical and Canon sell also a photocamera with the same feature - you can even zoom
while capturing!!!

After Research I found out that the first DIGITAL IXUS could make movies with 640 x 480 @20 fps but with 7 sec. duration...
Sorry for confusing you...

[scharfis_brain]

to complete the list of camera weirdness:
the sony dsc cams (the ones with MPEG-VX) are capping at a very odd framerate.
the manual says 16 fps.

but the real world is much more crappy.
it is much more jerky than 15 fps or 12,5 fps.
sony creates a MPEG-1 stream that has 25fps.
but the stream is only filled to 2/3 with unique frames.
this means every 3rd frame gets dropped by the cam.
so the produced mpeg stream is a I-Frame only Stream, but
every drop is coded as a blank P-Frame (no change)
the sequence shows up like this
iipiipiipiipiipiipiipiipiipiipiipiipiipiipiip...

fixing this will need several steps using avisynth, vdub and motionperfect.

I really like the german mis-pronounciation of Sony: "So Nie"


EDIT:

1)does someone know, whether there exist a hacked firmware for the Canon A60 supporting unlimited lenght of the movie?

2)does someone know, how I can abuse my A60 as movieplayer for other clips, that I uploaded to my CF card?
All tries failed because the cam seems to store a checksum or some other file information, that I cannot reproduce...

[vidiot]

scharfis_brain,
afaik there is no hacked camera OS available for canon cameras.
I guess the canons have 32 Mb of Memory internal which they use for capturing or fast mutli pictures.
But the cameras are also able to save to memory while caturing,
because the 3 minute movies with 320x240 pixels are ~52 Mb.

So it is possible maybe...

Just look for news about it here:
http://www.digitalkamera.de/
http://www.ixus-world.de
(sorry - both german only)

or one of the best sides for all kind of digital cameras:
http://www.steves-digicams.com/

[Sigma9]

Just for the record: My HP850 can zoom during video recording also.
The max length of the takes - well, this is also weird: the manual states 30s which is not true - it can be more. How much more, though, is kind of erratic: at some point the camera just stops recording and enters a "processing" mode which takes quite long. The max length is dependent on the free space on the memory card.

So my guess is: the cam first uses the card memory as a buffer for raw image data, once this is full it starts the compression work (in which the artefacts are introduced as well). Maybe there is a trick to have the camera NOT do this step and access the raw data on the card? Disadvantage would be the card is full with, say, 1min of video; advantage, though, it is raw data without artefacts.

/Sigma9

[scharfis_brain]

my canon A60 has all controls freezed during capturing.
only the stop button works :rolleyes:

(sorry - both german only)
Och, Deutsch ist doch supi :)