bad compression artifacts

[mathmax]

hi

I have a poorly encoded clip that comes from a streaming source. I try to get rid of the compression artifacts. Blocking and also some noise/smudge.
As the resolution is very low, the size of the noise becomes big when I upscale the clip. I'm not sure I can get rid of it without removing more details.. moreover it's a bit delicate to remove noise whereas the video is generally over smoothed... :-/
I came across these solution... I don't really know which is the best, and I would like your advices on this issue:

- I tried the solution explained here:
http://forum.doom9.org/showthread.php?p=1386559#post1386559

with dfttest:

dfttest (sigma=2, lsb=true)
DitherPost ()

with dfttest + motion compensation (maybe the zzz_denoise could be adapted?):
http://forum.doom9.org/showthread.php?p=1390594#post1390594

without denoiser:

GradFun3 ()
=> this last one almost doesn't have any effect..

- I also found this plugin:
http://compression.ru/video/deblocking/smartdeblocking_en.html

Which solution is the best in your opinion? I'm interested in any other suggestion :)

[Keiyakusha]

These are some of my thoughts about your topic. Not really helpful but probably maximum that is possible without actually seeing your source material.

The goal of the project by your 1st link is not restoring bad sources, but kind of "processing ordinary sources in a good way". I mean by only using what is described there you won't fix your video. You should fix it _before_ or _in the same time_. But only using what is described there is not a solution for you.
GradFun3 or some similar filters does wonderful effect but only in places when it is needed.
Deblocking filter on your last link is a something I personally would never use. Deblock from Manao usually works well.

[StainlessS]

@Mathmax,

If you also use DGDecode.dll (Mpeg2Source or autoloaded), then you may need to use "Deblock_Deblock()" to use the
correct plugin by Manao & Fizick.

[mathmax]

thank you for your answers :)
of course it might be difficult to advice without seeing the source..

here an example of the noise and smudges:
http://img4.imageshack.us/img4/6841/0000kk.png

and here you can see that the background needs to be smoothed, either by deblocking or debending..:
http://img850.imageshack.us/img850/47/20000c.png

a short video:
http://www.mediafire.com/file/mbboufsr4c5mmaw/sample2.avi

[MatLz]

@Mathmax,

If you also use DGDecode.dll (Mpeg2Source or autoloaded), then you may need to use "Deblock_Deblock()" to use the
correct plugin by Manao & Fizick.
Deblock() is the same in deblock or dgdecode dlls.
Exept that with Groucho2004 moded build, it is a lot faster.

[mathmax]

I wonder if deblock() is really the right filter to use here.. :-/

[StainlessS]

From DGDecode v1.5.8 docs:

Manao's H.264 Deblocking Filter. (v0.9.5)
Requires YV12 input.

From DeBlock (Manao & Fizick) v1.2

requirements: YV12, YUY2 Colorspace

I knew there was a difference, thought it was a better algorithm, but was colorspace instead.
Was not aware of groucho2004 moded build, thankyou. :)

EDIT:
I wonder if deblock() is really the right filter to use here.. :-/

There is blocking there too, but there does not seem a lot left to correct.

[MatLz]

Ah yeah there are differences, you are right.
Didn't notice because I mainly work with yv12.
But I confirm, even if the versions differ, that outputs are bit identical.
You can get it at hank315 site.

[StainlessS]

MatLz, does groucho2004 mod do YUY2 ?

[MatLz]

MatLz, does groucho2004 mod do YUY2 ?Nope.

[mathmax]

There is blocking there too, but there does not seem a lot left to correct.
so.. what would you advice to improve my source regarding these two problems (noise and blocking/bending)?

[StainlessS]

I meant "not a lot left to correct of the clip", DeBlock would not make a great deal of noticable difference.
As for the blending, what can one tell from a still frame.
Best I can suggest is to find a better source. Try a similar name to the sourced clip in google,
not an Avisynth type answer but it's the best chance of getting a better result.

EDIT: I dont know, maybe I might try VagueDenoise but I would not hope too much If I were you.
(Spacial smoother).

[mathmax]

I have two sources of the same video... both are streaming sources, both are badly compressed.. :(
but maybe it's possible to use them both (by comparing frames for instance) in order to get rid of some artifacts?

[Didée]

Way too bad to improve.

Though, resizing to 160x120 will make a lot of artifacts go away.

[Groucho2004]

MatLz, does groucho2004 mod do YUY2 ?

Here (http://www.mediafire.com/?8dsrz09zdn0bwhd) is a DeBlock.dll (v1.2) compiled with ICL 10.1, PGO and IPO. It's about 30% ~ 40% faster (on my SB 2500K) than the DLL from avisynth.org.ru.

[MatLz]

Nice work (again :D ) Groucho2004.
On my core2, it is 21% faster.
However, 1% slower than the dgdecode one.

[mathmax]

Look at this

source1:
http://img849.imageshack.us/img849/7676/source10000.png

source2:
http://img94.imageshack.us/img94/3117/source20000.png

blend of source1 + source2:
http://img812.imageshack.us/img812/6475/blend0000.png

I thought it might be good to blend the 2 sources.. cause the compression artifacts are different on each, and the blending kinda "averages" them.
Maybe there is a smarter way to get rid of the artifacts using the two sources.. what do you think?

[MatLz]

Just look at your sample and wwooaah, I do not know where to begin...
There also are a lot of missing frames and some dups.
If I were you, I would not waste my time trying to revive a dead horse.
Instead, I would "hide" all this mess with :
addgrain(150)

[StainlessS]

@Groucho2004,

Thankyou kind sir, I'm expecting great things from this.

@mathmax

Did not view them side-by-side but I thought, "much of a muchness", not too different.
It could though have some +ve impact. Exblend() recovers two copies of lost frames
from blended frames and they are usually not brilliant, but merged together are well
acceptable. Try it, if your happy, then so am I.


EDIT:- I dont mean try ExBlend(), that will NOT help, I mean try to blend the clips
in whatever way you choose.

[mathmax]

There also are a lot of missing frames and some dups.
that's why I want to use the two sources and combine them to recover as many frames as possible...


Instead, I would "hide" all this mess with :
addgrain(150)

impossible.. cause I want to use this source to complete a HQ video. Unfortunately the HQ version has a part missing, and I only have this bad source to complete. I need to make them match as much as I can... not an easy task
:-/

Else, do you think blending the similar frames of the two versions can help?

[StainlessS]

I hope that you tried Didée's fix, if anyone knows, he might.
Perhaps, size down as per Didée, then size up, with something
clever (eg Neddi) then after perhaps another foolhardy
attempt to get better quality, add grain.

[mathmax]

@Groucho2004,

Thankyou kind sir, I'm expecting great things from this.

@mathmax

Did not view them side-by-side but I thought, "much of a muchness", not too different.
It could though have some +ve impact. Exblend() recovers two copies of lost frames
from blended frames and they are usually not brilliant, but merged together are well
acceptable. Try it, if your happy, then so am I.


EDIT:- I dont mean try ExBlend(), that will NOT help, I mean try to blend the clips
in whatever way you choose.
thank you.. I thought I could blend them using a mask. The mask would select the less grainy parts of each video - i.e black on the areas where the first video is more grainy than the second video and white on the areas where the second video is more grainy than the first video.
But..how can I do that?

[mathmax]

I hope that you tried Didée's fix, if anyone knows, he might.
Perhaps, size down as per Didée, then size up, with something
clever (eg Neddi) then after perhaps another foolhardy
attempt to get better quality, add grain.
but I'll lose many details doing this... :-/

[StainlessS]

Sorry, but you lost me when you said 'many details'.

EDIT: try it!

EDIT: also, if you want to mess with the colors you would want to crop off
the border area so as not to confuse the manul (via histograms) or
auto correction via frame content (including border area, not wanted),
[eg ColorYUV(autogain="true") ]might look better but not by much.

[mathmax]

Sorry, but you lost me when you said 'many details'.

EDIT: try it!
well... :D I don't have much to lose in fact... but the video is already in a poor resolution... that sounds a bit crazy to downscale it..

[MatLz]

@mathmax
Do you mean you HAVE the missing frames in the other clip ?

@Groucho2004
Arf... an other % lost.

[mathmax]

@mathmax
Do you mean you HAVE the missing frames in the other clip ?

@Groucho2004
Arf... an other % lost.

I have two clips from streaming sources. Both are variable framerate and have different frames sometimes.. I want to gather them to get a higher framerate clip.

I also have an HQ video, but not full.. I need to complete the missing part of the concert with the two preceding clips.

[StainlessS]

Oh if only we lived in the days of "Bladerunner" then we could take snake scales
and by magic turn it into a hidef images, but then I'm probably dreaming,
(about a lamb and a robot, what the hell)

[mathmax]

EDIT: also, if you want to mess with the colors you would want to crop off
the border area so as not to confuse the manul (via histograms) or
auto correction via frame content (including border area, not wanted),
[eg ColorYUV(autogain="true") ]might look better but not by much.
I think the colors are ok, they match pretty well my HQ clip.

How would you create a mask that represent the difference of grain/noise between the two clips?

[StainlessS]

How would you create a mask that represent the difference of grain/noise between the two clips?

I'm not bright enough to do that, if it were warrented.
If you can merge two similar clips, that is the best chance
you have got to anything resembling a success.

In about 2007 to 2009 I spent an awfull long time trying to recover
a single clip. I eventually dug out a compiler and started coding
which I had not done for about 15 years. the clip that propted that was bad but not even in the same universe as your clip.
I would have given up way before.
I'm not trying to disuade you from trying, give it what you've got,and if you get it good, then tell us all how you did it.
I'm just not sure that there is much to be gained from it.
I understand, that it is important to you (go -on, tell us,
you're the big ugly geezer in the middle arn't you)

wish you the best of luck,
come back a tell us if you win.

ssS

[MatLz]

I have two clips from streaming sources. Both are variable framerate and have different frames sometimes.. I want to gather them to get a higher framerate clip.

I also have an HQ video, but not full.. I need to complete the missing part of the concert with the two preceding clips.
So you certainly will have to manualy insert the missing frames.
Easy, but will take a long time if this part is long.
Begin by deleting all the dups in the two clips.
Then do an insane usage of "trim" and "+".

For the filtering, really, don't expect to get "HQ" quality...
If Didée did suggest what he did suggest, there is a reason :
Avisynth can do magic, but can't do miracles.