Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th April 2013, 05:24   #261  |  Link
06_taro
soy sauce buyer
 
Join Date: Mar 2010
Location: United Kingdom
Posts: 164
A simple function to scan the whole video and output a log file, in which out-of-tv-range frames are logged.

PHP Code:
Function OutRange(clip cfloat "thr"int "plane"string "log")
{
    
plane = Default(plane0)
    
thr   String(Default(thr0))
    
Yeval "global ymax = YPlaneMax("   thr """)
             global ymin = YPlaneMin(""" 
thr ")"
    
Ueval "global umax = UPlaneMax("   thr """)
             global umin = UPlaneMin(""" 
thr ")"
    
Veval "global vmax = VPlaneMax("   thr """)
             global vmin = VPlaneMin(""" 
thr ")"
    
expe plane == Yeval
    
\    : plane == Ueval
    
\    : plane == Veval
    
\    : plane == Ueval Chr(10) + Veval
    
\    : plane == Yeval Chr(10) + Ueval Chr(10) + Veval
    
\    : AssertFalse"""OutRange: "plane"(0~4) value error!""" )
    
expy "ymax > 235 || ymin < 16"
    
expu "umax > 240 || umin < 16"
    
expv "vmax > 240 || vmin < 16"
    
expc plane == expy
    
\    : plane == expu
    
\    : plane == expv
    
\    : plane == expu " || " expv
    
\    : plane == expy " || " expu " || " expv
    
\    : AssertFalse"""OutRange: "plane"(0~4) value error!""" )
    
wriy """ "YMax " + String(ymax) + "YMin " + String(ymin) """
    
wriu """ "UMax " + String(umax) + "UMin " + String(Umin) """
    
wriv """ "VMax " + String(vmax) + "VMin " + String(vmin) """
    
expw plane == wriy
    
\    : plane == wriu
    
\    : plane == wriv
    
\    : plane == wriu """ + "" + """ wriv
    
\    : plane == wriy """ + "" + """ wriu """ + "" + """ wriv
    
\    : AssertFalse"""OutRange: "plane"(0~4) value error!""" )

    
c
    FrameEvaluate
(expe)
    
WriteFileIf(Default(log"OutRange.log"),
    \           
expc,
    \           
"current_frame"""" "" """expw)

thr: A percentage on how many percent of the pixels are allowed above or below minimum. Optional and defaults to 0.
plane: Which plane to check, same as mvtools 2 - 0:Luma; 1:Chroma U; 2:Chroma V; 3:Both chromas; 4:All. Optional and defaults to 0.
log: The filename of output log. Optional and defaults to "OutRange.log"

Example:
PHP Code:
AVISource("source.avi")
OutRange(00log="sample.log"
use avs2avi/FFmpeg/x264 or anything to run it, and you'll find the log file named "sample.log".
Original frame is returned untouched, so you can add any other filters after it or even use it in encoding as only a check for assurance.
Scene-change not implemented, so only frame-based detection, no scene-based detection.

Or if you don't even want to manually override false detection, you can simply use ConditionalFilter to do frame level range-adaptive filtering:
PHP Code:
AVISource("source.avi")
tv f3kdb(keep_tv_range=True)
pc f3kdb(keep_tv_range=False)
ConditionalFilter(pctv"YPlaneMax > 235 || YPlaneMin < 16""==""True"

Last edited by 06_taro; 20th April 2013 at 05:49.
06_taro is offline   Reply With Quote
Old 20th April 2013, 11:41   #262  |  Link
x265
Registered User
 
Join Date: Oct 2012
Posts: 176
http://www.mediafire.com/?y4zndt3cf34fm4ca

Could you analyze the log and tell me whether i should set tv-range=true or false?

Code:
mpeg2source("C:\Users\Abhijith Nair\Desktop\Anime\Vol1..d2v")
OutRange(log="sample.log")

Last edited by x265; 20th April 2013 at 12:21.
x265 is offline   Reply With Quote
Old 20th April 2013, 14:34   #263  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by 06_taro View Post
A simple function to scan the whole video and output a log file, in which out-of-tv-range frames are logged.
A neatly written function, especially the way the processing of plane is all done at compile-time (by building different expressions to be evaluated at run-time).

For information, a couple of small points on the following code:
Quote:
Code:
    Yeval = "global ymax = YPlaneMax("   + thr + """)
             global ymin = YPlaneMin(""" + thr + ")"
    Ueval = "global umax = UPlaneMax("   + thr + """)
             global umin = UPlaneMin(""" + thr + ")"
    Veval = "global vmax = VPlaneMax("   + thr + """)
             global vmin = VPlaneMin(""" + thr + ")"
- The variables ymax, etc, don't actually need to be global, as they are only set and used at runtime, where they have script-level scope.
- Triple quotes are not needed in order to include a newline in a string (only needed when the string itself contains quote characters).

So the code could (if you like) be written simply as:
Code:
    Yeval = "ymax = YPlaneMax(" + thr + ")
             ymin = YPlaneMin(" + thr + ")"
    Ueval = "umax = UPlaneMax(" + thr + ")
             umin = UPlaneMin(" + thr + ")"
    Veval = "vmax = VPlaneMax(" + thr + ")
             vmin = VPlaneMin(" + thr + ")"
I suppose you could also do the plane range-check once at the start rather than repeating it each time plane is used.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 20th April 2013, 15:41   #264  |  Link
06_taro
soy sauce buyer
 
Join Date: Mar 2010
Location: United Kingdom
Posts: 164
Quote:
Originally Posted by x265 View Post
http://www.mediafire.com/?y4zndt3cf34fm4ca

Could you analyze the log and tell me whether i should set tv-range=true or false?

Code:
mpeg2source("C:\Users\Abhijith Nair\Desktop\Anime\Vol1..d2v")
OutRange(log="sample.log")
At lease those 0-255 frames are clearly PC range, if not badly edited animations I've seen somewhere with PC-range credit overlayed on TV-range background....

It is suggested to manually check those 14-236 frames or somewhat similar, as those range flows might be resulted from encoding artefacts but not editing, hence might still be TV-range. Anyway, in either cases, I couldn't say it need keep_tv_range=True, some flows from grain generator shouldn't be that harmful to a source which already contains over/under-flows.

Quote:
Originally Posted by Gavino View Post
A neatly written function, especially the way the processing of plane is all done at compile-time (by building different expressions to be evaluated at run-time).

For information, a couple of small points on the following code:

- The variables ymax, etc, don't actually need to be global, as they are only set and used at runtime, where they have script-level scope.
- Triple quotes are not needed in order to include a newline in a string (only needed when the string itself contains quote characters).

So the code could (if you like) be written simply as:
Code:
    Yeval = "ymax = YPlaneMax(" + thr + ")
             ymin = YPlaneMin(" + thr + ")"
    Ueval = "umax = UPlaneMax(" + thr + ")
             umin = UPlaneMin(" + thr + ")"
    Veval = "vmax = VPlaneMax(" + thr + ")
             vmin = VPlaneMin(" + thr + ")"
I suppose you could also do the plane range-check once at the start rather than repeating it each time plane is used.
Thanks for those suggestions. My quick written script without any double check suffered some dirty formats.

Updated here:
PHP Code:
Function OutRange(clip cfloat "thr"int "plane"string "log")
{
    
plane = Default(plane0)
    
thr   String(Default(thr0))
    
Assertplane >= && plane <= 4"""OutRange: "plane"(0~4) value error!""" )

    
Yeval "ymax = YPlaneMax(" thr ")
             ymin = YPlaneMin(" 
thr ")"
    
Ueval "umax = UPlaneMax(" thr ")
             umin = UPlaneMin(" 
thr ")"
    
Veval "vmax = VPlaneMax(" thr ")
             vmin = VPlaneMin(" 
thr ")"
    
expe plane == Yeval
    
\    : plane == Ueval
    
\    : plane == Veval
    
\    : plane == Ueval Chr(10) + Veval
    
\    :              Yeval Chr(10) + Ueval Chr(10) + Veval

    expy 
"ymax > 235 || ymin < 16"
    
expu "umax > 240 || umin < 16"
    
expv "vmax > 240 || vmin < 16"
    
expc plane == expy
    
\    : plane == expu
    
\    : plane == expv
    
\    : plane == expu " || " expv
    
\    :              expy " || " expu " || " expv

    wriy 
""" "YMax " + String(ymax) + "YMin " + String(ymin) """
    
wriu """ "UMax " + String(umax) + "UMin " + String(Umin) """
    
wriv """ "VMax " + String(vmax) + "VMin " + String(vmin) """
    
expw plane == wriy
    
\    : plane == wriu
    
\    : plane == wriv
    
\    : plane == wriu """ + "" + """ wriv
    
\    :              wriy """ + "" + """ wriu """ + "" + """ wriv

    c
    FrameEvaluate
(expe)
    
WriteFileIf(Default(log"OutRange.log"),
    \           
expc,
    \           
"current_frame"""" "" """expw)

Actually I personally still prefer triple quotes with line wrapping, to keep a good habit for some other language users like python
06_taro is offline   Reply With Quote
Old 21st April 2013, 16:53   #265  |  Link
x265
Registered User
 
Join Date: Oct 2012
Posts: 176
How do i find a good value for Y Cb Cr?
x265 is offline   Reply With Quote
Old 22nd April 2013, 14:26   #266  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by x265 View Post
How do i find a good value for Y Cb Cr?
You need to find a balance between detail retention and debanding effect by your eye. You can use Histogram("luma") to make it easier to check effect of the filter.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 9th June 2013, 08:43   #267  |  Link
Yellow_
Registered User
 
Join Date: Sep 2009
Posts: 378
Did anyone work on a 16bit image sequence to Stacked LSB/MSB mentioned a few pages back? Hoping to find a method to import 16bit tif's and png's.
Yellow_ is offline   Reply With Quote
Old 10th June 2013, 13:22   #268  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
I tested 16 bit png with VapourSynth and it worked just fine. I used vsimagereader for that - maybe you can poke Chikuzen about tiff support.
sneaker_ger is offline   Reply With Quote
Old 10th June 2013, 14:21   #269  |  Link
Yellow_
Registered User
 
Join Date: Sep 2009
Posts: 378
sneaker_ger, thanks for the reply, I'm happy with 16bit png so no probs. Currently using Avisynth with Dither Tools, http://forum.doom9.org/showthread.ph...96#post1632396 but I should really get back into pursuing VapourSynth.
Yellow_ is offline   Reply With Quote
Old 2nd August 2013, 10:13   #270  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
gcc support:
https://github.com/dubhater/flash3ky...nd/commits/gcc

I just hit it with a hammer until it worked, so many of those changes are probably Doing It Wrong™. I'm pretty sure I broke the SSE code paths with those "#if 1". The C code seems to work fine though. No idea if msvc or icl still compile it.

Criticism and suggestions welcome.
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 2nd August 2013, 12:30   #271  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by jackoneill View Post
gcc support:
https://github.com/dubhater/flash3ky...nd/commits/gcc

I just hit it with a hammer until it worked, so many of those changes are probably Doing It Wrong™. I'm pretty sure I broke the SSE code paths with those "#if 1". The C code seems to work fine though. No idea if msvc or icl still compile it.

Criticism and suggestions welcome.
Thanks! Just had a quick look, I think your aligned buffer declaration won't work with VS, IIRC __declspec(align(*)) must be put before type name.

Will look into it if I can get some free time in the coming weekend...
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 2nd August 2013, 13:00   #272  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Quote:
Originally Posted by SAPikachu View Post
Thanks! Just had a quick look, I think your aligned buffer declaration won't work with VS, IIRC __declspec(align(*)) must be put before type name.

Will look into it if I can get some free time in the coming weekend...
This claims it's fine: http://stackoverflow.com/questions/7...54801#12654801
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 3rd August 2013, 03:14   #273  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by jackoneill View Post
Maybe I was wrong then. Will confirm in VS later.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 12th August 2013, 18:45   #274  |  Link
sl1pkn07
Pajas Mentales...
 
Join Date: Dec 2004
Location: Spanishtán
Posts: 496
thanks for the GCC port!
sl1pkn07 is offline   Reply With Quote
Old 19th August 2013, 03:36   #275  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
The master branch is now compilable by GCC 4.8.1 (Tested compilation under Ubuntu 13.04). Be warned that I haven't got time to actually try to run it yet, so it is likely to have bugs. The next step is port the unit test project to GCC too, to ensure the SSE optimization part is correct.

@jackoneill, thanks for your patch. Though I only used the patch for reference only, most fixes are rewritten to be better fit into the project.

To compile it, ensure GCC 4.8.1+ (we need g++ actually) and Python 3 is installed, then run:

Code:
./waf configure # If you need to specify where GCC is located at, use this: CXX=/path/to/your/g++-4.8 ./waf configure
./waf build
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 19th August 2013, 11:07   #276  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Cool!

Unfortunately, './waf build' fails here.

./waf configure -vvv: https://dpaste.de/ZKJW3/
./waf build -vvv: https://dpaste.de/QGEfR/
Maybe you can find something useful there.

I have gcc 4.8.1 and python 3.3.2.
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 19th August 2013, 11:49   #277  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by jackoneill View Post
Cool!

Unfortunately, './waf build' fails here.

./waf configure -vvv: https://dpaste.de/ZKJW3/
./waf build -vvv: https://dpaste.de/QGEfR/
Maybe you can find something useful there.

I have gcc 4.8.1 and python 3.3.2.
Should be fixed in latest commit, can you try again?
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 19th August 2013, 14:09   #278  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
That works.

Now it crashes in a most bizarre place, with a most bizarre error: https://dpaste.de/Sh7t0/

I call it like this:
Code:
c.f3kdb.Deband(ret, y=128)
Input is a 720×480 YUV420P8 clip from d2vsource.
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 19th August 2013, 15:00   #279  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by jackoneill View Post
That works.

Now it crashes in a most bizarre place, with a most bizarre error: https://dpaste.de/Sh7t0/

I call it like this:
Code:
c.f3kdb.Deband(ret, y=128)
Input is a 720×480 YUV420P8 clip from d2vsource.
OK this needs some debugging.. Will check it when I got time.

EDIT: Fixed in latest commit.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3

Last edited by SAPikachu; 20th August 2013 at 02:49.
SAPikachu is offline   Reply With Quote
Old 28th September 2013, 11:11   #280  |  Link
madshi
Registered Developer
 
Join Date: Sep 2006
Posts: 9,140
I've been working on a debanding algorithm for madVR. I've taken the core algorithm idea of flash3kyuu_deband (which is really extremely simple), but modified it a bit. Wanted to let you know what I did, so you can implement the same thing in flash3kyuu_deband, if you like. Basically there's one point where the algorithm decides whether to use the original pixel value or the average of the 4 reference pixels. I've improved this decision making which allowed me to increase the thresholds a little bit. This results in stronger debanding, with hopefully not much more detail loss.

Of course the additional checks will eat up quite a bit of performance (and might be hard to implemented with SSE?), so I'm not sure if you want to do this, but that's your choice, of course. Some of the checks might be a bit redundant, I'm not sure. But I thought I'd rather add a few more checks to make sure the higher thresholds don't come with too many negative side effects...

Basically my decision making looks like this:

Code:
// orgPixel = original pixel value
// refPixel = one of the 4 reference pixels selected by the algorithm
// surPixel = one of the 8 pixels directly surrounding the original pixel
// refPixelsAvg = simple mean average of the 4 refPixels
// localContrast = max dif between the 8 surPixels and the orgPixel
// surroundContrast = max dif between the 4 * 9 surPixels left, top, right and bottom of the "localContrast" 9 pixel block
// maxRefPixelsDif = max dif between the 4 refPixels and the orgPixel
// refPixelsDifSum = sum of the absolute dif between each refPixel and the orgPixel

float3 result = ( (abs(refPixelsAvg - orgPixel) > 2.0 / 255.0) ||
                  (localContrast                > 2.5 / 255.0) ||
                  (surroundContrast             > 3.5 / 255.0) ||
                  (maxRefPixelsDif              > 3.5 / 255.0) ||
                  (refPixelsDifSum              > 6.5 / 255.0)    ) ? orgPixel : refPixelsAvg;

// in pixel shaders one 8bit step is 1.0 / 255.0
Here's the end result, with no grain and no dithering as part of the debanding algorithm. madVR processes in high bitdepth and applies TPDF random dithering as a last step, anyway, so grain/dithering is not needed as part of the algorithm:


Last edited by madshi; 28th September 2013 at 11:15.
madshi is offline   Reply With Quote
Reply

Tags
avisynth, deband

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.