Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 1st January 2013, 14:17   #641  |  Link
jackoneill
unsigned int
 
jackoneill's Avatar
 
Join Date: Oct 2012
Location: 🇪🇺
Posts: 760
Quote:
Originally Posted by Myrsloik View Post
And why do you hate empty strings in python? I'll apply the qfatal stuff with spaces tomorrow but the rest is rejected. The precedence of logic operators is also well defined so I don't intend to change it just to please a very high compiler warning level.
I think both gcc and clang warn about those empty strings, but they do the job anyway.
__________________
Buy me a "coffee" and/or hire me to write code!
jackoneill is offline   Reply With Quote
Old 1st January 2013, 22:39   #642  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Debugging help needed

I'm finally going completely crazy. I have narrowed down the python memory leak quite a bit but I just can't explain the last part. If anyone happens to be good at debugging python (perhaps you've even used cython too) memory leaks and want to help please contact me. I have a fairly simple case that can reproduce the issue but I simply don't know what triggers it in the first place.

Unless this bug is found it will never be possible to simply press F2 again and again in vdub to refresh a script.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 2nd January 2013, 18:56   #643  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,699
quick update for Waf version.

Code:
Index: bootstrap.py
===================================================================
--- bootstrap.py	(revision 405)
+++ bootstrap.py	(working copy)
@@ -2,7 +2,7 @@
 
 import os, stat, urllib2
 
-f = urllib2.urlopen('http://waf.googlecode.com/files/waf-1.7.6')
+f = urllib2.urlopen('http://waf.googlecode.com/files/waf-1.7.8')
 
 with open('waf', 'wb') as waf:
     waf.write(f.read())
BTW, Myrsloik, do you have a preferred method in which we send patches to you?
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 2nd January 2013, 19:17   #644  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by Adub View Post
quick update for Waf version.

Code:
Index: bootstrap.py
===================================================================
--- bootstrap.py	(revision 405)
+++ bootstrap.py	(working copy)
@@ -2,7 +2,7 @@
 
 import os, stat, urllib2
 
-f = urllib2.urlopen('http://waf.googlecode.com/files/waf-1.7.6')
+f = urllib2.urlopen('http://waf.googlecode.com/files/waf-1.7.8')
 
 with open('waf', 'wb') as waf:
     waf.write(f.read())
BTW, Myrsloik, do you have a preferred method in which we send patches to you?
I'm not that picky but if you open an issue for it on the bug tracker and add it as an attachment I definitely won't miss it. I'm happy as long as it's in some kind of diff/patch format, really.

Anyway, applied.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 4th January 2013, 07:57   #645  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,699
Cleanup for some of the invert_example.c comments: https://gist.github.com/4450489
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 4th January 2013, 17:50   #646  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by Adub View Post
Cleanup for some of the invert_example.c comments: https://gist.github.com/4450489
Applied. I've also finally managed to solve most of the evils of python so the next version will leak a lot less memory.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 4th January 2013, 22:49   #647  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,699
Quote:
Originally Posted by Myrsloik View Post
Applied. I've also finally managed to solve most of the evils of python so the next version will leak a lot less memory.
Excellent, good to hear. I've begun work on my CUDA environment. I'm getting used to the Waf build system and incorporating the CUDA compiler into the mix. It's almost up an ready for dev, but I've got a few CFLAGS issues to fix first.
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 5th January 2013, 11:28   #648  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by HolyWu View Post
Will the missing operator ^(power) in Expr be available in the next version as well?
I have no optimized code for it so just use exp and log to get the same result.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 5th January 2013, 21:38   #649  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by Myrsloik View Post
I have no optimized code for it so use exp and log to get the same result.
For a RPN/LUT dummy like myself, how would I write this mt_lut expression using Expr?
Code:
mt_lut("x 128 - 1.49 * x 128 - 2 ^ x 128 - 2 ^ 9 + / * 128 +")
I also think it would be a good idea for existing mt_lut/x/y/z expressions to be compatible with Expr with no additional gimmicks.
Reel.Deel is offline   Reply With Quote
Old 5th January 2013, 23:09   #650  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by Reel.Deel View Post
For a RPN/LUT dummy like myself, how would I write this mt_lut expression using Expr?
Code:
mt_lut("x 128 - 1.49 * x 128 - 2 ^ x 128 - 2 ^ 9 + / * 128 +")
I also think it would be a good idea for existing mt_lut/x/y/z expressions to be compatible with Expr with no additional gimmicks.
Well... theoretically I could add the ^ operator but it would do you no good since it's very slow just for squaring things. In this case x*x is much faster. So I give you this equivalent that doesn't even need ^ and removes some duplicate evaluation:
Code:
"x 128 - 1.49 * x 128 - dup * dup 9 + / * 128 +"
(warning, completely untested but should work and if it doesn't just tell me and I'll actually try it myself)

When you go from a lut to actually evaluating the expression for every pixel you have to be more careful about how you write it too or it could become slow.

Bonus material by jfs, the original expression in readable form:
Code:
(x-128)*1.49 * (((x-128)^2) / ((x-128)^2 + 9)) + 128
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 6th January 2013, 02:44   #651  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by Myrsloik View Post
(warning, completely untested but should work and if it doesn't just tell me and I'll actually try it myself)
Expr expression works as expected, the only only thing is that mt_lut by default only processes the luma. So I read the Expr documentation and added the following and now the results look identical.
Code:
avs = core.avs.mt_lut(c1=src, expr="x 128 - 1.49 * x 128 - 2 ^ x 128 - 2 ^ 9 + / * 128 +", Y=3, U=1, V=1)
vs =  core.std.Expr(clips=[src], expr=["x 128 - 1.49 * x 128 - dup * dup 9 + / * 128 +", "", ""])
Thanks for the bonus material. Much easier to understand.
Reel.Deel is offline   Reply With Quote
Old 7th January 2013, 00:37   #652  |  Link
wOxxOm
Oz of the zOo
 
Join Date: May 2005
Posts: 208
Quote:
Originally Posted by Myrsloik View Post
Well... theoretically I could add the ^ operator but it would do you no good since it's very slow just for squaring things. In this case x*x is much faster. So I give you this equivalent that doesn't even need ^ and removes some duplicate evaluation
What about "something 1.05 ^"?
What about "something something ^"?
Also the calculation is done only once while building lut table, so the overhead may safely be neglected.

Quote:
Originally Posted by Reel.Deel View Post
I also think it would be a good idea for existing mt_lut/x/y/z expressions to be compatible with Expr with no additional gimmicks.
Absolutely.
There are lots of useful scripts with very complex expressions.

Last edited by wOxxOm; 7th January 2013 at 00:52.
wOxxOm is offline   Reply With Quote
Old 7th January 2013, 00:56   #653  |  Link
pie
Registered User
 
pie's Avatar
 
Join Date: Sep 2012
Posts: 17
What about "something 1.05 ^"?
What about "something something ^"?

I'm not too sure with RPN, but something like this?
something log 1.05 * exp

For a^b
a log b * exp

--> e^(ln(a)*b)
pie is offline   Reply With Quote
Old 7th January 2013, 00:58   #654  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by wOxxOm View Post
what about "something 1.05 ^"?


Absolutely.
There are lots of useful scripts using very complex expressions.
Then you have to remember from school that a^x = e^(x*ln(a))

so you rewrite it to:
a log x * exp (or something log 1.05 * exp)

And I have to reiterate that just copying and pasting expressions is kind of a bad idea for performance reasons. Just take the one I changed around above as an example.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet

Last edited by Myrsloik; 7th January 2013 at 00:59. Reason: typo
Myrsloik is offline   Reply With Quote
Old 7th January 2013, 10:57   #655  |  Link
wOxxOm
Oz of the zOo
 
Join Date: May 2005
Posts: 208
Quote:
Originally Posted by Myrsloik View Post
just copying and pasting expressions is kind of a bad idea for performance reasons.
What performance?

Quote:
Originally Posted by wOxxOm View Post
the calculation is done only once while building lut table, so the overhead may safely be neglected.
This is not something one learns at school, I guess, but still it's obvious.
wOxxOm is offline   Reply With Quote
Old 7th January 2013, 12:17   #656  |  Link
Chikuzen
typo lover
 
Chikuzen's Avatar
 
Join Date: May 2009
Posts: 595
lutxyz requires at least about 16MiB(256*256*256*1byte).
Since a huge table is not settled in CPU cache, it may need very very slow memory access each time.
Therefore, runtime calcuration is often faster than LUT.
__________________
my repositories

Last edited by Chikuzen; 7th January 2013 at 12:33.
Chikuzen is offline   Reply With Quote
Old 7th January 2013, 13:44   #657  |  Link
wOxxOm
Oz of the zOo
 
Join Date: May 2005
Posts: 208
Chikuzen, is this somehow related to my remarks about the majority of lut scripts, taking in account that lutxyz should be avoided at all costs and thankfully not commonly used anyway? Also, don't forget that memory access penalty during building of that 16MB table is *nothing* compared with memory penalty during processing of lots of actual video frames.
wOxxOm is offline   Reply With Quote
Old 7th January 2013, 14:13   #658  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
It is lut access during processing of video frames that will have a heavy cache miss penalty.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 7th January 2013, 14:27   #659  |  Link
wOxxOm
Oz of the zOo
 
Join Date: May 2005
Posts: 208
StainlessS, I wrote about the same earlier, and to reiterate my point I'll rephrase: having a cpu-heavy instructions in expressions (like "^" - power) takes only an insignificant amount of overall processing time (it's calculated only one time, not each frame) so there's no need to exclude those in favour of non-practical 'performance gains' while degrading substantially ease of porting for existing lut-scripts.

Last edited by wOxxOm; 7th January 2013 at 14:29.
wOxxOm is offline   Reply With Quote
Old 7th January 2013, 14:31   #660  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by wOxxOm View Post
Also the calculation is done only once while building lut table, so the overhead may safely be neglected.
expr() in Vapoursynth doesn't use a LUT, it re-evaluates the expression for each pixel, hence Myrsloik's comments about how you should be careful with your RPN expressions. It's done that way because building LUT's for high bitdepth video is a really stupid idea.

People who like faith-based optimizations may note that expr() isn't really any significantly slower than mt_lutxy[z] in many cases; for simple expressions it can even be faster than mt_lutxy.

Last edited by TheFluff; 7th January 2013 at 14:34.
TheFluff is offline   Reply With Quote
Reply

Tags
speed, vaporware, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:42.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.