Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th January 2018, 13:19   #2921  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
It does work better now.
kolak is offline   Reply With Quote
Old 26th January 2018, 04:03   #2922  |  Link
hydra3333
Registered User
 
Join Date: Oct 2009
Location: crow-land
Posts: 540
Thank you Myrsloik, for your great work.
hydra3333 is offline   Reply With Quote
Old 26th January 2018, 13:14   #2923  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
R43-RC2

Code:
r43:
updated to zimg v2.7.2
a tuple is now only returned from get_output in python when alpha actually exist, this undoes the python syntax change in r41
fixed a possible hang on error in vspipe, regression introduced with the alpha support in r41
Second and probably final RC. Should restore the final stuff broken in R41. Go test it.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 26th January 2018, 15:05   #2924  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
I wanna remove useless samples in my training set, the ones that don't have enough high frequency components
lemme rephrase that, after I extracted the high frequency components from each frame, I wanna remove frames whose standard deviation is below a certain threshold, and record which frames were removed in a log file, then read the log file and remove the same frames in another sequence, how do I do that?
feisty2 is offline   Reply With Quote
Old 26th January 2018, 16:20   #2925  |  Link
WolframRhodium
Registered User
 
Join Date: Jan 2016
Posts: 162
Quote:
Originally Posted by feisty2 View Post
remove frames whose standard deviation is below a certain threshold, and record which frames were removed in a log file, then read the log file and remove the same frames in another sequence
The following method is 2-pass:

1st pass:
mvf.PlaneStatistics shows the method of calculating the std.

A Python list can be created outside the runtime function, then once the std of each frame inside the runtime function is calculated, the list is used to store the label.

The std of the whole video can be calculated via vspipe or simply vsedit's benchmark.

And then the list is stored in the disk.

2nd pass:
core.std.DeleteFrames can be used to delete those frames after reading the list.

Last edited by WolframRhodium; 26th January 2018 at 16:40.
WolframRhodium is offline   Reply With Quote
Old 26th January 2018, 18:14   #2926  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Quote:
Originally Posted by feisty2 View Post
I wanna remove useless samples in my training set, the ones that don't have enough high frequency components
lemme rephrase that, after I extracted the high frequency components from each frame, I wanna remove frames whose standard deviation is below a certain threshold, and record which frames were removed in a log file, then read the log file and remove the same frames in another sequence, how do I do that?
Are you doing the write and read log in one script.
lansing is offline   Reply With Quote
Old 26th January 2018, 18:19   #2927  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
how are you extracting / measuring / distinguishing between what is a "high frequency component" ?
poisondeathray is offline   Reply With Quote
Old 26th January 2018, 18:31   #2928  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by WolframRhodium View Post
The following method is 2-pass:

1st pass:
mvf.PlaneStatistics shows the method of calculating the std.

A Python list can be created outside the runtime function, then once the std of each frame inside the runtime function is calculated, the list is used to store the label.

The std of the whole video can be calculated via vspipe or simply vsedit's benchmark.

And then the list is stored in the disk.

2nd pass:
core.std.DeleteFrames can be used to delete those frames after reading the list.
seems like a lot of trouble but... gonna try it
I hate coding
feisty2 is offline   Reply With Quote
Old 26th January 2018, 18:32   #2929  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by lansing View Post
Are you doing the write and read log in one script.
one script or separate scripts, either way works for me long as its simple enough
feisty2 is offline   Reply With Quote
Old 26th January 2018, 18:41   #2930  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by poisondeathray View Post
how are you extracting / measuring / distinguishing between what is a "high frequency component" ?
u run a high pass filter (low pass filter + make dif) on ur clip and extract the high frequency components, it doesn't have to be extra accurate, I'm just removing useless image blocks from my super resolution neural net training set, image blocks without clear edges or fine details or textures are obviously garbage data to super resolution, they are doing nothing but wasting my training time, after u extracted the high frequencies, u calculate the standard deviation or the absolute or squared sum of the high frequencies to determine if a certain sample has enough amount of high frequencies for super resolution or not
feisty2 is offline   Reply With Quote
Old 26th January 2018, 19:58   #2931  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Don't post attachment, admins are all dead, nobody will approve it.
lansing is offline   Reply With Quote
Old 26th January 2018, 21:28   #2932  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,750
If it is short, use the CODE bbCode block; if it is longer, use any pastebin service (many have syntax highlighting for programming languages).

I am sure Wikipedia once had a comparison chart of such services, but it seems to be gone.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is online now   Reply With Quote
Old 27th January 2018, 00:40   #2933  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Quote:
Originally Posted by WolframRhodium View Post
A Python list can be created outside the runtime function, then once the std of each frame inside the runtime function is calculated, the list is used to store the label.
Looping through every frame of the video to store the frame property into a list has a huge time issue.

Code:
filter_list = [] # list to store hi freq frame numbers

for frame in clip.frames():
	frame_std = frame.props['PlaneSTD']
	frame_num = frame.props['FrameNum']
	if frame_std > 0.0031: # std threshold
		if frame_num not in filter_list:
			filter_list.append(frame_num)
My loop here took about 5 minutes to instantiate on a 720p 24 minutes video.
lansing is offline   Reply With Quote
Old 27th January 2018, 01:39   #2934  |  Link
WolframRhodium
Registered User
 
Join Date: Jan 2016
Posts: 162
Quote:
Originally Posted by lansing View Post
Looping through every frame of the video to store the frame property into a list has a huge time issue.

Code:
filter_list = [] # list to store hi freq frame numbers

for frame in clip.frames():
	frame_std = frame.props['PlaneSTD']
	frame_num = frame.props['FrameNum']
	if frame_std > 0.0031: # std threshold
		if frame_num not in filter_list:
			filter_list.append(frame_num)
My loop here took about 5 minutes to instantiate on a 720p 24 minutes video.
Try the runtime function
Code:
filter_list = []
def _PlaneStd(n, f, clip):
    frame_std = frame.props['PlaneSTD']
    if frame_std > 0.0031:
        filter_list.append(str(n))

    if n == clip.num_frames - 1:
        outFile = open('a.txt', 'w+')
        outFile.write('{}'.format("".join(filter_list)))

    return clip

clip = core.std.FrameEval(clip functools.partial(_PlaneStd, clip=clip), prop_src=clip)
or maybe the bottleneck is at file reading?

Last edited by WolframRhodium; 27th January 2018 at 01:44.
WolframRhodium is offline   Reply With Quote
Old 27th January 2018, 01:43   #2935  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
I'm not sure that script will always work properly. Frames can be (and commonly are) processed out of order. Just saying...

Since we're on the subject I'm planning to add support for writing frame properties to a file in vspipe. Unfortunately for you I didn't get around to implementing it yet. Also, using get_frame() in python is always slow, it basically forces single threading.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 27th January 2018, 03:01   #2936  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Quote:
Originally Posted by WolframRhodium View Post
Try the runtime function
Code:
filter_list = []
def _PlaneStd(n, f, clip):
    frame_std = frame.props['PlaneSTD']
    if frame_std > 0.0031:
        filter_list.append(str(n))

    if n == clip.num_frames - 1:
        outFile = open('a.txt', 'w+')
        outFile.write('{}'.format("".join(filter_list)))

    return clip

clip = core.std.FrameEval(clip functools.partial(_PlaneStd, clip=clip), prop_src=clip)
or maybe the bottleneck is at file reading?
I thought it was a bad practice to mutate data that was outside the scope of a standalone function.
lansing is offline   Reply With Quote
Old 28th January 2018, 04:38   #2937  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
well thank u all, I got it worked out alright now
feisty2 is offline   Reply With Quote
Old 29th January 2018, 20:43   #2938  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
R43 RC4

One final RC (for real this time) since many small nitpicky things were discovered.

Code:
r43:
updated to zimg v2.7.2
all vsscript functions are now internally guarded by a mutex to make them threadsafe
changed boxblur defaults to hradius and vradius=1, this makes more sense than having it throw an error with default arguments
added xyz as an alias for st428 in resize
add vsscript_getvsapi2 function which takes the required vapoursynth api version into account
a tuple is now only returned from get_output in python when alpha actually exist, this undoes the python syntax change in r41
fixed a possible hang on error in vspipe, regression introduced with the alpha support in r41
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 31st January 2018, 11:33   #2939  |  Link
monohouse
Registered User
 
Join Date: Jul 2007
Posts: 106
hi I just installed (on windows) vapor R42.1 with python but no matter how I do it I always get the error:
Code:
c:\Program Files (x86)\Python36-32>VSPipe.exe --info G:\test.vpy -
Failed to initialize VapourSynth environment
python is installed, I had both 32 and 64, I also tried to use the portable vapor - it give the same error
monohouse is offline   Reply With Quote
Old 31st January 2018, 11:34   #2940  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Kinnarps Chair
Posts: 2,548
Quote:
Originally Posted by monohouse View Post
hi I just installed (on windows) vapor R42.1 with python but no matter how I do it I always get the error:
Code:
c:\Program Files (x86)\Python36-32>VSPipe.exe --info G:\test.vpy -
Failed to initialize VapourSynth environment
python is installed, I had both 32 and 64, I also tried to use the portable vapor - it give the same error
Let me guess, you didn't install all windows updates...
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Reply

Tags
speed, vaporware, vapoursynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:04.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.