Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th September 2011, 21:15   #101  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by mp3dom View Post
Makes sense, but I have recompressed it again to uncompressed mov but the file is exactly the same (unless -vcodec v210 doesn't mean 'recompress to v210')
Maybe ffmpeg does "smart rendering" so it actually does the same as copy option.
kolak is offline   Reply With Quote
Old 17th September 2011, 01:51   #102  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
@mp3dom @kolak @poisondeathray

I just checked the qt format spec and found the reason. mp3dom was right and I was wrong, the offset should be 0x24 = 36. This mdat chunk has 64bit length - that is, the 4 bytes before "mdat" is "00 00 00 01", and a 8-byte length follows the tag, real data then follows the length. So file_head should be 8 bytes after mdat.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 02:03   #103  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by mp3dom View Post
Makes sense, but I have recompressed it again to uncompressed mov but the file is exactly the same (unless -vcodec v210 doesn't mean 'recompress to v210')
v210 is an uncompressed format, so recompress the file won't change the data.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 11:37   #104  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
@Sap...
That's great, I wrote the original script. Can I package it up with your credit? I wanted to know the quick way to find the file head, but I didn't want to read the huge manual.. now I know and I think I can fix this in script to be fully automatic, which is great!
I think we're really making progress on some quite usable high bit depth processing, even ffmeg is going to have ProRes soon...

ps what would be really useful is a type of parsing file reader that you could say fileread(int a, float b, str c ) etc. something like that, but choose word length to read only bytes, and nops to skip bytes. We need to read data files. Maybe a version of sashimi that reads to script variables instead of clips?

Last edited by jmac698; 17th September 2011 at 11:40.
jmac698 is offline   Reply With Quote
Old 17th September 2011, 12:20   #105  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by SAPikachu View Post
@mp3dom @kolak @poisondeathray

I just checked the qt format spec and found the reason. mp3dom was right and I was wrong, the offset should be 0x24 = 36. This mdat chunk has 64bit length - that is, the 4 bytes before "mdat" is "00 00 00 01", and a 8-byte length follows the tag, real data then follows the length. So file_head should be 8 bytes after mdat.
OK- so we need swapuv after script, otherwise colors are wrong. I will check with different files, but it looks like this is a rule (so script can be adjusted inside function).

btw...I compared (just quickly for now) quality and it looks very good and promising. Conversion was slow thought 1.5fps for full HD, but using only one core. Not sure how I can make it using many cores, but in theory it can run at acceptable speed. Looking good

Andrew

Last edited by kolak; 17th September 2011 at 12:25.
kolak is offline   Reply With Quote
Old 17th September 2011, 12:47   #106  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by jmac698 View Post
@Sap...
That's great, I wrote the original script. Can I package it up with your credit? I wanted to know the quick way to find the file head, but I didn't want to read the huge manual.. now I know and I think I can fix this in script to be fully automatic, which is great!
I think we're really making progress on some quite usable high bit depth processing, even ffmeg is going to have ProRes soon...

ps what would be really useful is a type of parsing file reader that you could say fileread(int a, float b, str c ) etc. something like that, but choose word length to read only bytes, and nops to skip bytes. We need to read data files. Maybe a version of sashimi that reads to script variables instead of clips?
Of course you can. Just do anything to it as you like.

I don't understand the second part. Do you want to directly parse the file header in script? I think that will be very hard and hackish since AviSynth lacks some useful features (like loops). A better method is integrate a more powerful scripting environment into it. Actually I am thinking of creating a python plugin environment, but I don't have much time to do it recently...
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 12:51   #107  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Another thing- is this Floyd dithering just pure implementation?

Filter which I compared it to use noise shaping/error diffusion plus optional random noise.
When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.

Can we do the same- is it good idea?


Andrew
kolak is offline   Reply With Quote
Old 17th September 2011, 12:55   #108  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by kolak View Post
OK- so we need swapuv after script, otherwise colors are wrong. I will check with different files, but it looks like this is a rule (so script can be adjusted inside function).

btw...I compared (just quickly for now) quality and it looks very good and promising. Conversion was slow thought 1.5fps for full HD, but using only one core. Not sure how I can make it using many cores, but in theory it can run at acceptable speed. Looking good

Andrew
We just need to adjust the file_head based on the mdat chunk mode. SwapUV is unneeded and wrong.

It is hard to get speed improvement using pure script implementation. Better to integrate it into a source plugin, like Sashimi.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 12:58   #109  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by SAPikachu View Post
We just need to adjust the file_head based on the mdat chunk mode. SwapUV is unneeded and wrong.

It is hard to get speed improvement using pure script implementation. Better to integrate it into a source plugin, like Sashimi.
But when I use new (as you said- correct offset) colors are wrong
Tried 2 totally different files- without swpuv colors are wrong.


Andrew

Last edited by kolak; 17th September 2011 at 13:03.
kolak is offline   Reply With Quote
Old 17th September 2011, 13:00   #110  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by kolak View Post
Another thing- is this Floyd dithering just pure implementation?

Filter which I compared it to use noise shaping/error diffusion plus optional random noise.
When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.

Can we do the same- is it good idea?


Andrew
f3kdb_dither is pure dithering. I think noise is unneeded unless the clip needs debanding. In next version you can use f3kdb on the 16bit clip.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 13:02   #111  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
Quote:
Originally Posted by kolak View Post
OK- so we need swapuv after script, otherwise colors are wrong.
No, wrong colors was with wrong offset (0x1c) and in that case the SwapUV fixed it. With right offset (0x24) the colors are ok from the start.

Quote:
When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.
Xscaler

Quote:
Originally Posted by SAPikachu View Post
f3kdb_dither is pure dithering. I think noise is unneeded unless the clip needs debanding. In next version you can use f3kdb on the 16bit clip.
Is it possible (in a future release, without hurry) to implement it as an optional parameter? About f3kdb, isn't it already working on 16bit clip as source? Or are you referring to the deband part (flash3kyuu)? In that case it would be very helpful to deband a 16bit clip and then dither to 8bit.

Last edited by mp3dom; 17th September 2011 at 13:13.
mp3dom is offline   Reply With Quote
Old 17th September 2011, 13:03   #112  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by kolak View Post
But when I sue new (as you said- correct offset) colors are wrong



Andrew
There is 2 types of mdat chunk. If the 4 bytes before mdat is 00 00 00 01, you need to add 8 to the offset after mdat. Otherwise the original offset is correct.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 13:05   #113  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by SAPikachu View Post
There is 2 types of mdat chunk. If the 4 bytes before mdat is 00 00 00 01, you need to add 8 to the offset after mdat. Otherwise the original offset is correct.
OK- understand now- thanks. Didn't get your original message: Will check and confirm.
Yes- my files don't have 00 00 00 01, so "normal" offset works fine.

Last edited by kolak; 17th September 2011 at 13:22.
kolak is offline   Reply With Quote
Old 17th September 2011, 13:32   #114  |  Link
SAPikachu
Registered User
 
SAPikachu's Avatar
 
Join Date: Aug 2007
Posts: 218
Quote:
Originally Posted by mp3dom View Post
Is it possible (in a future release, without hurry) to implement it as an optional parameter? About f3kdb, isn't it already working on 16bit clip as source? Or are you referring to the deband part (flash3kyuu)? In that case it would be very helpful to deband a 16bit clip and then dither to 8bit.
Yes I meant flash3kyuu_deband. I will make the new version when I have time.
__________________
f3kdb 1.5.1 / MP_Pipeline 0.18

ffms2 builds with 10bit output hack:
libav-9a60b1f / ffmpeg-1e4d049 / FFmbc-0.7.1
Built from ffms2 6e0d654 (hack a9fe004)

Mirrors: http://bit.ly/19TwDD3
SAPikachu is offline   Reply With Quote
Old 17th September 2011, 18:40   #115  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
Yes the script can be sped up alot, that is the reason I requested some new features in sashimi. Those features were finalized just yesterday, so now it is time to rewrite readv210 with the new features, which should speed up load the files a lot.

My understanding of dither is that it can be fixed or random. I read a whole textbook on it once (long time ago), and it seems that uncorrelated noise with shaping looked best. That old floyd-steinberg is easy to understand but causes it's own banding (there's only 5 shades for 4 dots).

Btw, I worked on my own algorithm for fixed dither, that always sums to the target value for the area, meaning you could perfectly reverse it back into a high bit-depth image at full resolution! It can't always work for some pixels though.

As for the file parser, I could use this idea for other things, speed is not an issue, like reading the histograms from colorlike per frame, it's only 256 numbers of text, or reading the in-pattern results in dgindex, just one line per frame.. and in this case, reading 8 bytes to determine file header - such a siimple task, that I don't want to write a whole plugin just for those numbers... because ultimiately I find coding up an idea in script far, far faster to get a practical result (even though it may be slow). And yes you can do loops in avisythn, however I don't need them to parse files, the plugin will have some simple logic for that.
jmac698 is offline   Reply With Quote
Old 17th September 2011, 19:03   #116  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by SAPikachu View Post
AviSynth lacks some useful features (like loops).
See GScript in my sig.
For the purist, you can also use recursive functions to create loops in vanilla Avisynth.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 17th September 2011, 19:25   #117  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
recursion and gscript are documented here
http://avisynth.org/mediawiki/Advanced_Scripting_Tips
jmac698 is offline   Reply With Quote
Old 17th September 2011, 19:42   #118  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Quote:
Originally Posted by jmac698 View Post

My understanding of dither is that it can be fixed or random. I read a whole textbook on it once (long time ago), and it seems that uncorrelated noise with shaping looked best. That old floyd-steinberg is easy to understand but causes it's own banding (there's only 5 shades for 4 dots).
I think thit is the reason why MS filter adds noise- to cover FS own banding.

To be honest- going from 10 to 8bit with MS filter looks good enough- I would rather like to have more speed.

What about these methods:

Jarvis, Judice, and Ninke dithering diffuses the error also to pixels one step further away. The dithering is coarser, but has fewer visual artifacts. It is slower than Floyd–Steinberg dithering because it distributes errors among 12 nearby pixels instead of 4 nearby pixels for Floyd–Steinberg.

Stucki dithering is based on the above, but is slightly faster. Its output tends to be clean and sharp.

Burkes dithering is a simplified form of Stucki dithering that is faster, but less clean than Stucki dithering.

As a goal I would see 2x faster than RT dithering for full HD on eg. 12 core PC, with quality as FS.


Andrew
kolak is offline   Reply With Quote
Old 17th September 2011, 19:47   #119  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: Poland
Posts: 2,843
Current solution can give about 12fps on 8 core machine (older Xeons- not i7)- just need to run many instances. MS filter is a bit faster.


Andrew

Last edited by kolak; 17th September 2011 at 19:54.
kolak is offline   Reply With Quote
Old 18th September 2011, 00:02   #120  |  Link
Fullmetal Encoder
Registered User
 
Join Date: Jan 2011
Posts: 107
Quote:
Originally Posted by SAPikachu View Post
Yes I meant flash3kyuu_deband. I will make the new version when I have time.
Thank you very much for this. I eagerly await the full higher bit depth capability. Thanks also for providing us with such a high quality dithering tool. It's very impressive compared to what else is available.
Fullmetal Encoder is offline   Reply With Quote
Reply

Tags
avisynth, deband

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.