FlipVertical and New BitBlt

[ARDA]

This is an update and different development for flipvertical, at least one part of it
that can work in place and also use avx instruction in new machines.
To avoid conflict names you must use as;

Fvertcal()

If you have my old flips.dll in your avisynth plugin folder you should replace it by
this new Vertical.dll

But actually this update was an excuse to develop a new bitblt, something I started many
years ago and has been sleeping in my disks and changing from one machine to another at least
for the last eight years, and as I never arrive to finish this project, I release it now the
way it is.
This new bitblt has only been tested in this plugin and others of my own use, never tested
deeply as a substitute of the internal one, only a few tests, so I cannot guarantee for now
full compatibility and free of bugs. It makes use of sse2, Ssse3 and avx instructions depending
on the machine on which is running.

This project includes four files from Agner Fog's libraries, cachesize32.asm, cputype32.asm,
instrset32.asm, unalignedfaster32.asm and some slightly modified subroutines from memcpy32.asm
You can find them in http://www.agner.org/optimize/asmlib.zip
All original Agner Fog's sources are also included in this file

Version 1.0 Fvertcal.7z (https://mega.nz/#!tI8jwYBC!PQdhAO9SEhNw9WSSNTqU_t-mgwY5Dkd-9C5AVlfNgjg)
Version 1.0 Fvertcal.zip (https://mega.nz/#!QNll3DxY!bYqD-0bJwlbdgFJUMjixCAcmytcAzI6tnXKCScC1YeQ)

Version 1.01 Fvertcal.7z (https://mega.nz/#!UVtgiRAR!dwBcEE9h2VshPswxmz5wzrVZFnb7WfNV5X8OAXv7clY)

Version 1.002 Fvertcal.dll (https://mega.nz/#!FFdW0ZRK!_stE39sFhzDjMlnB_vI7aEfHq8KEroH5adkxqRqfCr0)

Version 1.003 Fvertcal.dll (https://mega.nz/#!1EMXRZoS!6co0as92JfKA-ot5xyOY0a_sz3ugtsWCFAPi2AuZyO0)

Version 1.004 Fvertcal.dll (https://mega.nz/#!hVtShDAY!305YLuXYMoL_pkbiXRSvudnh30i2KKuEBK8knhC68ug)

Version 1005 Fvertical.dll (https://mega.nz/#!RFdRHaKI!qa15JjN58CuiPlIAu2m6H-sJFDwkfphAXq5ghNg-Qh8)


I hope this can be usefull
ARDA

[Groucho2004]

Quick test on my i5-2500K (Sandy Bridge):

Script:
blankclip(length = 1000, width = 5000, height = 3000, color=$005B8B).killaudio().assumefps(50, 1)
#flipvertical()
#fvertical()


Result with "flipvertical()"
[Runtime info]
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 124.8 | 148.5 | 141.8
Memory usage (phys | virt): 121 | 120 MB
Thread count: 1
CPU usage (average): 24%
Time (elapsed): 00:00:07.050


Result with "fvertical()"
[Runtime info]
Frames processed: 1000 (0 - 999)
FPS (min | max | average): 121.3 | 141.3 | 135.9
Memory usage (phys | virt): 121 | 120 MB
Thread count: 1
CPU usage (average): 23%
Time (elapsed): 00:00:07.360


This is on XP, so no AVX optimizations used.

[ARDA]

If you have any doubt about the performance of any filter, I propose the following script

MPEG2Source("your source")
# or any source you like and use always the same to get a little more accurate benchmarks
# and test always the same frames each time. 9000 frames it is a good quantity for this script.

#TemporalSoften(4,8,8,15,2) # use this line to force a non writable src and test when a new video frame
# is created by your filter or not. It is just an example.

AvsTimer(frames=1000, name="ANYONE",type=3, frequency=x?, total=false, quiet=true)# use your cpu frequency

# Put here your filter to benchmark

#flipvertical()
#fvertical()

AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=x?, difference=1, total=false)# use your cpu frequency

Open the scipt in virtualdub, set direct stream copy, set an initial frame and an end frame.
Open debugview(google), set a filter highlight in debugview, in this example *ANYONE*
Go back to virtualdub and Run video analysis pass. You will see in debug view windows the results every 1500 frames.

If anyone knows and wants to propose any other more accurate method to benchmark, please post
here to discuss it.

I hope this can be usefull
ARDA

[ARDA]

@Groucho2004

The variation that your benchmark shows is too small, and your test is measuring blanckclip as well, and the efect it has on memory
Please try the method I propose and tell me what results you have

Thanks ARDA

[Groucho2004]

If you have any doubt about the performance of any filter, I propose the following script

MPEG2Source("your source")
# or any source you like and use always the same to get a little more accurate benchmarks
# and test always the same frames each time. 9000 frames it is a good quantity for this script.

#TemporalSoften(4,8,8,15,2) # use this line to force a non writable src and test when a new video frame
# is created by your filter or not. It is just an example.

AvsTimer(frames=1000, name="ANYONE",type=3, frequency=x?, total=false, quiet=true)# use your cpu frequency

# Put here your filter to benchmark

#flipvertical()
#fvertical()

AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=x?, difference=1, total=false)# use your cpu frequency

Open the scipt in virtualdub, set direct stream copy, set an initial frame and an end frame.
Open debugview(google), set a filter highlight in debugview, in this example *ANYONE*
Go back to virtualdub and Run video analysis pass. You will see in debug view windows the results every 1500 frames.

If anyone knows and wants to propose any other more accurate method to benchmark, please post
here to discuss it.

I hope this can be usefull
ARDA
I measured it with AVSMeter. Its timer is very accurate (particularly considering the timer peculiarities with multi-core CPUs) and I can't see the advantage in using AVSTimer for such a simple script.

I used "blankclip" instead of a "real" source because it's extremely fast and does not add any overhead.

[Groucho2004]

@Groucho2004

The variation that your benchmark shows is too small, and your test is measuring blanckclip as well, and the efect it has on memory
Please try the method I propose and tell me what results you have

Thanks ARDA
How about you post some results?

[ARDA]

In my haswell laptop. intel family 6 model 45h
Test done with the above method described.
with a size clip of 5000 x 3000 (Y8)
I will be doing more tests but please give some time.

with avstimer Fvertical()
VirtualDub.exe [91497] ANYONE = 302 fps
VirtualDub.exe [92997] ANYONE = 308 fps
VirtualDub.exe [94497] ANYONE = 309 fps
VirtualDub.exe [95997] ANYONE = 306 fps
VirtualDub.exe [97497] ANYONE = 289 fps
VirtualDub.exe [98997] ANYONE = 305 fps

with avstimer Flipvertical()
VirtualDub.exe [91499] ANYONE = 263 fps
VirtualDub.exe [92999] ANYONE = 268 fps
VirtualDub.exe [94499] ANYONE = 270 fps
VirtualDub.exe [95999] ANYONE = 270 fps
VirtualDub.exe [97499] ANYONE = 270 fps
VirtualDub.exe [98999] ANYONE = 268 fps

Thanks ARDA

[Groucho2004]

@Groucho2004

The variation that your benchmark shows is too small, and your test is measuring blanckclip as well, and the efect it has on memory
Please try the method I propose and tell me what results you have

Thanks ARDA
I get the same results with your method but only when I use the default frequency of my CPU (3300 MHz). I actually have it overclocked to 4000 MHz.
AvsTimer obviously uses the rdtsc instruction to measure time based on the CPU time stamp but this is highly unreliable, especially with modern CPUs.

[ARDA]

AvsTimer obviously uses the rdtsc instruction to measure time based on the CPU time stamp
but this is highly unreliable, especially with modern CPUs.

Extracted from avstimer docs:

The default method type=2 is based on the windows functions GetThreadTimes,
which measures the time the filter chain spends within the thread, by which is
it is called. This is the most accurate method as long as no other threads are
used by the filter chain. Currently I know of no example which violates this
assumption. However, as hyperthreading becomes more and more fashionable times
may change. If one uses the option type=1, then time is measured with GetProcessTimes.
Then the time of the process, which runs the Avisynth script, is measured,
while the process is executing the filter chain. However, some threads of this
process, which have nothing to do with the filters, but run parallel may
artificially reduce the frame rate. If type=0 or type=3 is used, then instead of
process or thread time the absolute time is taken. Then also other processes may
influence the frame rates to the downside. If type=0 then the function
QueryPerformaceCounter is used, if type=3 then the cpu instruction RDTSC is used.
type=3 has certainly the smallest overhead, but during initialisation 1 second is
necessary to determine the cpu frequency and on some notebooks the cpu frequency
may change over time obliterating all the results. To avoid the delay one may
also specify the cpu frequency with the frequency option. This must happen with
the first type=3 timer or any other timer befor the first type=3 timer otherwise
the frequency option is ignored. Thus AvsTimer(type=3, frequency= 1300) defines
a type= 3 timer for a cpu with 1300 MHZ. One can get the exact cpu frequency by
running a type= 3 timer without the frequency option. Then the exact cpu frequency
is displayed in the debugview window. Timers, which are paired together by the
difference option, shouldbe of the same type, otherwise the results do not make
sense and may even yield negative frame rates. Timers of type!= 2 have an
additional advantage. With such timers the frame rate of the entire process
running the script - we call this the total frame rate - can be measured and
is displayed by default.
Yes in the method I proposed, type=3 difines that rdtsc wil be used, if you
prefer QueryPerformaceCounter counter use type=0. You will get different results
but almost sure with the same distance between them.
Anyway for more accurate tests we should have to set real time priority and set
a thread affinity in the source plugin code, but that is not real life and I think
it wouldn't run under Windows XP(not sure)
There are a lot of academic discussion here and there all over the net about which is
the most accurate method to benchmark codes, I wouldn't like this thread become
about this subject.
I did not do the benchmarks with the example in RGBB32 cause it would take to much time
and with this resolution(5000x3000) new bitblt is also applied with non temporal stores.
In my machine and with a y8 clip here are the results;

Source= 5000x3000 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
fvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe AvsTimer 0.8.1
VirtualDub.exe AvsTimer 0.8.1
VirtualDub.exe [91499] ANYONE = 306 fps
VirtualDub.exe [92999] ANYONE = 305 fps
VirtualDub.exe [94499] ANYONE = 305 fps
VirtualDub.exe [95999] ANYONE = 305 fps
VirtualDub.exe [97499] ANYONE = 310 fps
VirtualDub.exe [98999] ANYONE = 310 fps

Use type=2 GetThreadTimes
VirtualDub.exe [91499] ANYONE = 305 fps
VirtualDub.exe [92999] ANYONE = 317 fps
VirtualDub.exe [94499] ANYONE = 311 fps
VirtualDub.exe [95999] ANYONE = 323 fps
VirtualDub.exe [97499] ANYONE = 331 fps
VirtualDub.exe [98999] ANYONE = 313 fps

Use type=3 RDTSC
VirtualDub.exe [91497] ANYONE = 302 fps
VirtualDub.exe [92997] ANYONE = 308 fps
VirtualDub.exe [94497] ANYONE = 309 fps
VirtualDub.exe [95997] ANYONE = 306 fps
VirtualDub.exe [97497] ANYONE = 289 fps
VirtualDub.exe [98997] ANYONE = 305 fps

************************************************************************************

Source= 5000x3000 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe [91499] ANYONE = 262 fps
VirtualDub.exe [92999] ANYONE = 267 fps
VirtualDub.exe [94499] ANYONE = 267 fps
VirtualDub.exe [95999] ANYONE = 266 fps
VirtualDub.exe [97499] ANYONE = 267 fps
VirtualDub.exe [98999] ANYONE = 267 fps

Use type=2 GetThreadTimes
VirtualDub.exe [91499] ANYONE = 261 fps
VirtualDub.exe [92999] ANYONE = 275 fps
VirtualDub.exe [94499] ANYONE = 274 fps
VirtualDub.exe [95999] ANYONE = 268 fps
VirtualDub.exe [97499] ANYONE = 264 fps
VirtualDub.exe [98999] ANYONE = 284 fps

Use type=3 RDTSC
VirtualDub.exe [91499] ANYONE = 263 fps
VirtualDub.exe [92999] ANYONE = 268 fps
VirtualDub.exe [94499] ANYONE = 270 fps
VirtualDub.exe [95999] ANYONE = 270 fps
VirtualDub.exe [97499] ANYONE = 270 fps
VirtualDub.exe [98999] ANYONE = 268 fps


All above test shows an increase in performance of around 18% for flip vertical
In the conditions of a Y8 clip of 5000*3000 the plugin is using the new bitblt
by using non temporal stores, at least in my machine in which the largest cache
is a L3 of 3MB.
Under other conditions the difference can arrive till 30% or more; maybe in a few days
if I have time I will published more tests.

Thanks ARDA

[jpsdr]

The link for asmlib.zip is not working, at least for me.

[ARDA]

Try again please

[jpsdr]

It's working now.

EDIT :
Very interesting this asmlib !!!!
I didn't know about it, time to update all my projects...:D

:thanks:

[ARDA]

Very interesting this asmlib !!!!
I didn't know about it, time to update all my projects...

Happy you find something usefull.
If your projects include something relative to new bitblt or memcpy in avisynth I
encourage you to include my new source, and improve it if you find something wrong, anyway it
is a good idea to read all manuals in Agner Fog's page and mainly the assembler optimization.

Thanks ARDA

[jpsdr]

If your projects include something relative to new bitblt or memcpy in avisynth I
encourage you to include my new source, and improve it if you find something wrong


Big problem is that i'm absolutely not an yasm guy...:o
Other problem is that your code seems 32 bits only (because i've tried to take a look to find this new bitblt...:D).
So, unless there is something i'm more able to understand, i think for now i'll stay only with asmlib and allready build libraries.

[ARDA]

Big problem is that i'm absolutely not an yasm guy...
Other problem is that your code seems 32 bits only (because i've tried to take a look to find
this new bitblt...).
So, unless there is something i'm more able to understand, i think for now i'll stay only with
asmlib and allready build libraries.
Yes, all this project is 32bits only, new bitblt is all in BitBlt_SSE2_avs.asm file,
it is in the zip(see first post)
If you want a bitblt for 64 bits don't expect it soon from my side.
The day we have an avisynth for 64bits stable, faster than 32 bits and
reliable maybe I will think about it.
If I donnot remember wrong assembler codes in asmlib project are all
in yasm/nasm sintax it would be a good idea start looking at them without fear,
if you intend taking advantage of them. What a wonderful word is open source!

Thanks ARDA

[Reel.Deel]

The day we have an avisynth for 64bits stable, faster than 32 bits and reliable maybe I will think about it.


Sounds a lot like AviSynth+ r1576 :D.

[ARDA]

Source= 1920x1080 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe [91498] ANYONE = 2163 fps
VirtualDub.exe [92998] ANYONE = 2263 fps
VirtualDub.exe [94498] ANYONE = 2283 fps
VirtualDub.exe [95998] ANYONE = 2298 fps
VirtualDub.exe [97498] ANYONE = 2301 fps
VirtualDub.exe [98998] ANYONE = 2305 fps

Use type=2 GetThreadTimes
VirtualDub.exe [91498] ANYONE = 2595 fps
VirtualDub.exe [92998] ANYONE = 2667 fps
VirtualDub.exe [94498] ANYONE = 2400 fps
VirtualDub.exe [95998] ANYONE = 2233 fps
VirtualDub.exe [97498] ANYONE = 2909 fps
VirtualDub.exe [98998] ANYONE = 2182 fps

Use type=3 RDTSC
VirtualDub.exe [91499] ANYONE = 2047 fps
VirtualDub.exe [92999] ANYONE = 2272 fps
VirtualDub.exe [94499] ANYONE = 2267 fps
VirtualDub.exe [95999] ANYONE = 2259 fps
VirtualDub.exe [97499] ANYONE = 2265 fps
VirtualDub.exe [98999] ANYONE = 2237 fps

Source= 1920x1080 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
fvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe [91498] ANYONE = 4825 fps
VirtualDub.exe [92998] ANYONE = 4801 fps
VirtualDub.exe [94498] ANYONE = 4804 fps
VirtualDub.exe [95998] ANYONE = 4819 fps
VirtualDub.exe [97498] ANYONE = 4870 fps
VirtualDub.exe [98998] ANYONE = 4853 fps

Use type=2 GetThreadTimes
VirtualDub.exe [91498] ANYONE = 6000 fps
VirtualDub.exe [92998] ANYONE = 5333 fps
VirtualDub.exe [94498] ANYONE = 5053 fps
VirtualDub.exe [95998] ANYONE = 3840 fps
VirtualDub.exe [97498] ANYONE = 5647 fps
VirtualDub.exe [98998] ANYONE = 6400 fps

Use type=3 RDTSC
VirtualDub.exe [91498] ANYONE = 4830 fps
VirtualDub.exe [92998] ANYONE = 4823 fps
VirtualDub.exe [94498] ANYONE = 4817 fps
VirtualDub.exe [95998] ANYONE = 4829 fps
VirtualDub.exe [97498] ANYONE = 4851 fps
VirtualDub.exe [98998] ANYONE = 4843 fps
These tests were done to test different kind of methods to measure performance under avstimer
These tests shows that fvertical is around 100% faster, in fact we should say that new fvertical
in place is faster than internal avisynth bitblt, so it is not a fair comparation, but
this was the one of the objectives of this plugin, to get better performance for flip vertical
when posible.
More tests soon.

Thanks ARDA

[ARDA]

Source= 720x576 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
fvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe [91498] ANYONE = 30549 fps
VirtualDub.exe [92998] ANYONE = 30837 fps
VirtualDub.exe [94498] ANYONE = 30766 fps
VirtualDub.exe [95998] ANYONE = 30768 fps
VirtualDub.exe [97498] ANYONE = 30690 fps
VirtualDub.exe [98998] ANYONE = 30673 fps

Use type=2 GetThreadTimes
VirtualDub.exe [91498] ANYONE = 24000 fps
VirtualDub.exe [92998] ANYONE = 24000 fps
VirtualDub.exe [94498] ANYONE = 24000 fps
VirtualDub.exe [95998] ANYONE = 16000 fps
VirtualDub.exe [97498] ANYONE = 32000 fps
VirtualDub.exe [98998] ANYONE = 48000 fps

Use type=3 RDTSC
VirtualDub.exe [91498] ANYONE = 30725 fps
VirtualDub.exe [92998] ANYONE = 30835 fps
VirtualDub.exe [94498] ANYONE = 30936 fps
VirtualDub.exe [95998] ANYONE = 30952 fps
VirtualDub.exe [97498] ANYONE = 30488 fps
VirtualDub.exe [98998] ANYONE = 30806 fps
Source= 720x576 (Y8)
AvsTimer(frames=1000, name="ANYONE",type=0, frequency=1700, total=false, quiet=true)
flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=0, frequency=1700, difference=1, total=false)

Use type=0 QueryPerformaceCounter
VirtualDub.exe [91497] ANYONE = 9094 fps
VirtualDub.exe [92997] ANYONE = 12647 fps
VirtualDub.exe [94497] ANYONE = 12826 fps
VirtualDub.exe [95997] ANYONE = 13012 fps
VirtualDub.exe [97497] ANYONE = 12892 fps
VirtualDub.exe [98997] ANYONE = 12935 fp

Use type=2 GetThreadTimes
VirtualDub.exe [91498] ANYONE = 16000 fps
VirtualDub.exe [92998] ANYONE = 8727 fps
VirtualDub.exe [94498] ANYONE = 24000 fps
VirtualDub.exe [95998] ANYONE = 8727 fps
VirtualDub.exe [97498] ANYONE = 12000 fps
VirtualDub.exe [98998] ANYONE = 10667 fps

Use type=3 RDTSC
VirtualDub.exe [91498] ANYONE = 12403 fps
VirtualDub.exe [92998] ANYONE = 13077 fps
VirtualDub.exe [94498] ANYONE = 12927 fps
VirtualDub.exe [95998] ANYONE = 12930 fps
VirtualDub.exe [97498] ANYONE = 12985 fps
VirtualDub.exe [98998] ANYONE = 12993 fps
This resolution shows and increase performance of around 100%, is almost the same
condition than previous tests

[Groucho2004]

OK, I tested it properly now. In order to measure the very short time for each call of *vertical() I just call it several times. "colorbars()" is extremely fast and does not influence the results.

Here's the script:
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 4999)
#test_flipvertical()
#test_fvertical()

function test_flipvertical(clip c)
{
last = c
flipvertical().flipvertical().flipvertical().flipvertical().flipvertical()
flipvertical().flipvertical().flipvertical().flipvertical().flipvertical()
return last
}

function test_fvertical(clip c)
{
last = c
fvertical().fvertical().fvertical().fvertical().fvertical()
fvertical().fvertical().fvertical().fvertical().fvertical()
return last
}

flipvertical:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 195.1 | 201.1 | 199.1
Memory usage (phys | virt): 15 | 14 MB
Thread count: 1
CPU usage (average): 25%
Time (elapsed): 00:00:25.109

fvertical:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 687.9 | 736.4 | 730.7
Memory usage (phys | virt): 12 | 12 MB
Thread count: 1
CPU usage (average): 25%
Time (elapsed): 00:00:06.843

Very impressive!

[TheFluff]

now benchmark it against a bitblt that simply uses a memcpy from a modern runtime instead

I suspect the only reason that this possible to optimize is that Avisynth's ~optimized~ bitblt is an ancient piece of garbage written for P4's and ancient Athlons, which doesn't really produce great results on modern CPU's. Agner Fog's memcpy implementation was - by his own benchmarks - only barely faster than Microsoft's back in 2008 (https://sourceware.org/ml/libc-help/2008-08/msg00007.html). Replacing Avisynth's bitblt with a wrapper around memcpy and compiling with a modern runtime (haha, who am I kidding, this is Avisynth) would probably speed it up a lot.

[ARDA]

@Groucho2004
I like the way you have done to reduce the effect of colorbars.
Glad to see you find it impressive, but that result is a little bit deceiving cause
several things

1) Fvertical with that script is working in place it does not apply the new bitblt
While flipvertical is always using internal bitblt that has wrong bypasses
cache for your sandybridge that I guess has at least 4 mb of L3 cache
so it is doing a non temporal store to a new frame created

2) As Fvertical is working in place(on source), after two or three iterations
all data will be in cache and the speed improvement will be giant
Unluckily your results are not really true in real life where you never call
10 times flipvertical, except you are doing an special effect.

To make the test most trustable you should include any filter that makes the source
unwritable, but anyway in the second call of fvertical it will be writable again
and the benchmark result will not be real either.
According to my experience more than 120% of increase is almost impossible

Thank you for the time spent. ARDA

[ARDA]

@TheFluff
All thlis filter was done as an excuse to develop a new bitblt where Agner Fog's code are adapted and included to work in avisynth
If you take a look in the code you will find a dedicated code of 4500 assebly lines with many explanations in comments
I hope you like it

Thanks ARDA

[TheFluff]

If you take a look in the code you will find a dedicated code of 4500 assebly lines with many explanations in comments

that's genuinely horrifying

[ARDA]

Sad to see You don't like my code

Thanks anyway ARDA

[Groucho2004]

Fvertical with that script is working in place it does not apply the new bitblt
You're right. Using just a single call we're back to the results I already reported. On XP, fvertical is slower than flipvertical, possibly because AVX instructions are not supported.

[ARDA]

You're right. Using just a single call we're back to the results I already reported. On XP, fvertical is slower than flipvertical, possibly because AVX instructions are not supported.

There must something wrong in my code, something that deviate to the wrong soubrutine in your machine, you should get at least 100% of increase in performance with that resolution except you had included something in your script that makes the source non writable , in that case you should find
an increase around 15%. I will take a look and see what I find.

Thanks ARDA

[ARDA]

[Script]
blankclip(length = 9000, width = 5000, height = 3000, color=$005B8B).killaudio().assumefps(50, 1)
#flipvertical()
fvertical()

[Clip info]
Number of frames: 9000
Length (hh:mm:ss.ms): 00:03:00.000
Frame width: 5000
Frame height: 3000
Framerate: 50.000 (50/1)
Colorspace: RGB32

[Runtime info]
Frames processed: 9000 (0 - 8999)
FPS (min | max | average): 21.62 | 86.07 | 76.91
Memory usage (phys | virt): 127 | 126 MB
Thread count: 1
CPU usage (average): 23%
Time (elapsed): 00:01:57.021

with AvsTimer Use type=3 RDTSC
VirtualDub.exe [91499] ANYONE = 79 fps
VirtualDub.exe [92999] ANYONE = 78 fps
VirtualDub.exe [94499] ANYONE = 79 fps
VirtualDub.exe [95999] ANYONE = 80 fps
VirtualDub.exe [97499] ANYONE = 76 fps
VirtualDub.exe [98999] ANYONE = 78 fps
[Script]
blankclip(length = 9000, width = 5000, height = 3000, color=$005B8B).killaudio().assumefps(50, 1)
flipvertical()
#fvertical()

[Clip info]
Number of frames: 9000
Length (hh:mm:ss.ms): 00:03:00.000
Frame width: 5000
Frame height: 3000
Framerate: 50.000 (50/1)
Colorspace: RGB32

[Runtime info]
Frames processed: 9000 (0 - 8999)
FPS (min | max | average): 19.27 | 81.19 | 71.75
Memory usage (phys | virt): 127 | 126 MB
Thread count: 1
CPU usage (average): 23%
Time (elapsed): 00:02:05.428

with AvsTimer Use type=3 RDTSC
VirtualDub.exe [91497] ANYONE = 73 fps
VirtualDub.exe [92997] ANYONE = 74 fps
VirtualDub.exe [94497] ANYONE = 74 fps
VirtualDub.exe [95997] ANYONE = 74 fps
VirtualDub.exe [97497] ANYONE = 74 fps
VirtualDub.exe [98997] ANYONE = 75 fps
Finally I had some time to make this test with a clip 5000 x 3000 in RGB32
As you can see fvertical is still faster, in this case only beetween 4% and 7%,
but this result is coherent with the size of the frame, in fact 20000 x 3000 bytes
cause is RGB32 (4 bytes for each pixel).
Anyway it seems to me that something is wrong I will take a look in the code to see
if I can get a little bit more performance.Besides that Grocuho2004's maachine is giving me
some headache, let's see what I find.
It is probably that with bigger sizes you will have similar results. It seems that
the example chosen is one of the worst conditions for this new bitblt.
When bitblt is applied en fvertical, results are always between 5% to 25% increase
in performance, depending on the maachine script etc.., which is actually the real
improvemnt in the new bitblt.

[ARDA]

[Script]
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 8999)
#flipvertical()
fvertical()

[Clip info]
Number of frames: 9000
Length (hh:mm:ss.ms): 00:06:00.000
Frame width: 1920
Frame height: 1080
Framerate: 25.000 (25/1)
Colorspace: YV12

[Runtime info]
Frames processed: 9000 (0 - 8999)
FPS (min | max | average): 1090 | 2303 | 2077
Memory usage (phys | virt): 18 | 17 MB
Thread count: 1
CPU usage (average): 23%
Time (elapsed): 00:00:04.333
[Script]
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 8999)
flipvertical()
#fvertical()

[Clip info]
Number of frames: 9000
Length (hh:mm:ss.ms): 00:06:00.000
Frame width: 1920
Frame height: 1080
Framerate: 25.000 (25/1)
Colorspace: YV12

[Runtime info]
Frames processed: 9000 (0 - 8999)
FPS (min | max | average): 618.8 | 1766 | 1557
Memory usage (phys | virt): 18 | 17 MB
Thread count: 1
CPU usage (average): 23%
Time (elapsed): 00:00:05.780
As I have not a SandyBridge with Windows XP so I have built a version that donnot use
Avx instructions to see what happens in my machine, tested your script but
with 9000 frames and with your own tool avsmeter. The results I get are that fvertical
is around 25% faster. That makes suspect that my code is wrong and is using
the wrong subroutines in your machine. Investigating!

Thanks ARDA

[Groucho2004]

One more time for comparison:
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical()
#fvertical()


flipvertical: 3713
fvertical: 3261

[ARDA]

@Groucho2004

New bitblt is designed to detect some cpu characteristics and deviate to the
appropriate routines accordingly with the cpu on which is running.
After analyzing your scripts for the tests I gathered that Blankclip and colorbars
always return a non writable source, so with only one call fvertical will always apply
new bitblt, but when you call fvertical several times, from second time onward fvertical
will be done in place at least for normal resolutions(below 2 1/2 times largest cache ).
Then I can conclude definitely that there is a wrong selection of the routines for your
cpu within my new bitblt code. Something I will try to fix, but I need your help
for that. I hope you can give me the following data;

Cpu Vendor
Cpu Family
Cpu Model
Instructions set
Operating system
L1, L2, L3 sizes (think there is not L4 in your cpu)
Frequency(normal, and the one is running)

With that data I will try to know which routines should be used. Then if you agree
I will prepare some tests versions I will send you by PM to confirm and fix the selection
of routines for your cpu. If someone else has similar results Groucho20004 and a similar
machine,I ask you please put your results here and help us fix this bug

Thanks in advance. ARDA

[Groucho2004]

Cpu Vendor
Cpu Family
Cpu Model
Instructions set
L1, L2, L3 sizes
Info here (http://ark.intel.com/products/52210/Intel-Core-i5-2500K-Processor-6M-Cache-up-to-3_70-GHz)

Operating system
XP32_SP3

Frequency(normal, and the one is running)
3.3 GHz, oc to 4 GHz

[ARDA]

@Groucho2004
Thanks for the help, unluckily the intel page is a little poor in information
at least after a quick sight, but I think I'll manage with what I have.
While investigating problem with Groucho2004 cpu and my code I've made some small changes
that increase around 1% with some special care of prefetches values and loop size.
Please I ask everybody who has time, to test and post here the results
Version 1.01 Fvertcal.7z (https://mega.nz/#!UVtgiRAR!dwBcEE9h2VshPswxmz5wzrVZFnb7WfNV5X8OAXv7clY)

[Groucho2004]

Thanks for the help, unluckily the intel page is a little poor in information
at least after a quick sight, but I think I'll manage with what I have.
How about this (http://pastebin.com/FGtfi4i8)?

[ARDA]

Perfect thank you

[ARDA]

Working with the help of Groucho2004 we could figure out that the problem
of performance there is with new bilblt in his cpu was a wrong bypass cache
value, actually it was fixed in 50% of largest cache which is a thumb
rule, but avisynth gobbles up 1024kb of cache memory or more depending on
the script, that is why this value will be always a trade off
For all previously exposed I need volunteers to test those values in
different configurations; it is a boring task where many tests must be done.
(around 10).
The version to start test is 1.002
You can download dll in the link. New sources will be avaiable when version 1.1

Version 1.002 Fvertcal.dll (https://mega.nz/#!FFdW0ZRK!_stE39sFhzDjMlnB_vI7aEfHq8KEroH5adkxqRqfCr0)

[ARDA]

Some new results in Groucho2004's cpu
Intel i5-2500K (Sandy Bridge)3.3 GHz, oc to 4 GHz. L3 cache 6.0 MB
All the following tests were done with Fvertical version 1.002(see previous and first post)

#colorbars(width = 720, height = 576, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #15542 fps
#fvertical() #24528 fps

#colorbars(width = 1392, height = 992, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #5217 fps
#fvertical() #7322 fps

#colorbars(width = 1392, height = 1024, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #5053 fps
#fvertical() #5443 fps

#colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #3701 fps
#fvertical() #4069 fps

We shall still be tuning, but please I need volunteers;) I hope these results arouse the enthusiasm of good colleagues and come to help us to fine tuning this project.

Thanks in advance ARDA

[TurboPascal7]

Just wondering - have you ever tried measuring real-world performance improvements with your memcpy implementation? Maybe replacing internal BitBlt with it and running something more or less heavy like qtgmc?

[ARDA]

Just wondering - have you ever tried measuring real-world performance improvements with
your memcpy implementation? Maybe replacing internal BitBlt with it and running something
more or less heavy like qtgmc?


This is part of the first post of the thread

This new bitblt has only been tested in this plugin and others of my own use, never tested
deeply as a substitute of the internal one, only a few tests, so I cannot guarantee for now
full compatibility and free of bugs. It makes use of sse2, Ssse3 and avx instructions depending
on the machine on which is running.


To define real life performance is quite difficult , if we are measuring
bitblt-memcpy performance I have first to finish tuning most bypasses cache I can test,
with fvertical that in fact only test when a row loop is executed, other filters that make
direct use of bitblt or other special tools, that is why I am asking for volunteers to help.

I don't know qtgmc internals, maybe it does not use bitblt-memcpy at all, I should need
to study it. Anyway avisynth can create so complex and different chains that it is almost
imposible to preview and measure the use or not and/or the weight of bitblt-memcpy in general performance.
That could lead to conclude, is it worth to do faster assembler implementations?
Maybe better to leave compilers do the job, or maybe everytime we jump to a new
higher resolution we buy a new and faster cpu. There are a lot of academic discussion
all over in the net about it , I wouldn't like this thread becomes a discussion about
this subject. Mainly because it is more related to commercial politics of corporations
and ideology of Universities about how to prepare new software engineers
and definitely this is not the place for such a polemic.
I can only add that luckily old and some new avisynth developer always tried
to squeeze up the better performance possible in each step. I still remember
when my cpu was a thunderbird(2001?) with only Isse, and thanks to Steady-Sh0dan bitblt-memcpy
I could use it for many years .

Every filter or part of avisynth authors should take care of the performance of their work
Besides that when you measure the performance of one or another script, you will always
get a performance influenced by the slowest part of such script, which will hide the fastest part of it
Anyway it is quite obvious that bitblt-memcpy(weight??) is one of the steps that can influence in general
avisynth performance, but it is not the only one.

This implementation of bitblt-memcpy was done to use it in a plugin, for now
to integrate to avisynth it is a bit complex, cause there will be a lot of redundant parts
In spite of that I have made a few tests with a quick integration trying to see if
if I could detect some bugs, but din't do a deep research, it was not the purpose at this
stage of the development. Be patient, everything can arrive; nowadays Steady-Sh0dan bitblt
implementation has more than 10 years. Has anyone asked why so many years without
a new implementation?
For all above exposed for now I will focus to finish the work I am doing with Groucho2004
and other volunteers, and as I have few free time that will be enough for now.

I hope you find something usefull;)
Thanks ARDA

[Myrsloik]

I ported the optimized BitBlt in VapourSynth to avisynth and made a flippin' good plugin too!

Usage: FlipVFaster()

Link (https://dl.dropboxusercontent.com/u/73468194/FlipVFaster.7z)

[ARDA]

I ported the optimized BitBlt in VapourSynth and made a flippin' good plugin too!
Usage: FlipVFaster()


Happy to see you are trying to use it, anyway I will upload the new version, of
bitblt cause many bypasses cache has changed and you will be getting low performance
in some cpus. I don't have VapourSynth instaled so I cannot give you feedback in this case

This link is only new bitblt.asm for developers
Only BitBlt_SSE2_avs.asm (https://mega.nz/#!QQNxTKQQ!st_0m_gSkrnBABq46-oRKc6sK_J-omVaitRnRfZx3nY)

Soon I will upload Fvertical.dll V1.003 with more changes in cache bypass values
But I still need volunteers to test more cpus.

Thanks ARDA:rolleyes:

[TheFluff]

I don't have VapourSynth instaled so I cannot give you feedback in this case

Read the post. It's an Avisynth plugin. Benchmarks welcome.

[ARDA]

Read the post. It's an Avisynth plugin. Benchmarks welcome.

Sorry , quick read, I will do some benchmarks and publish here, but I donnot know if Myrsloik has updated BitBlt_SSE2_avs.asm. Let us wait
till he can confirm. I will also ask Myrsloik if he can release the code to see how he has done the integration, cause it is a little tricky.

@TheFluff
Maybe you can also post some benchmarks results, they will be welcome.:rolleyes:

Thanks ARDA

[Myrsloik]

Sorry , quick read, I will do some benchmarks and publish here, but I donnot know if Myrsloik has updated BitBlt_SSE2_avs.asm. Let us wait
till he can confirm. I will also ask Myrsloik if he can release the code to see how he has done the integration, cause it is a little tricky.

@TheFluff
Maybe you can also post some benchmarks results, they will be welcome.:rolleyes:

Thanks ARDA

I didn't touch any of your asm. It's all original code from me.

[ARDA]

I didn't touch any of your asm. It's all original code from me
I was asking if you updated the asm code with the link in this post ,it was specially released because of you
http://forum.doom9.org/showthread.php?p=1736739#post1736739
Besides that if you take a look to fvertical.cpp you will see that it need some extra calls previously like.

Pass_sizeframe((dstframesize)+(dstframesizeU)+(dstframesizeU));

I will publish a whole explanation about the architecture of this new bitblt and how to use it.

Thanks ARDA

[ARDA]

@all developers

This version of bitblt-memcpy is still under heavy development, mainly cause the bypass values
relatives to each cpu must be fixed manually, I never could find a general formula, I would
ask to wait and help me with your cpus to find most exact possible value by doing some tests
like Groucho2004 has done. I release here fviertical. dll version 1.003 that I think it quite
near to be definitive(do benchmaks with it), after that I will release source code again

@all developers and users
Version 1.003 Fvertcal.dll (https://mega.nz/#!1EMXRZoS!6co0as92JfKA-ot5xyOY0a_sz3ugtsWCFAPi2AuZyO0)

Thanks ARDA

[ARDA]

To discuss about bitlt-memcpy specifically please follow this new thread

http://forum.doom9.org/showthread.php?p=1736845#post1736845

And we stay here with all related with fvertical
Thnaks ARDA

[TheFluff]

I was asking if you updated the asm code with the link in this post ,it was specially released because of you
http://forum.doom9.org/showthread.php?p=1736739#post1736739
Besides that if you take a look to fvertical.cpp you will see that it need some extra calls previously like.

Pass_sizeframe((dstframesize)+(dstframesizeU)+(dstframesizeU));

I will publish a whole explanation about the architecture of this new bitblt and how to use it.

Thanks ARDA

I guess we have to explain the joke since it apparently went completely over your head. Myrsloik hasn't looked at your code at all, he wrote his own flipvertical implementation using Vapoursynth's bitblt, which looks like this:
static inline void vs_bitblt(void *dstp, int dst_stride, const void *srcp, int src_stride, size_t row_size, size_t height) {
if (height) {
if (src_stride == dst_stride && src_stride == (int)row_size) {
memcpy(dstp, srcp, row_size * height);
} else {
const uint8_t *srcp8 = (const uint8_t *)srcp;
uint8_t *dstp8 = (uint8_t *)dstp;
for (size_t i = 0; i < height; i++) {
memcpy(dstp8, srcp8, row_size);
srcp8 += src_stride;
dstp8 += dst_stride;
}
}
}
}
It's probably faster than Avisynth 2.6's implementation and insignificantly slower than yours. If you were sane, you would have benchmarked that, but I don't think you have.

[jpsdr]

I guess we have to explain the joke since... It's probably faster than Avisynth 2.6's implementation...

You've copied what i've done i nnedi... :D

inline void PlanarFrame::BitBlt(uint8_t *dstp,int dst_pitch,const uint8_t *srcp,int src_pitch,int row_size,int height)
{
if ((height==0) || (row_size==0)) return;

if ((height==1) || ((dst_pitch==src_pitch) && (src_pitch==row_size))) A_memcpy(dstp,srcp,(size_t)src_pitch*(size_t)height);
else
{
for (int y=0; y<height; y++)
{
A_memcpy(dstp,srcp,row_size);
dstp+=dst_pitch;
srcp+=src_pitch;
}
}
}


But you missed the case of height=1.
.... Ok, i'm going out... :p

[jmac698]

fvertical:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 98.98 | 159.3 | 145.9
Memory usage (phys | virt): 16 | 12 MB
Thread count: 1
CPU usage (average): 49%

Time (elapsed): 00:00:34.272

flipvertical:
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 18.97 | 49.62 | 45.19
Memory usage (phys | virt): 19 | 15 MB
Thread count: 1
CPU usage (average): 49%

Time (elapsed): 00:01:50.644

323% speedup, very impressive!

t7300@2GHz
fam 6 model f step b
mmx,sse,sse2,sse3,ssse3
l1 2x32k l2 4M
2c2t

benchmark as in
http://forum.doom9.org/showthread.php?p=1736148#post1736148

[jmac698]

AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 15.85 | 49.96 | 45.00
Memory usage (phys | virt): 23 | 23 MB
Thread count: 3
CPU usage (average): 49%

Time (elapsed): 00:01:51.121

same speed as AviSynth 2.60, build:Mar 31 2015 [16:38:54] (2.6.0.6)

[jmac698]

flipvfaster, avisynth+ (and all were win8.1 x64)
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 18.00 | 48.78 | 40.75
Memory usage (phys | virt): 22 | 23 MB
Thread count: 3
CPU usage (average): 49%

Time (elapsed): 00:02:02.707

so, guess TheFluff is wrong, and this new bitblit is far superior to anything existing

also I'm getting a bug in flipvfaster, the V channel is always 0.

[ARDA]

@jmac698
Thanks for your contribution, it is very important, mainly after this litlle storm that
has unfocus me from the task we have here, that is, for now to fix the bypass cache values for
any different architecture and configuration. But we must consider that these are tests that
sometimes measure some fvertical codes that work in place, against internal flipvertical that in
all cases uses bitblt-memcpy, and in the majority of cases by using backward block copy techniques
and finished with non temporal store, why all that? cause steady-Sh0dan bitblt-memcpy were written
in a time when most of cpus had a L2 of 256kb or 512kb(maximum) and the bypasses value were fixed for that
largest cache sizes, which made non temporal store the best option almost always and the hardware
predictors were not as good as are in nowadays machines, and by doing backward, bitblt avoided that
destination buffer were loaded in cache and the correspondant penalizations, cause caches lines
continous flushing.
For all that I ask you a little more effort by doing the following test, you can use this
script in which please post back your results, those frame sizes chosen are just examples
you can do any others if you want, but it is probably I will be asking you for two or three
more tests after analyzing your results:rolleyes:


# Insert here
# Vendor ,Family ,Model ,instruction set ,Largest cache

#test320x256yv12
#colorbars(width = 320, height = 256, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test448x320yv12
#colorbars(width = 448, height = 320, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test640x480yv12
#colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test720x576yv12
#colorbars(width = 720, height = 576, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test1392x992yv12
#colorbars(width = 1392, height = 992, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test1392x1024yv12
#colorbars(width = 1392, height = 1024, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test1920x1080yv12
#colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 99999)
#flipvertical() #..... fps
#fvertical() #..... fps

#test5000x3000RGB32
#colorbars(width = 5000, height = 3000, pixel_type = "rgb32").killaudio().assumefps(25, 1).trim(0, 9999)
#flipvertical() #..... fps
#fvertical() #..... fps

Thanks in advance ARDA:thanks:

[Myrsloik]

Did I really manage to put a bug in that simple code?

Here's an updated FlipVFaster (https://dl.dropboxusercontent.com/u/73468194/FlipVFaster2.7z). Don't get scared when you see the speed. Any previous speed tests done on subsampled planar formats were wrong in the previous version. So only HolyWu's tests show the objective truth.

[ARDA]

I simply pick 720p, 1080p and 4K resolution for common use cases. As FlipVFaster doesn't process
V plane correctly, I use a Y8 clip for benchmark.

#ColorBars(width=1280, height=720, pixel_type="YV12").ConvertToY8().KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # your cpu 10693 fps #my cpu 6456
#FVertical() # your cpu 15782 fps #my cpu 9108
#FlipVFaster(newbitblt=true) # your cpu 18281 fps #my cpu 8645

#ColorBars(width=1920, height=1080, pixel_type="YV12").ConvertToY8().KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # your cpu 4624 fps #my cpu 2726
#FVertical() # your cpu 7384 fps #my cpu 3484
#FlipVFaster(newbitblt=true) # your cpu 8619 fps #my cpu 3107

#ColorBars(width=3840, height=2160, pixel_type="YV12").ConvertToY8().KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
#FlipVertical() # your cpu 768.9 fps #my cpu 520.4
#FVertical() # your cpu 727.6 fps #my cpu 597.4
#FlipVFaster(newbitblt=true) # your cpu 681.9 fps #my cpu 536.3
Your results are surprising for me and not fully coherents, that is why I will
prepared a special release taking into account your machine, if you agree I will
send you by pm. It seems to me that is the same problem I had with Groucho2004 cpu
around the bypass cache value.
And in the meanwhile I ask you please, to do the tests again
according to next post

Thanks ARDA;)

[ARDA]

In post http://forum.doom9.org/showthread.php?p=1735823#post1735823 there is a proposal of
a method to do benchmarks without the influence of other parts of the script.

Here is the script I use

MPEG2Source("xxxxxx.d2v")#720,576 or any other form of real clip source
crop(8,72,700,432,align=false)
ConvertToY8()
#LanczosResize(1920,1080)
#LanczosResize(1280,720)
#LanczosResize(3840,2160)
AvsTimer(frames=1000, name="ANYONE",type=3, frequency=1700, total=false, quiet=true)
#fvertical()
#FlipVFaster(newbitblt=true)
#flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=1700, difference=1, total=false)

An here the results in my cpu with the same sizes

1280,720
VirtualDub.exe [91471] FlipVFaster = 4591 fps
VirtualDub.exe [94471] FlipVFaster = 5761 fps
VirtualDub.exe [95971] FlipVFaster = 5858 fps
VirtualDub.exe [97471] FlipVFaster = 5887 fps
VirtualDub.exe [98971] FlipVFaster = 5837 fps

VirtualDub.exe [91498] fvertical = 12431 fps
VirtualDub.exe [92998] fvertical = 12927 fps
VirtualDub.exe [94498] fvertical = 12818 fps
VirtualDub.exe [95998] fvertical = 12800 fps
VirtualDub.exe [97498] fvertical = 12823 fps
VirtualDub.exe [98998] fvertical = 12749 fps

VirtualDub.exe [91498] flipvertical = 5546 fps
VirtualDub.exe [92998] flipvertical = 5691 fps
VirtualDub.exe [94498] flipvertical = 5751 fps
VirtualDub.exe [95998] flipvertical = 5756 fps
VirtualDub.exe [97498] flipvertical = 5693 fps
VirtualDub.exe [98998] flipvertical = 5672 fps

1920,1080
VirtualDub.exe [91498] FlipVFaster = 2188 fps
VirtualDub.exe [92998] FlipVFaster = 2241 fps
VirtualDub.exe [94498] FlipVFaster = 2229 fps
VirtualDub.exe [95998] FlipVFaster = 2232 fps
VirtualDub.exe [97498] FlipVFaster = 2258 fps
VirtualDub.exe [98998] FlipVFaster = 2237 fps

VirtualDub.exe [91498] fvertical = 4702 fps
VirtualDub.exe [92998] fvertical = 4778 fps
VirtualDub.exe [94498] fvertical = 4740 fps
VirtualDub.exe [95998] fvertical = 4765 fps
VirtualDub.exe [97498] fvertical = 4849 fps
VirtualDub.exe [98998] fvertical = 4783 fps

VirtualDub.exe [91498] flipvertical = 2213 fps
VirtualDub.exe [92998] flipvertical = 2298 fps
VirtualDub.exe [94498] flipvertical = 2283 fps
VirtualDub.exe [95998] flipvertical = 2286 fps
VirtualDub.exe [97498] flipvertical = 2298 fps
VirtualDub.exe [98998] flipvertical = 2254 fps

3840,2160
VirtualDub.exe [91499] FlipVFaster = 498 fps
VirtualDub.exe [92999] FlipVFaster = 511 fps
VirtualDub.exe [94499] FlipVFaster = 512 fps
VirtualDub.exe [95999] FlipVFaster = 513 fps
VirtualDub.exe [97499] FlipVFaster = 512 fps
VirtualDub.exe [98999] FlipVFaster = 508 fps

VirtualDub.exe [91499] fvertical = 552 fps
VirtualDub.exe [92999] fvertical = 558 fps
VirtualDub.exe [94499] fvertical = 568 fps
VirtualDub.exe [95999] fvertical = 568 fps
VirtualDub.exe [97499] fvertical = 567 fps
VirtualDub.exe [98999] fvertical = 568 fps

VirtualDub.exe [91499] flipvertical = 468 fps
VirtualDub.exe [92999] flipvertical = 478 fps
VirtualDub.exe [94499] flipvertical = 480 fps
VirtualDub.exe [95999] flipvertical = 478 fps
VirtualDub.exe [97499] flipvertical = 479 fps
VirtualDub.exe [98999] flipvertical = 478 fps

As you can see, the results differ a lot from one kind of benchmar to another
I don't want to open a new polemic about benchmark methods, but just to confirm if you
also have differents results in your cpu:eek:

Thanks ARDA

[Reel.Deel]

Here's my results using AviSynth+ r1576, FVertical v1.003, and the updated FlipVFaster:

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # 4601 fps
#FVertical() # 9359 fps
#FlipVFaster(newbitblt=true) # 10160 fps

ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # 1938 fps
#FVertical() # 4305 fps
#FlipVFaster(newbitblt=true) # 4756 fps

ColorBars(width=3840, height=2160, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
#FlipVertical() # 468.3 fps
#FVertical() # 627.3 fps
#FlipVFaster(newbitblt=true) # 577.3 fps


http://s13.postimg.org/h5nnelwyv/i7_4930k.png

[ARDA]

Here's my results using AviSynth+ r1576, FVertical v1.003, and the updated FlipVFaster
It seems the same situation in holywu cpu, luckily now I have data to fix bypass cache values. In the meantime if you have time, can you please do the benchmarks I asked holywu in the post previous yours.http://forum.doom9.org/showthread.php?p=1736973#post1736973


thanks in advance ARDA:rolleyes:

[ARDA]

@all

As I mentioned before, this bitblt-memcpy is still under heavy development, till I can fix all values in the TableByPass (at the end of BitBlt_SSE2_avs) benchmarks value must be considered provisory and as a reference for analysis; That is why I invite more users to join us as volunteers so we shall finish this quickly.:eek:

Thanks ARDA;)

[jmac698]

I had trouble getting avstimer to work. Here's how to do it:
http://www.avstimer.de.tf/ doesn't load for me. I got a version from
http://www.avisynth.nl/users/warpenterprises/
which needs
http://www.avsrecursion.de.tf/
which needs
http://www.dll-files.com/dllindex/dll-files.shtml?msvcr71

(the latter two need to be put in C:\Windows\SysWOW64 for windows 8.1 x64)
otherwise, you get this error:
Platform return code 126: The specified module could not found

Then I tried to open the benchmark script in virtualdub, but got "avi import error:(Unknown) (80040154)"
I had to reinstall avisynth. I think the problem was that uninstalling avisynth+ didn't return my system to using avisynth.

Now I'll try to get some benchmarks.

[ARDA]

@jmac698

sorry you you've gone into so many troubles, if you continue having problems with avstimer I think
i have statically compiled version that needs nothing to be used.

Thanks a lot for you effort, waiting results ARDA

[jmac698]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#FlipVertical() # 1243 fps
#FVertical() # 3201 fps
#FlipVFaster(newbitblt=true) # 1210 fps

Arda wins still.

I'm not getting very accurate timing, for example:
[9180] [1499] ANYONE = 1115.2 fps
[9180] [2999] ANYONE = 1208.7 fps
[9180] [4499] ANYONE = 1218.0 fps
[9180] [5999] ANYONE = 1204.1 fps
[9180] [7499] ANYONE = 1216.1 fps
[9180] [8999] ANYONE = 1210.7 fps


AviSynth 2.60, build:Mar 31 2015
win8.1 x64
T7300@2GHz
sse,sse2,sse3,ssse3
L1 32k
L2 4M

[jmac698]

ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#FlipVertical() # 474 fps
#FVertical() # 500 fps
#FlipVFaster(newbitblt=true) # 471 fps
avisynth 2.6
2ghz
l2 4M

[ARDA]

I'm not getting very accurate timing, for example:
[9180] [1499] ANYONE = 1115.2 fps
[9180] [2999] ANYONE = 1208.7 fps
[9180] [4499] ANYONE = 1218.0 fps
[9180] [5999] ANYONE = 1204.1 fps
[9180] [7499] ANYONE = 1216.1 fps
[9180] [8999] ANYONE = 1210.7 fps
If you are using blanclip as source, try to change it with a real clip
source , somthing like this


MPEG2Source("anyone.d2v") # or any other real clip source
#LanczosResize(1920,1080)#use the sizes you used in test with #avsmeter
#LanczosResize(1280,720)
#LanczosResize(3840,2160)
AvsTimer(frames=1000, name="ANYONE",type=3, frequency=???, total=false, quiet=true)
#fvertical()
#flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=????, difference=1, total=false)

Be carefull with the frequency value, it is a divisor in avstimer math

Thank you again and keep on ARDA:rolleyes:

[jmac698]

ColorBars(width=3840, height=2160, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
#FlipVertical() # 126 fps
#FVertical() # 204 fps
#FlipVFaster(newbitblt=true) # 112 fps
interesting, 1080p wasn't much speedup but the others were.

[Groucho2004]

I'm not getting very accurate timing
If you really have to use AVSTimer, read the documentation carefully, otherwise you'll likely to get completely wrong results.

ColorBars() is a perfectly good substitute for a "real" clip, especially since it's very fast and therefore does not influence the result when using AVSMeter. Try just running ColorBars() without any filters, you'll see what I mean. Even on my ancient laptop (Core 2 Duo, 2.5 GHz) 1 frame takes about 1 microsecond (1000000 fps).

[ARDA]

@Grocuho2004

I beg you please, donnot misunderstand my request from jmac698 as if I don't trust on your
tool, nothing more far than my intention, but colorbars and blankclip always return a non
writable src which forces the use of bitblt, temporal or non temporal, and the results that
jmac698 has provided with avsmeter and colorbars always show that my code is fastest always,
I don't need anymore tests like that, it seems that the bypasses for a machine with L2 4Mb and
S_sse3 in biltblt are correct. With my request to do sometest with avstimer, I am trying to test
the other part of this plugin when eventually it can work in place, thing that is not too frequent,
but the work is already done, and I need of some more bypasses are in the project are working ok.
I hope my explanation is good enough for you:rolleyes:

Thanks ARDA

[Groucho2004]

@Grocuho2004

I beg you please, donnot misunderstand my request from jmac698 as if I don't trust on your
tool, nothing more far than my intention, but colorbars and blankclip always return a non
writable src which forces the use of bitblt, temporal or non temporal, and the results that
jmac698 has provided with avsmeter and colorbars always show that my code is fastest always,
I don't need anymore tests like that, it seems that the bypasses for a machine with L2 4Mb and
S_sse3 in biltblt are correct. With my request to do sometest with avstimer, I am trying to test
the other part of this plugin when eventually it can work in place, thing that is not too frequent,
but the work is already done, and I need of some more bypasses are in the project are working ok.
I hope my explanation is good enough for you:rolleyes:

Thanks ARDA
OK, 'nuff said.:D

[TheFluff]

So, thus far we've proven that the new bitblt probably isn't faster (or if it is, insignificantly so) and that you're trying to optimize a filter that even in the case where it has to copy the frame runs at around 10,000 fps (correct to an order of magnitude). Do you really use flipvertical() hundreds of times in a script? Because if you don't, this optimization is rather meaningless despite that impressive-sounding claim that you've doubled the performance. Is that really worth several thousand lines of assembler when it can be done with two dozen lines of C++? There's clarity, maintainability and correctness to consider as well.

Thus spake Donald Knuth (peace be upon him):
The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.
He said this in 1974 and it's just as true today, if not more so. Get with the times, already.

[jmac698]

I've used a lot of blits in scripts before, such as this one http://forum.doom9.org/showthread.php?t=158230
It needs several bitblts per frame

[Myrsloik]

I'm back with another updated FlipVFaster (https://dl.dropboxusercontent.com/u/73468194/FlipVFaster3.7z).

I've added a new function called FlipVFastest in addition to FlipVFaster. It will knock your socks off.

[TheFluff]

I'm back with another updated FlipVFaster (https://dl.dropboxusercontent.com/u/73468194/FlipVFaster3.7z).

I've added a new function called FlipVFastest in addition to FlipVFaster. It will knock your socks off.

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#flipvertical() # 7085 fps
#fvertical() # 10922 fps
#flipvfaster(newbitblt=true) # 11517 fps
#flipvfastest() # 1517663 fps


http://i.imgur.com/dHoYhsv.png

Measured with AVSMeter. No, that isn't a troll or a typo, flipvfastest really is two orders of magnitude faster.

[jmac698]

flip fastest is giving me 130,000fps - but I don't believe it, obviously. In reality it's twice as slow as flip faster. I'm going back to avsmeter.

[TheFluff]

In reality it's twice as slow as flip faster.

How did you measure this?

[jmac698]

@fluff
fps in virtualdub was much worse, but that test was invalid.

avsmeter

fvertical: 52.28
flipvertical: 42.94
flipvfastest: 52.71

avstimer
fvertical: 653
flipvertical: 314
flipvfastest: 25212

I changed the script a bit:
#Create a red square that moves
red_yuv=$4164D4#in REC 601
#Make a moving red square
global square=blankclip(length=1,width=50,height=50,pixel_type="YUY2",fps=24000, fps_denominator=1001, color_yuv=red_yuv,sixteen_bit=true)
background=ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
ScriptClip(background,"""
overlay(square,x=103-25+(current_frame%160)*8,y=240-25)
""")
AvsTimer(frames=1000, name="ANYONE",type=3, frequency=2000, total=false, quiet=true)
fvertical()
#FlipVFastest()
#flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=2000, difference=1, total=false)
avsmeter ran instantly with just colorbars

[TheFluff]

I changed the script a bit:
#Create a red square that moves
red_yuv=$4164D4#in REC 601
#Make a moving red square
global square=blankclip(length=1,width=50,height=50,pixel_type="YUY2",fps=24000, fps_denominator=1001, color_yuv=red_yuv,sixteen_bit=true)
background=ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
ScriptClip(background,"""
overlay(square,x=103-25+(current_frame%160)*8,y=240-25)
""")
AvsTimer(frames=1000, name="ANYONE",type=3, frequency=2000, total=false, quiet=true)
fvertical()
#FlipVFastest()
#flipvertical()
AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=2000, difference=1, total=false)
avsmeter ran instantly with just colorbars
with that kind of script+avsmeter you're basically benchmarking the avisynth script interpreter, not flipping the image

avsmeter ran instantly with just colorbars
well yes that will kinda happen when your only "filtering" is something capable of running at literally a million fps

e: just use a longer clip, tack on .loop() to the end of the source line:
AVSMeter 2.1.1 (x86)
AviSynth+ 0.1 (r1576, x86) (2.6.0.5)

Number of frames: 10000000
Length (hh:mm:ss.ms): 111:06:40.000
Frame width: 1280
Frame height: 720
Framerate: 25.000 (25/1)
Colorspace: YV12

Frames processed: 10000000 (0 - 9999999)
FPS (min | max | average): 567002 | 1544474 | 1454652
Memory usage (phys | virt): 16 | 13 MB
Thread count: 1
CPU usage (average): 24%

Time (elapsed): 00:00:06.874

[jmac698]

now with 4k frames:

avstimer
fvertical: 89.1
flipvertical: 88.4
flipvfastest: 75078

ok at this point I'm suspicious, maybe it's only doing the calculation if something after it is actually using the data, for example I got this to run faster if I followed it with pointresize(960,540) (113,000fps). This is why I made the moving square, just to make sure it wasn't "cheating". Now if I start with 1920 and resize to 3840, it slows down to 90,000fps.

I'm not really sure what's going on.

[jmac698]

@fluff:

yes you're right about avsmeter, however I kept adding to the length until it exceeded the maximum possible value and it was still instant, so I didn't pursue it. I'll see if I can recreate your example.

[jmac698]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999999)
#fvertical()
FlipVFastest()
#flipvertical()
loop(100)
326,000fps with avsmeter

but being able to measure an essentially null function isn't really the problem, I think it's just being smart about not doing anything to parts of the frame that aren't being used further.

If that's the case, the benchmarking has to be improved. If it's not the case, my apologies and congrats for making an amazingly fast filter :)

[jmac698]

compare

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#fvertical()
FlipVFastest()
flipvertical()

loop(10)

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#fvertical()
#FlipVFastest()
flipvertical()

loop(10)


1: 1650fps
2: 1423fps

so it is taking a measurable amount of time

[AzraelNewtype]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999999)
#fvertical()
FlipVFastest()
#flipvertical()
loop(100)
326,000fps with avsmeter

but being able to measure an essentially null function isn't really the problem, I think it's just being smart about not doing anything to parts of the frame that aren't being used further.

If that's the case, the benchmarking has to be improved. If it's not the case, my apologies and congrats for making an amazingly fast filter :)

Why exactly are you putting loop() after the flipping is already done?

[jmac698]

Nothing to worry about, just leftover code from when I was trying to measure something that was too fast. All it does is multiply the frame count.

[AzraelNewtype]

Nothing to worry about, just leftover code from when I was trying to measure something that was too fast. All it does is multiply the frame count.

It multiplies the frame count, but not the calls to the functions you're benchmarking. At least, it doesn't if you put it there. TheFluff suggested appending it to the source for a reason.

[jmac698]

duhhh yes of course

[Groucho2004]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999999)
#fvertical()
FlipVFastest()
#flipvertical()
loop(100)
326,000fps with avsmeter

but being able to measure an essentially null function isn't really the problem, I think it's just being smart about not doing anything to parts of the frame that aren't being used further.
I have a hunch that the plugin is cheating somehow and Myrsloik is having some fun but without seeing the source code...

By the way, colorbars() creates a clip with ~107000 frames, your ".Trim(0, 9999999)" is ignored.

[Myrsloik]

I have a hunch that the plugin is cheating somehow and Myrsloik is having some fun but without seeing the source code...

By the way, colorbars() creates a clip with ~107000 frames, your ".Trim(0, 9999999)" is ignored.

I'm definitely having fun, yes. But you can't prove the output is wrong. I win, you lose!

Hint: The code is even shorter now

[jpsdr]

If flipvertical swaps top/bottom, the fastest way is just to change the pointer adress to the bottom of picture, and reverse the value of the pitch. It works like this in VDub if you're using old/original interface : src is guarantee to be the top of the picture, but never said it's the top of the data, and never assume that pitch is always positive.
If Myrsloik is using a trick like this, it's possible that if you put after his pluggin another pluggin that assume that pitch is always positive, you may create a crash. Now, if in avisynth it's a rule that pitch is always positive, and Myrsloik is doing something like this, finaly ouput is wrong (if it's the case, i win... :D).
To check it, you should create a pluggin which does nothing, but test if pitch is negative

[Myrsloik]

That's an interesting thought. But I invoke the universal law of API usage.

"If the API doesn't block it it's fine"

[jpsdr]

I think i'll check and probaby update nnedi to not assume that pitch is always positive. I'm always doing that i VDub filters, but i don't know when or where, in doom9 i was told that for avisynth pitch is always positive, so, i assumed it was an avisynth rule/standard. If finaly it's not the case, i personnaly don't mind (in fact, from my point of view it's more logical), and check and update my avisynth pluggins to not think that pitch is always positive.

About VDub, in the old times when only RGB32 was supported, data was stored in bmp format, so the top of data was the bottom of picture, and also src was. In the last VDub API, for me it's more logical, src is always the top of picture, whatever how data are stored, so pitch can be either positive or negative. That's why pitch always have been a ptrdiff_t.

So finaly, pluggin is not cheating, it just being smart and "worrying about efficiency in the right places and at the right times"

[wonkey_monkey]

If flipvertical swaps top/bottom, the fastest way is just to change the pointer adress to the bottom of picture, and reverse the value of the pitch. It works like this in VDub if you're using old/original interface : src is guarantee to be the top of the picture, but never said it's the top of the data, and never assume that pitch is always positive.

Interesting idea, but isn't GetReadPtr defined as returning the start of the entire frame in memory, i.e. its first (lowest address) byte? Not as the first byte of the "first" line...

Anyone who was expecting to use memcpy, for example, to copy a few lines at a time would find their plugin broken.

The API docs go as far as to suggest this:

int buffer_size = src->GetPitch() * src->GetHeight(); //YUY2, interleaved

to calculate frame size in bytes, so it seems a positive pitch is at least expected to be safe to assume.

[Myrsloik]

Interesting idea, but isn't GetReadPtr defined as returning the start of the entire frame in memory, i.e. its first (lowest address) byte? Not as the first byte of the "first" line...

Anyone who was expecting to use memcpy, for example, to copy a few lines at a time would find their plugin broken.

The API docs go as far as to suggest this:



to calculate frame size in bytes, so it seems a positive pitch is at least expected to be safe to assume.

There's very little real reference documentation for Avisynth and the original intentions are most likely lost. In most other frameworks negative pitch is totally a thing and there's no reason to think otherwise here. I believe all internal functions handle negative pitch at least.

Avisynth also actively discourages memcpy since it provides BitBlt which does handle it properly. Besides, if you do a proper check before memcpy you'd find that src_pitch != dst_pitch so you'll have to go line by line anyway. (this happens if your filter is after cropping so you have to support that properly anyway)

Anyway, if it's not allowed YOU MUST MAKE THE API REJECT IT. Simple rule. If it's accepted and passed on you'll simply get someone else's plugin blamed for crashes.

[jpsdr]

For fun and personnal usage, i've created a kind of "pitcure/image" class. I provide several informations : The start/top of picture, the start/top of the whole picture (in case subframe/cropping was created) and the start of the frame in memory. The last one can be different of previous, if it's a BMP memory data which is created.


void * AddrStartCurrentImage(void) const {return(ImageAddr);}
void * AddrStartFullImage(void) const {return(ImageAddr0);}
void * AddrStartFullData(void) const {return(DataAddr);}


So, i personnaly totaly agree with Myrsloik point of view.

[Myrsloik]

I think there's one test missing in this flip comparison. Force line by line copy.

For example:
ColorBars().Crop(a bit to the left and right).Flip()

Then you can't simply copy one huge region anymore. I suspect it will result in similar results but you never know...

Obviously I mean memcpy comparison.

[mawen1250]

E3-1230v2
Windows7 x64
AviSynth+ r1576

ColorBars(1920, 1080, "YV12") #7097595fps

#FlipVertical() # 3109fps
#FVertical() # 4000fps
#FlipVFaster() # 4600fps
#FlipVFastest() # 2830798fps
Every zero-copy filter will reduce the speed to about half, and here's the case of minimal function calls.

[ARDA]

With the help of Ree.Deel we'got to fix the performance of the new bitblt in a range
of values (sizeframe) when temporal store is chosen, this should be better for any cpu with 8 mb
of L3 cache (HollyWu's one) or larger with avx. I need more volunteers please, but nowadays
I would like to see any result in AMD machines, nobody has posted one if I am not wrong. Those who has
old cpus but are SSE2 capable can also post results. This project is intended to have
backward compatibility for all. It is not a competition between machines. It is an
open source project to be shared.
Link to download here or in the first post
Version 1.004 Fvertcal.dll (https://mega.nz/#!hVtShDAY!305YLuXYMoL_pkbiXRSvudnh30i2KKuEBK8knhC68ug)
I hope you find it usefull ARDA

[ARDA]

Here's my results using AviSynth+ r1576, FVertical v1.003, and the updated FlipVFaster:

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # 4601 fps
#FVertical() # 9359 fps
#FlipVFaster(newbitblt=true) # 10160 fps

ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FlipVertical() # 1938 fps
#FVertical() # 4305 fps
#FlipVFaster(newbitblt=true) # 4756 fps

ColorBars(width=3840, height=2160, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
#FlipVertical() # 468.3 fps
#FVertical() # 627.3 fps
#FlipVFaster(newbitblt=true) # 577.3 fps

And now a comparation between Fvertical versions 1.003 and 1.004(both in first post)


Here you go:

#size frame 1382400, bypass 5033168, tpa store in 12mb L3 cache
#ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FVertical() # 9359 fps version 1.003 # 10956 fps version 1.004

#size frame 3110400, bypass 5033168, tpa store in 12mb L3 cache
#ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#FVertical() # 4305 fps version 1.003 # 4996 version 1.004

Nice improvement :)

If anyone has had bad results with fvertical with these same resolutions, I beg you please repeat with
version 1.004 (see first post) and post here. Soon version 1.005
Thanks in advance ARDA

[ARDA]

Small changes, some improvemnt when non temporal row loops in avx machines or up
Still need volnteers to benchmark some AMD machines
Enjoy
Version 1005 Fvertical.dll (https://mega.nz/#!RFdRHaKI!qa15JjN58CuiPlIAu2m6H-sJFDwkfphAXq5ghNg-Qh8)

Thanks ARDA

[ARDA]

With the colaboration of many members of this forum, I dared to do important changes
Most of cache bypass values have been adjusted, some controls had been avoided, I hope not to have
introduced many bugs.
Still need volnteers to benchmark some AMD machines
Enjoy
Version 1006 Fvertical.dll (https://mega.nz/#!8ItFARgY!lvb8Vv5qjJurghNQ8JLaLrTb4K29m-5FeASHKVaShbY)

If you want to test more options about the new bitblit, follow this link with a new toy to
play with
NewTool ( http://forum.doom9.org/showthread.php?p=1740869#post1740869)

Thanks ARDA

[jmac698]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#fvertical()# 1523 fps
flipvertical()# 1740
oops some probs here


ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
fvertical()# 589.6 fps
#flipvertical()# 580
better

avisynth 2.606

[ARDA]

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 99999)
#fvertical()# 1523 fps
flipvertical()# 1740
oops some probs here

Most of cache bypass values have been adjusted, some controls had been avoided, I hope not to have
introduced many bugs.

Yes I have introduced some bug, please test version 1.003 and 1.004
from the first post.
Thanks ARDA;)

[jmac698]

ok

#ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#fvertical()# 589.6 fps 1.004
#fvertical()# 608.4 fps 1.006
#flipvertical()# 580

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
fvertical()# 4296 fps 1.004
#fvertical()# 1523 fps 1.006
#flipvertical()# 1740

[ARDA]

ok

#ColorBars(width=1920, height=1080, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 9999)
#fvertical()# 589.6 fps 1.004
#fvertical()# 608.4 fps 1.006
#flipvertical()# 580

ColorBars(width=1280, height=720, pixel_type="YV12").KillAudio().AssumeFPS(25, 1).Trim(0, 49999)
fvertical()# 4296 fps 1.004
#fvertical()# 1523 fps 1.006
#flipvertical()# 1740

fixing, fixing
Thank you ARDA

[jmac698]

Ever see the changelog for x265?
http://hg.videolan.org/x265
2 weeks ago Dnyaneshwar G asm: reduced 'vpermq' instructions, improved 1040c->800c, 22%
6 days ago Dnyaneshwar G asm: avx2 code for sad_x3_64xN, improved over 40% than SSE
6 days ago Dnyaneshwar G asm: avx2 code for sad_x3_32xN, improved over 40% than SSE
3 weeks ago Dnyaneshwar G asm: AVX2 code for pixel_var primitive, improved over 40% than SSE
2 weeks ago Rajesh Paulraj asm: avx2 8bpp code for chroma_p2s[16xN] for i420, i422, improved over 25% than SSE
2 weeks ago Rajesh Paulraj asm: avx2 8bpp code for convert_p2s[16xN],improved over 25% than SSE
2 weeks ago Rajesh Paulraj asm: avx2 code for sad_x4_48x64, improved over 25% than SSE
2 weeks ago Rajesh Paulraj asm: avx2 code for sad_x4_64xN, improved over 40% than SSE

This all adds up - if you look at http://www.compression.ru/video/codec_comparison/h264_2012/ you can see a comprhensive test of a previous codec, x264, as it progresses in both speed and quality over time, and with such a compute heavy task, the asm is a significant part. Things like motion search can't be cheated on without impacting quality, it just has to be optimized.

Also, on my machine, his version is 34% faster than the compilers, and this a fair comparison, even though he's using an old compiler, because I'm also using an old machine.

The many times faster version posted is using a trick to avoid memcpy in the first place, not an algorithmic speedup, that doesn't mean that there aren't times when a memcpy has to be used. Flipping vertical isn't the endgame here.

So what is a practical endgame? Perhaps replacing the virtualdub function,
BitBlt(g_hdcAboutDisplay, 0, 0, r.right, r.bottom, hdc, 0, 0, SRCCOPY);.
This would allow realtime screen capturing, and a 34% difference could make or break 30fps capture, for example.

ps h265 report is due any day now
http://www.compression.ru/video/codec_comparison/call_for_codecs_15.html

[foxyshadis]

Does shaving one or even ten seconds off of your 4-12 hour encode really add up in the end? Because that's what we're talking, a thousandth to a hundredth of a second difference per frame, even on your decade old Core 2 Duo. There's no way that anything else you could with each frame would make the savings worthwhile; encoding captured frames is still 99% of the time.

Back in the Pentium MMX and P2 days, gaming with a faster BitBlt was a big deal, but those days are long gone now.

[ARDA]

Does shaving one or even ten seconds off of your 4-12 hour encode really add up in the end?
Because that's what we're talking, a thousandth to a hundredth of a second difference per frame,
even on your decade old Core 2 Duo. There's no way that anything else you could with each
frame would make the savings worthwhile; encoding captured frames is still 99% of the time.

Back in the Pentium MMX and P2 days, gaming with a faster BitBlt was a big deal, but
those days are long gone now.
You will not find a line in my posts where I say that the optimitization of bitblt will
increase generall performance in fact I've said that it was very difficult to know
Anyway avisynth can create so complex and different chains that it is almost
imposible to preview and measure the use or not and/or the weight of bitblt-memcpy in general performance.

But also added that understimate the importance of assembly optimization was a tendency
That could lead to conclude, is it worth to do faster assembler implementations?
Maybe better to leave compilers do the job, or maybe everytime we jump to a new
higher resolution we buy a new and faster cpu. There are a lot of academic discussion
all over in the net about it , I wouldn't like this thread becomes a discussion about
this subject. Mainly because it is more related to commercial politics of corporations
and ideology of Universities about how to prepare new software engineers
and definitely this is not the place for such a polemic.

I've also pointed my doubts about the weight of bitblt-memcpy in general performance
Anyway it is quite obvious that bitblt-memcpy(weight??) is one of the steps that can influence in general
avisynth performance, but it is not the only one.

And maybe this is my approach to an opinion about what in my modest opinion should be the
spirit in avysinth development. Maybe this statement has offended anyone, my apologies
if that was the case.
Every filter or part of avisynth authors should take care of the performance of their
work
A minor precision, the implementation of Steady Bitblt was once ISSE and early SSE and had a relative
importance because it was made together with many other avisynth optimizations.
It was aperiod of great and enthusiastic development with many people contributing for that, and Sh0dan
leading the process.

This all adds up - if you look at http://www.compression.ru/video/code...son/h264_2012/
you can see a comprhensive test of a previous codec, x264, as it progresses in both speed
and quality over time, and with such a compute heavy task, the asm is a significant part.
Things like motion search can't be cheated on without impacting quality, it just has to
be optimized.
If this jmac698 post is taked out of context and simplified it to the point you will not gain nothing
cause the rest of process will take 90% of the time, we can conclude infinite things that
are not what jmac698 is saying; he is claiming that general assemabler optimizations are
important in many projects and enviroments, it was just an example.

Finally as I have said previously I didn't want this thread to become a battle of ideas
about the utility of optimizations or not, but I failed, this discussion entered by the window
with an intention that I still cannot manage to figure out.
Avisynth in an open source project and this process there will contributions more important
than others, my intention was to get help to finish a difficult part of the job, cause
many cpus are needed to check(bypass cahce values) and share this code with the community,
no more no less.
Allow me please without understimating anyone to end my participation in this discussion, and
I will focus to finish this work.
I hope you all can understand

Thanks to all ARDA

[jpsdr]

If you like working on this, learn things, and want to achieve a goal, so be it.

But, if you provide something only for x86 and nothing for x64, just to warn you that the part of the community interested risk to be greatly reduced.

[jmac698]

I finally ran my tests for version 1.007
https://www.sendspace.com/file/7nhf8z

[ARDA]

have a hunch that the plugin is cheating somehow
.......... is having some fun but without seeing the source code...

Finally I've had some free time and could work on flipvertical, and now we all can have
some fun and also we all win, nobody loses.And we also can satisfy our curiosity
of how to do a FlipVertical without using memcpy. I don't know if anyone has published
sonething similiar before, anyway I hope it can be usefull to clarify some points.
This is an open source version of the magic almost 0 cost version for flipvertical
by using subframe and subframeplanar. In fact this option needs only two lines of code.
You can see them in bold in the source below, most of the work is done with avisynth api
and internal avisynth frame managment.
This version of flip vertical has almost 0 cost, cause in fact it does nothing except a
reasignment of pointers and invert pitch.
It is a simple code without any optimization if there be any.

It was a bad idea to choose flipvertical to study bitblt/memcpy, cause the best
memcpy is that one is never done. But also because flipvertical only use the row memcpy
way, in fact the slowest form to do it. My mistake was to emulate the way flipvertical is
implemented in avisynth, without analyzing deeply this function,my apologies for that.
My intention was to use something familiar for everybody and simple to use as a benchmark tool.

This thread was intended to be a place to discuss, receive feedback and help for the development
of bitblit/memcpy, mainly to get the bypass caches values for different cpus. But this, the main purpose
of this thread was completely distorted, became a competition between different versions of
flipvertical, which in fact is not important itself.My apologies for that too.
If I get some more free time, perhaps in a near future I will resume the study of bitblt/memcpy in another thread.

Usage: NewFVertical()

Here's the code:

#include <windows.h>
#include "avisynth.h"
class NewFVertical : public GenericVideoFilter
{
public:
NewFVertical(PClip _child, IScriptEnvironment* env);
~NewFVertical();
PVideoFrame __stdcall GetFrame(int n, IScriptEnvironment* env);
};
NewFVertical::NewFVertical(PClip _child, IScriptEnvironment* env): GenericVideoFilter(_child){}
NewFVertical::~NewFVertical(){}
PVideoFrame __stdcall NewFVertical::GetFrame(int n,IScriptEnvironment* env)
{
PVideoFrame src = child->GetFrame(n, env);
const int src_pitch = src->GetPitch();
const int height = src->GetHeight();
const int yoffset = src_pitch*(height-1);
const int row_size = src->GetRowSize();

if (!src->GetPitch(PLANAR_U)){
return env->Subframe(src, yoffset, -src_pitch, row_size, height);
}
else{
return env->SubframePlanar
(src, yoffset, -src_pitch, row_size, height,
src->GetPitch(PLANAR_U)*(src->GetHeight(PLANAR_U)-1),
src->GetPitch(PLANAR_V)*(src->GetHeight(PLANAR_V)-1),-(src->GetPitch(PLANAR_U)));
}
};
AVSValue __cdecl Create_NewFVertical(AVSValue args, void*, IScriptEnvironment* env)
{
return new NewFVertical (args[0].AsClip(), env);
};
const AVS_Linkage *AVS_linkage = 0;
extern "C" __declspec(dllexport) const char* __stdcall
AvisynthPluginInit3(IScriptEnvironment* env, const AVS_Linkage* const vectors)
{
AVS_linkage = vectors;
env->AddFunction("NewFVertical", "c", Create_NewFVertical, 0);
return "NewFVertical, Flip vertically a clip";
};
NewFVertical Source and dll (https://mega.nz/#!UZsHnIoA!yrE2Hm5bZoGyR1QxI_gI6aozttEvYL76P_9nNbcTNDo)

[Nevilne]

This thread is literally the essence of doom9 (bar ripping).

[colours]

Just going to leave this here. (https://i.imgur.com/VSR1pTB.png)

If your intention had been to analyse or improve memcpy, you could've posted your findings to a forum about that. Not the "Avisynth Development" forum for, you know, the development of AVS and AVS plugins.

Just so you know, minimising memcpy/blits was deemed to be so irrelevant to processing with AVS in this day and age that AVS+ developers decided to make Crop() default to blitting every time, and this has negatively affected basically nobody.

[Chikuzen]

Just so you know, minimising memcpy/blits was deemed to be so irrelevant to processing with AVS in this day and age that AVS+ developers decided to make Crop() default to blitting every time, and this has negatively affected basically nobody.

That's not AVS+ way but VapourSynth way.
Unless crop the left side, avs+ does not execute memcpy.
And AVS+ avoid memcpy if crop left value is mod32.

[colours]

That's not AVS+ way but VapourSynth way.

My bad; I must've confused the two.

Point still holds, though: https://github.com/AviSynth/AviSynthPlus/commit/4a03e061aad110dd9b9892f2011829d7f320215e