Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
23rd October 2007, 15:05 | #661 | Link |
Registered User
Join Date: Sep 2006
Posts: 201
|
First result for the original MV_BUFFER_FRAMES=10
vim:tabstop=6 Tuesday, October 23, 2007 12:58:52 Frame Total Variation 15858 1 1.816673 15859 2 3.181405 15861 3 0.210538 Test complete. So at least I can now see an error - three close frames out of nearly 150,000. @Fizick: Before I test with the modified MVTools, can I just check with you, will they all produce bitwise identical output when run with no SetMTMode? If so, I only need run the SetMTMode variants & compare to my baseline avi, which will cut the time for each run from nearly 8 hours to only 1.5 hours. BTW The links you posted to the dlls are broken - you missed the s off the end of each mvtool. Last edited by mroz; 23rd October 2007 at 15:21. |
23rd October 2007, 18:02 | #662 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
mroz,
Thanks for fast testing. (I corrected a links). Yes, all MVTools without SetMTMode must produce identical results (only speed may vary a little). Modifications is not big, it is not a solution of the problem, but some indicative test.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
23rd October 2007, 18:53 | #663 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Fizick: a possible (temporary) solution could be to use a smart pointer to MVGroupOfFrames that correctly decremented the refcount of MVGroupOfFrames when it goes out of scope and only allow recycling of MVGroupOfFrames that has a refcount of 0. Currently only IncRefcount is implemented for MVGroupOfFrames.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ Last edited by tsp; 23rd October 2007 at 18:59. |
23rd October 2007, 20:25 | #664 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
tsp,
yes, and buffer size must be changed from constant (it may be not enough) to variable.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
23rd October 2007, 21:15 | #665 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Fizick: I will try implementing the MVGroupOfFrames smartpointer and a variable buffersize
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ |
23rd October 2007, 22:39 | #666 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Ok a new version that incorporates the above suggestion is ready. Please try it and see if it improves anything when used with setmtmode. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp.zip
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ |
24th October 2007, 03:57 | #667 | Link | ||
Registered User
Join Date: Sep 2006
Posts: 201
|
Quote:
Quote:
Here are my corresponding results for your MV_BUFFER_FRAMES=6 dll Code:
vim:tabstop=6 Tuesday, October 23, 2007 17:53:11 Frame Total Variation 11035 1 0.015687 11037 2 0.018902 11038 3 0.248984 11086 4 0.349582 11090 5 0.972039 15095 6 1.939990 15098 7 0.280023 16865 8 0.721465 16866 9 0.812497 16889 10 0.670940 16890 11 1.173558 17302 12 0.038992 17305 13 0.505871 17306 14 0.719297 18118 15 0.089515 20351 16 0.502222 20354 17 0.210579 22854 18 0.376654 24835 19 1.557945 24838 20 0.165073 28903 21 0.197361 28906 22 0.091904 29180 23 0.232695 29182 24 0.322055 29183 25 0.373993 30282 26 0.510847 30424 27 0.019494 30426 28 0.110264 30798 29 0.373128 30802 30 0.260479 31510 31 0.228801 33562 32 0.211502 33565 33 0.082752 33657 34 0.125897 33670 35 1.098033 33673 36 0.058188 35916 37 0.302044 37413 38 0.024555 37416 39 0.248225 40088 40 0.365724 41457 41 0.024742 41460 42 0.077433 42949 43 0.379954 42950 44 0.346376 43619 45 0.728546 43622 46 0.182506 44745 47 0.629474 44746 48 0.399838 46131 49 0.021676 46134 50 1.285427 48111 51 0.603243 48114 52 0.082384 48194 53 0.166780 51767 54 0.038720 51770 55 0.071487 52195 56 0.209815 52198 57 0.053900 52971 58 0.046540 52974 59 4.742179 55760 60 0.520754 55761 61 0.170728 55762 62 0.237919 57478 63 0.295764 61243 64 0.337114 61246 65 0.081116 61503 66 0.092666 61506 67 0.048821 61607 68 0.289923 61610 69 0.082652 61687 70 0.152583 61690 71 0.175322 62591 72 0.920901 62594 73 0.357663 62823 74 0.711127 63833 75 2.121323 63834 76 3.187475 64275 77 0.001378 64961 78 0.162714 64962 79 0.345326 67111 80 0.002054 67114 81 0.187752 69703 82 0.040604 69706 83 0.476924 71515 84 0.708087 71518 85 0.101423 71834 86 0.353010 73215 87 0.048512 73218 88 0.375491 74395 89 0.028145 74398 90 0.184067 74921 91 0.010113 74922 92 0.176437 75031 93 1.834347 75034 94 0.122357 77595 95 0.355491 77598 96 0.125475 78220 97 0.178333 78222 98 0.295954 78223 99 0.385514 78555 100 0.036707 78558 101 0.038212 79946 102 0.019544 79950 103 0.314157 80103 104 0.072031 80106 105 0.443512 80794 106 0.231408 80798 107 0.429387 82511 108 3.504525 82514 109 0.131558 83095 110 0.241904 83098 111 0.229109 83167 112 0.042083 83170 113 0.502277 83946 114 1.424533 84443 115 0.632131 85035 116 1.133995 85038 117 0.234205 86291 118 2.460492 86294 119 0.241094 89296 120 0.267791 89298 121 0.630590 89299 122 0.276484 89646 123 0.175880 91951 124 0.657542 91952 125 0.011080 91953 126 0.386147 92610 127 0.351812 92613 128 0.278424 93207 129 0.117969 93208 130 0.293646 94782 131 0.339254 94785 132 0.127108 95146 133 0.055489 95149 134 0.245969 95480 135 0.032614 95481 136 0.241876 96537 137 0.145288 96720 138 0.001228 96724 139 0.246610 96725 140 0.059872 96781 141 0.075450 96866 142 1.591221 96869 143 0.085255 99546 144 0.090596 99549 145 0.111317 99646 146 0.028165 99649 147 0.356581 99974 148 4.203999 99977 149 0.189322 100734 150 0.052224 100737 151 0.317698 101410 152 0.380507 101413 153 0.268456 101446 154 0.851510 101449 155 0.201051 101566 156 0.627114 101569 157 0.112604 102066 158 0.909575 102069 159 0.114531 103164 160 0.172163 103165 161 0.194189 104458 162 2.803576 104459 163 0.095143 104461 164 0.217942 104798 165 0.232049 104801 166 0.390398 106811 167 0.479852 106813 168 1.080945 106814 169 0.366071 107395 170 0.290446 107397 171 0.756489 107398 172 0.094236 108499 173 0.035672 108501 174 0.772666 109116 175 0.013842 109117 176 0.693476 109263 177 0.419134 109265 178 0.433827 109887 179 0.037398 109889 180 0.456709 110363 181 0.638587 110365 182 0.273036 111294 183 0.519425 112855 184 0.436112 112856 185 0.461187 112858 186 0.197535 113339 187 0.465803 113342 188 0.735912 118453 189 0.144465 118456 190 0.255532 118482 191 0.030365 118483 192 0.052658 118484 193 0.430398 119627 194 0.005023 119628 195 0.373068 119993 196 0.174600 119996 197 0.494724 120457 198 0.356260 120460 199 0.047622 124524 200 0.323934 125926 201 0.825262 125927 202 0.336890 125928 203 0.762235 125929 204 0.391387 125930 205 0.597881 125931 206 0.380590 130107 207 0.216572 130109 208 0.507754 130153 209 0.247313 133415 210 0.019257 133416 211 0.025065 133418 212 0.018010 133851 213 0.830890 133853 214 0.449552 134842 215 2.091352 134843 216 2.824228 134845 217 0.755368 134846 218 2.100201 134902 219 0.515646 134905 220 0.719432 134906 221 3.407452 135722 222 0.713153 137604 223 0.508993 137605 224 1.093383 137607 225 0.392873 137608 226 0.923373 138706 227 1.288582 138709 228 0.682757 143484 229 3.782573 143693 230 0.248797 143695 231 1.875079 143696 232 0.889129 Now I know I don't need to repeat the non MT runs for the new dlls I should be able to post all other results by tomorrow. Edit: Will of course also test tsp's mod. Last edited by mroz; 24th October 2007 at 04:00. |
||
24th October 2007, 05:03 | #668 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
thanks tsp!
I independenty made patched version too (it support MVDegran1 and MVAnalyse only) http://avisynth.org.ru/tmp/mvtools.dll probably your is more smart. (More comments in evening.) mroz, thanks for test! probably we really get weak chain.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
24th October 2007, 13:32 | #669 | Link |
Guest
Posts: n/a
|
since i have athlon 3200+, when i use this script i get about 4-5fps.
how can i use mt with this script to see how it goes? should i use mt with lsf or frfun7? FRFun7(Lambda=1.1,T=6.0,Tuv=0) dull = last sharp = dull.LimitedSharpenfaster( ss_x=1.0, ss_y=1.0,smode=3, strength=60, overshoot=1,special=true ) Soothe( sharp, dull, 20 ) Tweak(sat=1.1,bright=-8,cont=1.1) @ where to get the latest MT? Edit: i have found some info that mt is meant for cce encoding, not for divx/xvid, so can we use it with x264? Last edited by Vesi; 24th October 2007 at 14:22. |
24th October 2007, 16:36 | #670 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Vesi: Unless you have a hyperthreading or multicore or multiprocessor computer you wouldn't see any improve using the mt plugin.
Fizick: I more or less copied the PVideoClip class and modified it to use MVGroupOfFrames pointer instead of VideoClip pointer and changed the return type of MVFrames::GetNewFrame and MVFrames::GetFrame to the new PMVGroupOfFrames class and fixed all the compiler error what came afterwards (about conversion from PMVGroupOfFrames to MVGroupOfFrames * not possible). So it should work with all the functions that uses idx
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ |
24th October 2007, 22:07 | #671 | Link | |
Registered User
Join Date: Sep 2006
Posts: 201
|
Quote:
Your dll above resulted in a silent crash about 1% of the way into the encode. There's no useful info in the Megui log from Mencoder. It just seems to have terminated prematurely. I've not had any other problems on this machine since building it a couple of months ago, but I suppose it could still be anything. I'm re running that encode now & currently it's about 24% of the way through, so if the problem was down to mvtools, it isn't deterministic, but then if it's a threading issue one wouldn't expect it to be. Update: it crashed at 68% of the way through. Again, no useful log info; it just terminated unexpectedly. Regarding anticipated results, I can say before performing the comparisons that I've noticed the more errors there are, the more compressible the output, thus file size is some indication of error rate. For the buffer size variations it looks as though there's a very rapid increase in error numbers as the buffer size drops. The table below shows this. Under version, b<x> indicates the Fizick builds with buffer size <x>. Code:
Version Error # Undersizing (as a fraction of clean output) b4 ? 0.0092 b5 ? 0.0024 b6 232 0.0000039 b10 3 0.000000062 b15 ? 0.00000000032 tsp ? 0 (though not bitwise identical) Last edited by mroz; 24th October 2007 at 22:47. |
|
25th October 2007, 04:34 | #672 | Link |
Registered User
Join Date: Sep 2006
Posts: 201
|
Results:
Code:
Version Error # Undersizing (as a fraction of clean output) b4 128886 0.0092 b5 75865 0.0024 b6 232 0.0000039 b10 3 0.000000062 b15 3 0.00000000032 tsp 2 0 (though not bitwise identical) Now for b<x>, x<=10, logged errors are typically 0.1 to 1 in magnitude & almost never below 0.01. The resulting artifacts shown by subtract are visible to the eye without any enhancement & resemble ghosting around moving edges, corresponding to incorrect buffers being accessed I assume that actually correspond to temporally nearby frames. Recall the log from b10: Code:
Tuesday, October 23, 2007 12:58:52 Frame Total Variation 15858 1 1.816673 15859 2 3.181405 15861 3 0.210538 Test complete. b15: Code:
Thursday, October 25, 2007 01:44:06 Frame Total Variation 76888 1 0.000386 128332 2 0.000904 128331 3 0.001176 Test complete. Code:
Thursday, October 25, 2007 02:07:57 Frame Total Variation 7037 1 0.001146 63052 2 0.003961 Test complete. By digitalrat They all occur on the right side & resemble a formless small patch of noise almost. Typically within a low contrast area of the video & subject to high motion. Do you think these are indicative of the multithreading problems? I know Fizick said the b<x> variations should produce pixel wise identical output to each other when run without MT, which implies the errors in b15 must be the result of interaction with MT, but does that also apply to the tsp modified MVTools? I'll run a non MT encode overnight using the tsp variant just to check. Can anyone draw any conclusions, or likely conclusions, from the above? Any comments? Anything more I can do? For completeness, here's the script I used to look for differences: Code:
SetMTMode(2,0) leaf1="noSetMT" leaf2="SetMT-t1-b10" global c1=AVISource("E:\Work\test\hfyu_MVMT-longtest-"+leaf1+".avi") global c2=AVISource("E:\Work\test\hfyu_MVMT-longtest-"+leaf2+".avi") global total=0 global file="E:\Work\test\compare-"+leaf1+"-"+leaf2+".log" BlankClip(c1, width=16, height=16) # maybe 15% faster than using c1 as our 'dummy' clip (& 100% faster than using subtract(c1,c2) with variation subtitled) WriteFileStart(file, """ "vim:tabstop=6"+chr(10) """, """ Time("%#c")+chr(10) """, """ "Frame"+chr(9)+"Total"+chr(9)+"Variation" """, append=true) WriteFileEnd(file, """ "Test complete."+chr(10) """) ScriptClip( """ variation = LumaDifference(c1,c2)+ChromaUDifference(c1,c2)+ChromaVDifference(c1,c2) # nb1 keeping writefile inside here allows us to store variation as local, reducing problems with running in MTMode 2 # nb2 aside: writefile doesn't want to when called inside FrameEvaluate; anyone know why? # nb3 sometimes the running total is reported incorrectly in MTMode 2 as another thread updates the total before code writes out data to log; # this isn't terribly important & can be corrected as is implicit in data set # nb4 compare against 0.0001 instead of 0 to allow for rounding errors - 2 identical files will report a variation of 0.000022 WriteFileIf(file, "variation>0.0001", "current_frame", "chr(9)", "thistotal", "chr(9)", "variation") global total = total + ((variation>0.0001) ? 1 : 0) thistotal = total # massively reduce chance of MTMode 2 screwing up logging of total return last """ ) |
25th October 2007, 05:11 | #673 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
tsp,
thanks, I see the source code now. some question about memory optimizing. I see if all buffer spaces are filled, you increase its size by doubling. Why double the buffer size instead of incrementing it? mroz, thanks for massive test info. there is no fast conclusion. wait. But it is better fogrget about my broken mvtools.dll. tsp mod shoold be better (more safe).
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
25th October 2007, 13:44 | #674 | Link |
Registered User
Join Date: Sep 2006
Posts: 201
|
Wrt the non-MT encode using tsp's build, you probably don't need me to tell you, but it did indeed result in a bitwise identical output to the other builds' non-MT output. So, whatever is causing the small variance in output, it is MT related.
|
25th October 2007, 16:53 | #675 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Fizick: mainly to avoid resizing the array to many times. But incrementing it might be better as it doesn't take much time to copy the pointers.
The corruption in the lower right corner could be because both thread are running MVPlane::Refine/RefineExt at the same time. I have created a new version that uses a critical section to avoid this. It is available from here mroz: Thanks for the testing. Could you try the new version above? [edit] ups wrong version. I have updated the file so please download again. This one should work. [/edit]
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ Last edited by tsp; 25th October 2007 at 17:16. |
25th October 2007, 20:03 | #676 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,729
|
I have seen this behaviour recently even without any MT - however the application I used to encode uses multithreading. I'm trying to reproduce it to provide a better report, I first need to figure out if it's the encoder or MVTools causing the issue.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
25th October 2007, 20:09 | #677 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
Tsp, thanks for update (I will mirror your version at my site after test).
Can two threads simultaniuosly refine different planes now? If not, is any performance penalty? Can somebody (mroz) make a speed test? (and artefact test too, of course)
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
25th October 2007, 23:18 | #678 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
Fizick: Yes they can refine different planes at the same time as there are one lock per plane. In a very small speedtest with a 720x576 source and this script
Code:
SetMTmode(2) function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx") { overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8) dct=default(dct,0) # use dct=1 for clip with light flicker idx=default(idx,1) # use various idx for different sources in same script fields=source.SeparateFields() # separate by fields backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct) forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct) backward_vec4 = fields.MVAnalyse(isb = true, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct) forward_vec4 = fields.MVAnalyse(isb = false, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct) fields.MVDegrain2(backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400,idx=idx) Weave() } source=AVISource("F:\20070513-193504.avi").trim(0,101) mvdegrain2i(source,4,1,1)
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ |
26th October 2007, 04:57 | #679 | Link |
AviSynth plugger
Join Date: Nov 2003
Location: Russia
Posts: 2,183
|
I do not understand why you use same critical section both for frame and for plane class?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick I usually do not provide a technical support in private messages. |
26th October 2007, 08:29 | #680 | Link |
Registered User
Join Date: Aug 2004
Location: Denmark
Posts: 807
|
it's not the same critical section. It is both defined in the MVFrames and MVPlane class. I use the same variable name for the CRITICAL_SECTION sorry if that is confusing. So it is one CRITICAL_SECTION per class instance.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/ |
Thread Tools | Search this Thread |
Display Modes | |
|
|