Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
26th July 2016, 23:12 | #1 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,695
|
Long GOP Artifacts
Fog is a problem for long GOP compression. I'm trying to salvage some spectacular footage taken at 2:00 a.m. under the midnight sun by a DJI Inspire drone in the park on the north side of Iceland. Here is a sample from the original 4K footage, cut with ffmpeg:
DJI Inspire Drone Video Long GOP Jerkiness.MOV As you will see, the car's motion is smooth, but amorphous objects like the dark ground and the fog are "pulsing." I'm pretty sure this is caused by the inability of the interframe compression algorithm to track the featureless fog, and therefore the pixels associated with that "object" drift aimlessly during for seven frames, and then get harshly moved when an I frame happens every 8 frames. The car and the mountains in the background are perfectly smooth. My goal is to reduce this harshness. One idea is to either sense (don't know how to do this) or at least mark the start of each GOP (I can afford to take the time to do this, if needed), and then replace the frames on either side with some sort of motion-estimate version. This won't fix everything, but it might reduce the impact of having the position of the fog moved all at once. For those of you who look at this, you might want to brighten it a bit (increase the top half of the histogram). When brighter, the artifacts really pop out. Also, I have no problem fixing the sensor noise in the shadows, adjusting the gamma, and adding some "punch." It is only the GOP issue that I'm trying to solve. Any other ideas or thoughts on how to approach this would be welcome. Last edited by johnmeyer; 26th July 2016 at 23:16. Reason: typo |
27th July 2016, 03:40 | #2 | Link |
Registered User
Join Date: Mar 2016
Location: Arkansas
Posts: 95
|
Quick testing showed that this helped on a 720x480 version of your video. (My netbook won't run 4k ) You may have to modify it a bit.
Code:
v=AVISource("F:\Drone OV.avi").converttoyv12() v.autolevels(12) a=last b=a.gaussresize(width(v)/8,height(v)/8).gblur(2).gaussresize(width(a),height(a)) y1=overlay(a,b,mode="subtract") y2=b.interframe("Faster","Smooth",60000,1001,cores=1).selectevery(4,0,3).temporalsoften(10,25,25,0).gblur(2) y3=merge(y1.temporalsoften(8,25,25,0),y1,.25) merge(y2,y3) temporalsoften(10,20,20,0) y4=last return y4.f3kdb() Edit: Link Updated Edit: Should allow more detail through. Edit: Added link to new video. Last edited by MWilson; 27th July 2016 at 14:25. Reason: Reverting |
27th July 2016, 05:38 | #3 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,695
|
I looked at the video you sent to me before I looked at the script and I guessed that MVTools2 (via Interframe in this case) was involved. Let's see if I understand your approach:
I assume you converted to an AVI in order to deal with the container format of the original video. The autolevels was to make it bright enough to see what's going on. It looks like you are doing some sort of subtraction of a blurred lower-res version of the video from the original video. I'm not sure what effect that produces and will have to try it out when I get back to my editing computer tomorrow. After that it looks like you double the framerate and select the interpolated frames. You soften and blur those and then merge them with various softened versions of the original. There is no doubt, from the sample you sent to me, that it does reduce the GOP artifacts but I won't know tomorrow how much violence has been done to the details. Despite the soft fuzzy nature of the video below the fog, there is still a lot of detail once the drone breaks through the fog and shows the mountains in the distance against the midnight sun. I don't want that to become too soft. However, if your technique works, I can mask your version with the original, letting the original video of the mountains show through the mask. I use Sony Vegas, and it is trivial to create a mask to let the mountains from the original video overlay your corrected version, and then keyframe that mask as the drone makes its ascent. I think you're on the right track with this and I'll work with it tomorrow. Thanks! I'm still very much open to other ideas and suggestions. This is actually a somewhat common problem that I've seen many times in other people's videos, so finding a good solution would benefit a lot of people other than just me. |
27th July 2016, 06:09 | #4 | Link |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
sorry for the late reply, just checked your sample and I think the problem is actually not the fog, it's the noise
see something like this? noise is dynamic in time dimension most of the time, and that makes it look less annoying in motion, and when you got large amount of noise in your vid, macroblocks will go random noise block alike, noise will be the dominate factor of SAD between 2 macroblocks and that messes motion estimation and compensation up, block matching works long as blocks are similar enough, and by "similar", that means the actual image, not noise, should be the dominate factor of SAD and SAD should be small, and in your case, block matching was screwed by noise, actually pretty similar situation like this, http://forum.doom9.org/showthread.php?t=173705 and now because me/mc was fucked, the noise in your compressed vid goes almost static in time dimension, and that's the cause of "jerkiness", trying to figure out a way to cure this... |
27th July 2016, 09:24 | #5 | Link |
Registered User
Join Date: Dec 2015
Posts: 59
|
B-frames in the dark already make some kind of freeze, especially in the noise. Because "lazy". It's a target issue in the evolution of MPEG. Old Divx4 was notorious. Later improved.
Masking, and to make data update even more lazy, might mitigate the issue. Within the dark-mask, a temporal smoother might almost cancel the new data from the i-frame. After done that, reverse processing might carry the remainder of the i-frame data into the earlier frames also. Means actual bidirectional temp-smooth. Needs then a threshold to stop that and update much faster (maybe adjust during ~4 frames while otherwise might be quite lazy during a whole second), when something relevant blends in within the dark area. |
27th July 2016, 11:58 | #6 | Link |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
@johnmeyer
the processed sample here, https://mega.nz/#!jx8TUAhB!VfsoBw2LR...0Fls1IH4RjICYY random stuff: 1. scaled to 1920x1080 cuz block matching at 4k is way too heavy to my crappy i7-4790k 2. ditched chroma planes, I simply don't care much about chroma and the artifacts show mostly on luma plane anyways. 3. losslessly compressed with avc lossless Does it look okay to you now? the solution is a bit complex, so, I'm waiting for your reply. I'll go on explaining if the answer is "yeah" |
27th July 2016, 12:58 | #7 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
Code:
Loadplugin("ffms2.dll") Import("ffms2.avsi") SSS=""" Global Glb_FrameType = FFPICT_TYPE # Maybe you want it somewhere else RT_DBaseSet(DB,current_frame,Glb_FrameType) RT_Subtitle("%d] %c",current_frame,Glb_FrameType) return last """ FN="big_buck_bunny_720p_h264.mov" FFmpegSource2(FN) DB = "Meyer.DB" RT_DbaseAlloc(DB,Framecount,"i") # FrameCount Records, Single field of type int, CAPS character code ie 'I'=73, 'P'=80, 'B'=66. ScriptClip(SSS,after_frame=true) and then real run with access to current GOP start, previous and next GOP (via backward/forward scan of DB). EDIT: By using another global and 2 additional int fields, could track previous I frame, and set 2nd field (field 1) to eg previous I frame, and then set previous I frame's (field 2) next I frame to current frame (if that makes sense), in real run, allow access to prev and next I frame numbers without scan backwards/forwards.
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 27th July 2016 at 13:10. |
|
27th July 2016, 13:42 | #8 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Quote:
the fog looks kind of normal and the auto and the mountain are shimmering like hell, cuz you merged interpolated frames with the source frame, and that's toxic, like that lame and dumb "ConvertFPS" trick, motion interpolation is far from perfect first, and just say if it was indeed perfect, you're merging frames at different time coordinates together, and that = ghosting + shimmering and other shit like that I didn't use no motion interpolation, you see the problem is NOT low framerate so motion interpolation is so not the solution |
|
27th July 2016, 13:52 | #9 | Link |
Registered User
Join Date: Mar 2016
Location: Arkansas
Posts: 95
|
Feisty,
Just return y4. FWIW, I like your solution. But belittling other folks' is flat out mean. Constructive criticism is one thing, but when you are trying to help someone, you don't expect hostility. Good Luck. |
27th July 2016, 14:00 | #10 | Link |
Registered User
Join Date: Sep 2007
Posts: 5,377
|
feisty, I'm interested in your approach (but it's not fun running your typical scripts which eat too much memory and run at 0.0001 FPS )
The motion is better, but is there a way to preserve the ground detail, and to work with CbCr ? Especially the ground rocks near the jeep, and they are present in the Y' channel, but filtered out in your video. I suppose masking/overlay is one option bring it back |
27th July 2016, 14:01 | #11 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Quote:
you were trying to be blunt and ended up being mean sorry then |
|
27th July 2016, 14:18 | #12 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Quote:
maybe... guess the sigmoidal light might work for this simple, remove "clp = core.std.ShufflePlanes(clp,0,vs.GRAY)" |
|
27th July 2016, 14:27 | #13 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
Script to Create DBase where for each record(frame) there are 3 int fields, 0) frame status, 1) frameNo of prev I frame, 2) frameNo of Next I frame. Code:
# https://forum.doom9.org/showthread.php?p=1775535#post1775535 Function DB_Write(string DB,int n,int type) { RT_DBaseSet(DB,n,Type,Glb_PrevIFrame,-1) GSCript(""" if(Type==73) { # I frame if(Glb_PrevIFrame>=0) { for(i=Glb_PrevIFrame,n-1) { RT_DBaseSetField(DB,i,2,n) # Set all previous GOP frames 'Next I frame field' to current frame n } } Global Glb_PrevIFrame=n # Remember current frame as previous I frame when at following frames } """) return 0 } #Loadplugin("ffms2.dll") Import("ffms2.avsi") # May need path if not in current directory FN="K:\TESTAVI\big_buck_bunny_720p_h264.mov" FFmpegSource2(FN) DB = "Meyer.DB" DB=RT_GetFullPathName(DB) SHOW =True FORCE=true # Use ForceProcessAVI Global Glb_PrevIFrame = -1 # Unset RT_DbaseAlloc(DB,Framecount,"iii") # FrameCount Records, 3 fields of type int, # Field 0) CAPS character code of current frame(record), ie 'I'=73, 'P'=80, 'B'=66. # Field 1) frame number(record) of previous I frame # Field 2) frame number(record) of next I frame SSS=""" FrameType = FFPICT_TYPE # Have to use FFPICT_TYPE from main level script, not global. (Show&&!FORCE) ? RT_Subtitle("%d] %c Prev=%d ",current_frame,FrameType,Glb_PrevIFrame) : NOP DB_Write(DB,current_frame,FrameType) # function as we want GScript return last """ ScriptClip(SSS,after_frame=true) (FORCE)?ForceProcessAVI():NOP # From TWriteAVI, Fast creation of DBase (or RT_ForceProcess from RT_ v2.0) Return Last And DBase Viewer/Checker Code:
#Loadplugin("ffms2.dll") Import("ffms2.avsi") # May need path if not in current directory FN="K:\TESTAVI\big_buck_bunny_720p_h264.mov" FFmpegSource2(FN) DB="Meyer.DB" DB=RT_GetFullPathName(DB) SSS=""" n=current_frame typ=RT_DBaseGetField(DB,n,0) prv=RT_DBaseGetField(DB,n,1) nxt=RT_DBaseGetField(DB,n,2) RT_Subtitle("%d] %c PrevI=%d NextI=%d",n,typ,prv,nxt) return last """ ScriptClip(SSS) return last EDIT: Req, RT_ Gscript (and TwriteAVI v2.0 if required).
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 25th April 2017 at 16:45. |
|
27th July 2016, 17:20 | #14 | Link | |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,695
|
Quote:
However, I feel like Spock in the original Star Trek "Operation -- Annihilate!" episode when McCoy tries to free him from the pain of a parasite. After the procedure Spock says: "I'm also quite blind. An equitable trade, Doctor. Thank you." So in this case, the pain of the pulsing is gone, but I'm now color blind because the video no longer has any color. Did you ditch the color planes in order to make it work, or did it simply help speed up the fix? I would obviously like to keep the color since that is part of the "payoff" when the drone breaks through the fog. However, if that is not possible, I actually might be able to create something this is visually even more dramatic, with the fog and below done in B&W, and then I mask in the original footage of the mountains. As I indicated earlier, that is a trivial operation with the NLE I use. I'm obviously very interested in the script you used and hope I can adapt it to my computer and setup. Thank you very much for doing this. |
|
27th July 2016, 17:30 | #15 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Quote:
kind of a personal habit |
|
27th July 2016, 18:06 | #16 | Link |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Code:
import vapoursynth as vs import mvmulti import Vine import Oyster core = vs.get_core() clp = core.lsmas.LWLibavSource("DJI Inspire Drone Video Long GOP Jerkiness .MOV") clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True) #convert to floating point precision and full range clp = core.std.ShufflePlanes(clp,0,vs.GRAY) #extract Y clp = core.fmtc.resample(clp,1920,1080,kernel="cubic",a1=-1,a2=0) #scale to 2k clp = core.fmtc.transfer(clp,"709","470bg",fulls=True,fulld=True) #brighten it up, dont want dark details all fucked up pre = Vine.helpers.padding(clp,17,17,17,17) #pad, in spatial dimensions pre = core.std.Reverse(pre)+pre+core.std.Reverse(pre) #pad, in temporal dimension pre = core.knlm.KNLMeansCL(pre, 16, 8, 1, 6.4) #very large time radius NLMeans, motion flow and denoising 2 in 1, more of motion flow here pre = core.std.Trim(pre,269,537) #remove padding in temporal dimension pre = core.std.CropRel(pre,17,17,17,17) #remove padding in spatial dimensions #"pre" looks super smooth in motion because of the motion flow alike NLMeans, and super washed out... supv = core.mvsf.Super(pre) #blah supd = core.mvsf.Super(core.std.MakeDiff(clp,pre)) #blah vec = mvmulti.Analyze(supv,tr=16,blksize=32,overlap=16,search=3,badrange=-24) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=16,overlap=8,search=3) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=8,overlap=4,search=3) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=4,overlap=2,search=3) #blah dif = mvmulti.DegrainN(core.std.Expr(clp,"0.5"), supd, vec, tr=16,thsad=10000,thscd1=10000,thscd2=255) #stabilize the difference between clp and pre clp = core.std.MergeDiff(pre,dif) #merge the difference back #no more washed out now supr = core.mvsf.Super(clp) #blah bv = core.mvsf.Analyze(supv,delta=3,blksize=32,overlap=16,search=3,badrange=-24,isb=True) #delta=3 is good, delta=1,2 ain't effective enough fv = core.mvsf.Analyze(supv,delta=3,blksize=32,overlap=16,search=3,badrange=-24,isb=False) #blah blr = core.mvsf.FlowBlur(clp, supr, bv, fv, 100,thscd1=10000,thscd2=255) #do a full time motion blur here and make the fog look smoother than ever blr = Oyster.helpers.thr_merge(blr,clp,thr=8/256,elast=2/256) #don't want motion blur on the entire frame #only want that on the fog, and fog is kind of the delicate and flat feature here #this filters out motion blur on strong and significant features like the auto and moutains clp = Vine.helpers.cutoff(blr,clp,8) #this filters out motion blur on high frequency features clp = core.grain.Add(clp, 2) #some extra grain here to make it look less... whatever clp = core.fmtc.transfer(clp,"470bg","709",fulls=True,fulld=True) #reverse the brightening clp.set_output() Last edited by feisty2; 27th July 2016 at 18:12. |
27th July 2016, 18:26 | #17 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
Quote:
I'm so fucking depressed and mad cuz I got matrix mechanics class the entire day and I was goddamn exhausted and my boyfriend bitched about how I can't cook when I got back to my apartment. really really could use some praising right fucking now |
|
27th July 2016, 18:33 | #18 | Link |
Registered User
Join Date: Feb 2002
Location: California
Posts: 2,695
|
Digesting this ...
It doesn't look like translating this into an AVISynth script is doable. Vapoursynth is a big hill to climb for this old fart ... need to figure out how many things to download, and which computer I need to use ... I'll post again after I've done a little homework. |
27th July 2016, 20:00 | #19 | Link | |
Registered User
Join Date: Mar 2012
Location: Texas
Posts: 1,666
|
Quote:
Looking at feisty's script this should be doable in AviSynth. Not too long ago I had some HFR video and there where parts of the video that was flickering (and pulsating a bit) due to artificial lighting. I tinkered a bit but never came up with anything solid, might have to play around with it some more. |
|
27th July 2016, 20:24 | #20 | Link | |
I'm Siri
Join Date: Oct 2012
Location: void
Posts: 2,633
|
No it's not, you got no Expr in avisynth and that makes the_merge impossible to port, and lut is useless because the precision is way too low, you need as much precision as you can under "470bg" light
Quote:
Fine, I was kidding... I just want some comfort from a stranger online, feels kind of pathetic now.. Last edited by feisty2; 27th July 2016 at 20:27. |
|
|
|