Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum Long GOP Artifacts
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 26th July 2016, 23:12 #1  |  Link johnmeyer Registered User   Join Date: Feb 2002 Location: California Posts: 2,456 Long GOP Artifacts Fog is a problem for long GOP compression. I'm trying to salvage some spectacular footage taken at 2:00 a.m. under the midnight sun by a DJI Inspire drone in the park on the north side of Iceland. Here is a sample from the original 4K footage, cut with ffmpeg: DJI Inspire Drone Video Long GOP Jerkiness.MOV As you will see, the car's motion is smooth, but amorphous objects like the dark ground and the fog are "pulsing." I'm pretty sure this is caused by the inability of the interframe compression algorithm to track the featureless fog, and therefore the pixels associated with that "object" drift aimlessly during for seven frames, and then get harshly moved when an I frame happens every 8 frames. The car and the mountains in the background are perfectly smooth. My goal is to reduce this harshness. One idea is to either sense (don't know how to do this) or at least mark the start of each GOP (I can afford to take the time to do this, if needed), and then replace the frames on either side with some sort of motion-estimate version. This won't fix everything, but it might reduce the impact of having the position of the fog moved all at once. For those of you who look at this, you might want to brighten it a bit (increase the top half of the histogram). When brighter, the artifacts really pop out. Also, I have no problem fixing the sensor noise in the shadows, adjusting the gamma, and adding some "punch." It is only the GOP issue that I'm trying to solve. Any other ideas or thoughts on how to approach this would be welcome. Last edited by johnmeyer; 26th July 2016 at 23:16. Reason: typo
 27th July 2016, 03:40 #2  |  Link MWilson Registered User   Join Date: Mar 2016 Location: Arkansas Posts: 95 Quick testing showed that this helped on a 720x480 version of your video. (My netbook won't run 4k ) You may have to modify it a bit. Code: v=AVISource("F:\Drone OV.avi").converttoyv12() v.autolevels(12) a=last b=a.gaussresize(width(v)/8,height(v)/8).gblur(2).gaussresize(width(a),height(a)) y1=overlay(a,b,mode="subtract") y2=b.interframe("Faster","Smooth",60000,1001,cores=1).selectevery(4,0,3).temporalsoften(10,25,25,0).gblur(2) y3=merge(y1.temporalsoften(8,25,25,0),y1,.25) merge(y2,y3) temporalsoften(10,20,20,0) y4=last return y4.f3kdb() Here's the (new) result:https://drive.google.com/open?id=0B7...UpCOE45YjlxaXc Edit: Link Updated Edit: Should allow more detail through. Edit: Added link to new video. Last edited by MWilson; 27th July 2016 at 14:25. Reason: Reverting
 27th July 2016, 05:38 #3  |  Link johnmeyer Registered User   Join Date: Feb 2002 Location: California Posts: 2,456 I looked at the video you sent to me before I looked at the script and I guessed that MVTools2 (via Interframe in this case) was involved. Let's see if I understand your approach: I assume you converted to an AVI in order to deal with the container format of the original video. The autolevels was to make it bright enough to see what's going on. It looks like you are doing some sort of subtraction of a blurred lower-res version of the video from the original video. I'm not sure what effect that produces and will have to try it out when I get back to my editing computer tomorrow. After that it looks like you double the framerate and select the interpolated frames. You soften and blur those and then merge them with various softened versions of the original. There is no doubt, from the sample you sent to me, that it does reduce the GOP artifacts but I won't know tomorrow how much violence has been done to the details. Despite the soft fuzzy nature of the video below the fog, there is still a lot of detail once the drone breaks through the fog and shows the mountains in the distance against the midnight sun. I don't want that to become too soft. However, if your technique works, I can mask your version with the original, letting the original video of the mountains show through the mask. I use Sony Vegas, and it is trivial to create a mask to let the mountains from the original video overlay your corrected version, and then keyframe that mask as the drone makes its ascent. I think you're on the right track with this and I'll work with it tomorrow. Thanks! I'm still very much open to other ideas and suggestions. This is actually a somewhat common problem that I've seen many times in other people's videos, so finding a good solution would benefit a lot of people other than just me.
 27th July 2016, 06:09 #4  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,630 sorry for the late reply, just checked your sample and I think the problem is actually not the fog, it's the noise see something like this? noise is dynamic in time dimension most of the time, and that makes it look less annoying in motion, and when you got large amount of noise in your vid, macroblocks will go random noise block alike, noise will be the dominate factor of SAD between 2 macroblocks and that messes motion estimation and compensation up, block matching works long as blocks are similar enough, and by "similar", that means the actual image, not noise, should be the dominate factor of SAD and SAD should be small, and in your case, block matching was screwed by noise, actually pretty similar situation like this, http://forum.doom9.org/showthread.php?t=173705 and now because me/mc was fucked, the noise in your compressed vid goes almost static in time dimension, and that's the cause of "jerkiness", trying to figure out a way to cure this...
 27th July 2016, 09:24 #5  |  Link geometer Registered User   Join Date: Dec 2015 Posts: 59 B-frames in the dark already make some kind of freeze, especially in the noise. Because "lazy". It's a target issue in the evolution of MPEG. Old Divx4 was notorious. Later improved. Masking, and to make data update even more lazy, might mitigate the issue. Within the dark-mask, a temporal smoother might almost cancel the new data from the i-frame. After done that, reverse processing might carry the remainder of the i-frame data into the earlier frames also. Means actual bidirectional temp-smooth. Needs then a threshold to stop that and update much faster (maybe adjust during ~4 frames while otherwise might be quite lazy during a whole second), when something relevant blends in within the dark area.
 27th July 2016, 11:58 #6  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,630 @johnmeyer the processed sample here, https://mega.nz/#!jx8TUAhB!VfsoBw2LR...0Fls1IH4RjICYY random stuff: 1. scaled to 1920x1080 cuz block matching at 4k is way too heavy to my crappy i7-4790k 2. ditched chroma planes, I simply don't care much about chroma and the artifacts show mostly on luma plane anyways. 3. losslessly compressed with avc lossless Does it look okay to you now? the solution is a bit complex, so, I'm waiting for your reply. I'll go on explaining if the answer is "yeah"
27th July 2016, 12:58   #7  |  Link
StainlessS
HeartlessS Usurer

Join Date: Dec 2009
Location: Over the rainbow
Posts: 9,499
Quote:
 One idea is to either sense (don't know how to do this) or at least mark the start of each GOP
Perhaps of use
Code:
Loadplugin("ffms2.dll")
Import("ffms2.avsi")

SSS="""
Global Glb_FrameType = FFPICT_TYPE                              # Maybe you want it somewhere else
RT_DBaseSet(DB,current_frame,Glb_FrameType)
RT_Subtitle("%d] %c",current_frame,Glb_FrameType)
return last
"""

FN="big_buck_bunny_720p_h264.mov"
FFmpegSource2(FN)

DB = "Meyer.DB"
RT_DbaseAlloc(DB,Framecount,"i")      # FrameCount Records, Single field of type int, CAPS character code ie 'I'=73, 'P'=80, 'B'=66.

ScriptClip(SSS,after_frame=true)
EDIT: You can remove the DBase stuff if all you want is the Global, but using DB allows presetting of all DBase fields,
and then real run with access to current GOP start, previous and next GOP (via backward/forward scan of DB).

EDIT: By using another global and 2 additional int fields, could track previous I frame, and set 2nd field (field 1) to eg previous I frame,
and then set previous I frame's (field 2) next I frame to current frame (if that makes sense), in real run, allow access to prev and next I frame
numbers without scan backwards/forwards.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 27th July 2016 at 13:10.

27th July 2016, 13:42   #8  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by MWilson Quick testing showed that this helped on a 720x480 version of your video. (My netbook won't run 4k ) You may have to modify it a bit. Code: v=AVISource("F:\Drone OV.avi").converttoyv12() v.autolevels(12) a=last.interframe("Faster","Smooth",60000,1001,cores=1).selectevery(4,0,3) b=a.gaussresize(width(v)/12,height(v)/12).gblur(2).gaussresize(width(a),height(a)) y1=overlay(a,b,mode="subtract") y2=b.interframe("Faster","Smooth",60000,1001,cores=1).selectevery(4,0,3).temporalsoften(10,25,25,0).gblur(2) y3=merge(y1.temporalsoften(8,25,25,0),y1,.25) merge(y2,y3) temporalsoften(10,20,20,0) y4=last y5=merge(a,y4,.9) return y5.f3kdb() Here's the result:https://drive.google.com/open?id=0B7...2p0SXpkdTFIVFE Edit: Should allow more detail through. Edit: Added link to video.
I'm just gonna be blunt here, your solution sucks
the fog looks kind of normal and the auto and the mountain are shimmering like hell, cuz you merged interpolated frames with the source frame, and that's toxic, like that lame and dumb "ConvertFPS" trick, motion interpolation is far from perfect first, and just say if it was indeed perfect, you're merging frames at different time coordinates together, and that = ghosting + shimmering and other shit like that

I didn't use no motion interpolation, you see the problem is NOT low framerate so motion interpolation is so not the solution

 27th July 2016, 13:52 #9  |  Link MWilson Registered User   Join Date: Mar 2016 Location: Arkansas Posts: 95 Feisty, Just return y4. FWIW, I like your solution. But belittling other folks' is flat out mean. Constructive criticism is one thing, but when you are trying to help someone, you don't expect hostility. Good Luck.
 27th July 2016, 14:00 #10  |  Link poisondeathray Registered User   Join Date: Sep 2007 Posts: 4,690 feisty, I'm interested in your approach (but it's not fun running your typical scripts which eat too much memory and run at 0.0001 FPS ) The motion is better, but is there a way to preserve the ground detail, and to work with CbCr ? Especially the ground rocks near the jeep, and they are present in the Y' channel, but filtered out in your video. I suppose masking/overlay is one option bring it back
27th July 2016, 14:01   #11  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by MWilson Feisty, Just return y4. FWIW, I like your solution. But belittling other folks' is flat out mean. Constructive criticism is one thing, but when you are trying to help someone, you don't expect hostility. Good Luck.
fine, I didn't mean to be mean but I always forget the difference between "blunt" and "mean" is fragile..
you were trying to be blunt and ended up being mean

sorry then

27th July 2016, 14:18   #12  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by poisondeathray (but it's not fun running your typical scripts which eat too much memory and run at 0.0001 FPS )

Quote:
 Originally Posted by poisondeathray is there a way to preserve the ground detail
maybe... guess the sigmoidal light might work for this
Quote:
 Originally Posted by poisondeathray and to work with CbCr ?
simple, remove "clp = core.std.ShufflePlanes(clp,0,vs.GRAY)"

27th July 2016, 14:27   #13  |  Link
StainlessS
HeartlessS Usurer

Join Date: Dec 2009
Location: Over the rainbow
Posts: 9,499
Quote:
 One idea is to either sense (don't know how to do this) or at least mark the start of each GOP
Further to above:

Script to Create DBase where for each record(frame) there are 3 int fields, 0) frame status, 1) frameNo of prev I frame, 2) frameNo of Next I frame.
Code:
# https://forum.doom9.org/showthread.php?p=1775535#post1775535

Function DB_Write(string DB,int n,int type) {
RT_DBaseSet(DB,n,Type,Glb_PrevIFrame,-1)
GSCript("""
if(Type==73) {                                      # I frame
if(Glb_PrevIFrame>=0) {
for(i=Glb_PrevIFrame,n-1) {
RT_DBaseSetField(DB,i,2,n)              # Set all previous GOP frames 'Next I frame field' to current frame n
}
}
Global Glb_PrevIFrame=n                         # Remember current frame as previous I frame when at following frames
}
""")
return 0
}

Import("ffms2.avsi")           # May need path if not in current directory

FN="K:\TESTAVI\big_buck_bunny_720p_h264.mov"
FFmpegSource2(FN)

DB = "Meyer.DB"
DB=RT_GetFullPathName(DB)

SHOW =True
FORCE=true                              # Use ForceProcessAVI
Global Glb_PrevIFrame = -1              # Unset
RT_DbaseAlloc(DB,Framecount,"iii")      # FrameCount Records, 3 fields of type int,
#  Field 0) CAPS character code of current frame(record), ie 'I'=73, 'P'=80, 'B'=66.
#  Field 1) frame number(record) of previous I frame
#  Field 2) frame number(record) of next I frame

SSS="""
FrameType = FFPICT_TYPE                         # Have to use FFPICT_TYPE from main level script, not global.
(Show&&!FORCE) ? RT_Subtitle("%d] %c Prev=%d ",current_frame,FrameType,Glb_PrevIFrame) : NOP
DB_Write(DB,current_frame,FrameType)            # function as we want GScript
return last
"""

ScriptClip(SSS,after_frame=true)

(FORCE)?ForceProcessAVI():NOP                      # From TWriteAVI, Fast creation of DBase (or RT_ForceProcess from RT_ v2.0)

Return Last

And DBase Viewer/Checker
Code:
#Loadplugin("ffms2.dll")
Import("ffms2.avsi")       # May need path if not in current directory

FN="K:\TESTAVI\big_buck_bunny_720p_h264.mov"
FFmpegSource2(FN)

DB="Meyer.DB"
DB=RT_GetFullPathName(DB)

SSS="""
n=current_frame
typ=RT_DBaseGetField(DB,n,0)
prv=RT_DBaseGetField(DB,n,1)
nxt=RT_DBaseGetField(DB,n,2)
RT_Subtitle("%d] %c PrevI=%d NextI=%d",n,typ,prv,nxt)
return last
"""

ScriptClip(SSS)
return last
I have not as yet downloaded your clip, so I used a mov that I do have.

EDIT: Req, RT_ Gscript (and TwriteAVI v2.0 if required).
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 25th April 2017 at 16:45.

27th July 2016, 17:20   #14  |  Link
johnmeyer
Registered User

Join Date: Feb 2002
Location: California
Posts: 2,456
Quote:
 Originally Posted by feisty2 @johnmeyer the processed sample here, https://mega.nz/#!jx8TUAhB!VfsoBw2LR...0Fls1IH4RjICYY random stuff: 1. scaled to 1920x1080 cuz block matching at 4k is way too heavy to my crappy i7-4790k 2. ditched chroma planes, I simply don't care much about chroma and the artifacts show mostly on luma plane anyways. 3. losslessly compressed with avc lossless Does it look okay to you now? the solution is a bit complex, so, I'm waiting for your reply. I'll go on explaining if the answer is "yeah"
The answer is definitely "yeah." Your approach does seem to have pretty much nailed the pulsing.

However, I feel like Spock in the original Star Trek "Operation -- Annihilate!" episode when McCoy tries to free him from the pain of a parasite. After the procedure Spock says: "I'm also quite blind. An equitable trade, Doctor. Thank you."

So in this case, the pain of the pulsing is gone, but I'm now color blind because the video no longer has any color.

Did you ditch the color planes in order to make it work, or did it simply help speed up the fix? I would obviously like to keep the color since that is part of the "payoff" when the drone breaks through the fog. However, if that is not possible, I actually might be able to create something this is visually even more dramatic, with the fog and below done in B&W, and then I mask in the original footage of the mountains. As I indicated earlier, that is a trivial operation with the NLE I use.

I'm obviously very interested in the script you used and hope I can adapt it to my computer and setup. Thank you very much for doing this.

27th July 2016, 17:30   #15  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by johnmeyer The answer is definitely "yeah." Your approach does seem to have pretty much nailed the pulsing. However, I feel like Spock in the original Star Trek "Operation -- Annihilate!" episode when McCoy tries to free him from the pain of a parasite. After the procedure Spock says: "I'm also quite blind. An equitable trade, Doctor. Thank you." So in this case, the pain of the pulsing is gone, but I'm now color blind because the video no longer has any color. Did you ditch the color planes in order to make it work, or did it simply help speed up the fix? I would obviously like to keep the color since that is part of the "payoff" when the drone breaks through the fog. However, if that is not possible, I actually might be able to create something this is visually even more dramatic, with the fog and below done in B&W, and then I mask in the original footage of the mountains. As I indicated earlier, that is a trivial operation with the NLE I use. I'm obviously very interested in the script you used and hope I can adapt it to my computer and setup. Thank you very much for doing this.
relax, it works on chroma as well, I disabled chroma processing cuz I do my algorithm designing stuff on Y and extend that to UV or RGB once I finished the designing like always..
kind of a personal habit

 27th July 2016, 18:06 #16  |  Link feisty2 I'm Siri     Join Date: Oct 2012 Location: void Posts: 2,630 Code: import vapoursynth as vs import mvmulti import Vine import Oyster core = vs.get_core() clp = core.lsmas.LWLibavSource("DJI Inspire Drone Video Long GOP Jerkiness .MOV") clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True) #convert to floating point precision and full range clp = core.std.ShufflePlanes(clp,0,vs.GRAY) #extract Y clp = core.fmtc.resample(clp,1920,1080,kernel="cubic",a1=-1,a2=0) #scale to 2k clp = core.fmtc.transfer(clp,"709","470bg",fulls=True,fulld=True) #brighten it up, dont want dark details all fucked up pre = Vine.helpers.padding(clp,17,17,17,17) #pad, in spatial dimensions pre = core.std.Reverse(pre)+pre+core.std.Reverse(pre) #pad, in temporal dimension pre = core.knlm.KNLMeansCL(pre, 16, 8, 1, 6.4) #very large time radius NLMeans, motion flow and denoising 2 in 1, more of motion flow here pre = core.std.Trim(pre,269,537) #remove padding in temporal dimension pre = core.std.CropRel(pre,17,17,17,17) #remove padding in spatial dimensions #"pre" looks super smooth in motion because of the motion flow alike NLMeans, and super washed out... supv = core.mvsf.Super(pre) #blah supd = core.mvsf.Super(core.std.MakeDiff(clp,pre)) #blah vec = mvmulti.Analyze(supv,tr=16,blksize=32,overlap=16,search=3,badrange=-24) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=16,overlap=8,search=3) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=8,overlap=4,search=3) #blah vec = mvmulti.Recalculate(supv,vec,tr=16,blksize=4,overlap=2,search=3) #blah dif = mvmulti.DegrainN(core.std.Expr(clp,"0.5"), supd, vec, tr=16,thsad=10000,thscd1=10000,thscd2=255) #stabilize the difference between clp and pre clp = core.std.MergeDiff(pre,dif) #merge the difference back #no more washed out now supr = core.mvsf.Super(clp) #blah bv = core.mvsf.Analyze(supv,delta=3,blksize=32,overlap=16,search=3,badrange=-24,isb=True) #delta=3 is good, delta=1,2 ain't effective enough fv = core.mvsf.Analyze(supv,delta=3,blksize=32,overlap=16,search=3,badrange=-24,isb=False) #blah blr = core.mvsf.FlowBlur(clp, supr, bv, fv, 100,thscd1=10000,thscd2=255) #do a full time motion blur here and make the fog look smoother than ever blr = Oyster.helpers.thr_merge(blr,clp,thr=8/256,elast=2/256) #don't want motion blur on the entire frame #only want that on the fog, and fog is kind of the delicate and flat feature here #this filters out motion blur on strong and significant features like the auto and moutains clp = Vine.helpers.cutoff(blr,clp,8) #this filters out motion blur on high frequency features clp = core.grain.Add(clp, 2) #some extra grain here to make it look less... whatever clp = core.fmtc.transfer(clp,"470bg","709",fulls=True,fulld=True) #reverse the brightening clp.set_output() Last edited by feisty2; 27th July 2016 at 18:12.
27th July 2016, 18:26   #17  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by johnmeyer If you can reduce the problem, you're a genius. I'll try to figure out how to upload enough without compromising the business negotiations. The people who own the video are trying to sell it to the car company whose vehicle is featured, and there is some real money involved. I'll start a new thread when I'm ready to do that. I want to see how far I can get by sensing the "jump" points and then using motion estimation to smooth out the video at those points.
I'm a genius obviously by that definition, and I wanna hear you say IT!
I'm so fucking depressed and mad cuz I got matrix mechanics class the entire day and I was goddamn exhausted and my boyfriend bitched about how I can't cook when I got back to my apartment.

really really could use some praising right fucking now

 27th July 2016, 18:33 #18  |  Link johnmeyer Registered User   Join Date: Feb 2002 Location: California Posts: 2,456 Digesting this ... It doesn't look like translating this into an AVISynth script is doable. Vapoursynth is a big hill to climb for this old fart ... need to figure out how many things to download, and which computer I need to use ... I'll post again after I've done a little homework.
27th July 2016, 20:00   #19  |  Link
Reel.Deel
Registered User

Join Date: Mar 2012
Location: Texas
Posts: 1,277
Quote:
 Originally Posted by feisty2 I'm a genius obviously by that definition, and I wanna hear you say IT! I'm so fucking depressed and mad cuz I got matrix mechanics class the entire day and I was goddamn exhausted and my boyfriend bitched about how I can't cook when I got back to my apartment. really really could use some praising right fucking now
Somethings are better left unsaid...

Quote:
 Originally Posted by johnmeyer Digesting this ... It doesn't look like translating this into an AVISynth script is doable.
Looking at feisty's script this should be doable in AviSynth. Not too long ago I had some HFR video and there where parts of the video that was flickering (and pulsating a bit) due to artificial lighting. I tinkered a bit but never came up with anything solid, might have to play around with it some more.

27th July 2016, 20:24   #20  |  Link
feisty2
I'm Siri

Join Date: Oct 2012
Location: void
Posts: 2,630
Quote:
 Originally Posted by Reel.Deel Looking at feisty's script this should be doable in AviSynth.
No it's not, you got no Expr in avisynth and that makes the_merge impossible to port, and lut is useless because the precision is way too low, you need as much precision as you can under "470bg" light
Quote:
 Somethings are better left unsaid...
I'm not that kind to help johnmeyer for nothing, and obviously I'm not getting paid, so it's for my ego, John said that would make me a genius!

Fine, I was kidding... I just want some comfort from a stranger online, feels kind of pathetic now..

Last edited by feisty2; 27th July 2016 at 20:27.