Forensic
7th May 2008, 06:30
Security camera recordings are a multi-billion dollar industry. Police departments and lawyers just want to buy software that provides usable evidence. I have conceived of a novel idea for rebuilding lossless video from lossy MPEG-4 video.
What if a lossless background image could be injected into every frame of a lossy video? This would provide partial lossless data for each of the pixel blocks located at the perimeter of any moving objects. Utilizing these modified pixel blocks, plus the original lossy pixel data, it should be possible to compute the DCT quantization errors and reconstruct the lossless video for those blocks.
A lossless background image could be captured by the same recording unit that provided the lossy video. Only this time the recorder would be set for the highest quality with no compression and with the same view, lighting, resolution and staging that had existed during the lossy recording that now requires enhancement. I understand that this concept will only work for video taken from a fixed position, because the background image must be nearly identical in both videos.
By looking at pixel similarities (and some movement logic) the "moving" objects could be identified and extracted frame by frame from the lossy video, and then imposed onto copies of the background image to create the partially lossless video that I described above.
I understand that having identified a set of known pixels from a block, trying to reverse the quantization loss on the whole block is a difficult process due to the unknown variables. The problem is not linear, but a set of solutions can be found by a brute force algorithm. As the target object moves, the relative positions of these newly defined lossless pixels may occupy different pixel blocks in the adjacent frames. The same pixels, now in the adjacent frame, will also have a set of solutions. With enough adjacent frames (weighted based on proximity to the first frame being adjusted), the potential solution set will be significantly reduced. If there is more than one set, the potential values of each remaining lossy pixels (for that one block) can be average for an extremely close estimate.
As the pixel blocks surrounding moving objects are rebuilt to their lossless state, the newly defined lossless pixels from these blocks could then be inserted onto adjacent frames. As the target object moves, the relative positions of these newly defined lossless pixels may occupy different pixel blocks in the adjacent frames. Some of these blocks (now deeper within the moving object), which had been completely lossy, will now include some lossless data. As newly reconstructed lossless data from several adjacent frames (again weighted based on proximity to the frame being adjusted) is added to each frame, the above process can be repeated to rebuilt additional pixel blocks.
Filter #1: Inpaint lossy moving images atop a lossless background
Filter #2 (Rerun filter until the image is restored as much as possible)
phase #1: Determine the DCT quantization error solution sets for each lossy pixel block of each frame independently
phase #2: For each lossy pixel block, relate identical pixel block locations in adjacent frames to determine which solution set(s) exist for all these frames. If a pixel has more than one solution value, average those values (ignore extreme values as they may be random noise).
phase #3: Inpaint newly calculated lossless pixels (from phase #2); into any pixel block on any frame that contains the same object but now in a different screen position.
With enough filter passes, the entire lossless video could be rebuilt. The beauty of this idea is that success should not be severely affected by frame rate, resolution or the level of compression. A similar concept of extraction and rebuilding pixels could be applied to Wavelet compression.
I do not control any of these videos, and thus suggestions to use better cameras or recorders are not within my control. My goal is to find a solution for "after-the-fact" video enhancement when DCT has removed the correlation between adjacent pixels of different pixel blocks. This is the issue with a majority of existing video equipment. It is why the police can't use most of the video evidence they receive. These are "after the fact" recordings and represent the multi-billion dollar security camera industry.
Nothing I have found helps compensate for DCT errors of MPEG-4 compression. Nor will any other Super-resolution, frame averaging or in-painting process. This is because DCT has removed the correlation between adjacent pixels of different pixel blocks. Attempts to use adjacent pixels for enhancement simply lead to a smoothing/blurring effect.
What do you think of this? If you choose to develop my idea, I would like to be an active Alpha/beta tester during its development and receive a licensed copy. Please post your thoughts and comments.
Does any filter already exist that can identify foreground objects and create a new video of just those objects atop a background image (what I called filter #1)?
Does any filter already exist that can reverse DCT errors when only some pixels of a pixel block are lossless (what I called filter #2)?
DC-CPP
What if a lossless background image could be injected into every frame of a lossy video? This would provide partial lossless data for each of the pixel blocks located at the perimeter of any moving objects. Utilizing these modified pixel blocks, plus the original lossy pixel data, it should be possible to compute the DCT quantization errors and reconstruct the lossless video for those blocks.
A lossless background image could be captured by the same recording unit that provided the lossy video. Only this time the recorder would be set for the highest quality with no compression and with the same view, lighting, resolution and staging that had existed during the lossy recording that now requires enhancement. I understand that this concept will only work for video taken from a fixed position, because the background image must be nearly identical in both videos.
By looking at pixel similarities (and some movement logic) the "moving" objects could be identified and extracted frame by frame from the lossy video, and then imposed onto copies of the background image to create the partially lossless video that I described above.
I understand that having identified a set of known pixels from a block, trying to reverse the quantization loss on the whole block is a difficult process due to the unknown variables. The problem is not linear, but a set of solutions can be found by a brute force algorithm. As the target object moves, the relative positions of these newly defined lossless pixels may occupy different pixel blocks in the adjacent frames. The same pixels, now in the adjacent frame, will also have a set of solutions. With enough adjacent frames (weighted based on proximity to the first frame being adjusted), the potential solution set will be significantly reduced. If there is more than one set, the potential values of each remaining lossy pixels (for that one block) can be average for an extremely close estimate.
As the pixel blocks surrounding moving objects are rebuilt to their lossless state, the newly defined lossless pixels from these blocks could then be inserted onto adjacent frames. As the target object moves, the relative positions of these newly defined lossless pixels may occupy different pixel blocks in the adjacent frames. Some of these blocks (now deeper within the moving object), which had been completely lossy, will now include some lossless data. As newly reconstructed lossless data from several adjacent frames (again weighted based on proximity to the frame being adjusted) is added to each frame, the above process can be repeated to rebuilt additional pixel blocks.
Filter #1: Inpaint lossy moving images atop a lossless background
Filter #2 (Rerun filter until the image is restored as much as possible)
phase #1: Determine the DCT quantization error solution sets for each lossy pixel block of each frame independently
phase #2: For each lossy pixel block, relate identical pixel block locations in adjacent frames to determine which solution set(s) exist for all these frames. If a pixel has more than one solution value, average those values (ignore extreme values as they may be random noise).
phase #3: Inpaint newly calculated lossless pixels (from phase #2); into any pixel block on any frame that contains the same object but now in a different screen position.
With enough filter passes, the entire lossless video could be rebuilt. The beauty of this idea is that success should not be severely affected by frame rate, resolution or the level of compression. A similar concept of extraction and rebuilding pixels could be applied to Wavelet compression.
I do not control any of these videos, and thus suggestions to use better cameras or recorders are not within my control. My goal is to find a solution for "after-the-fact" video enhancement when DCT has removed the correlation between adjacent pixels of different pixel blocks. This is the issue with a majority of existing video equipment. It is why the police can't use most of the video evidence they receive. These are "after the fact" recordings and represent the multi-billion dollar security camera industry.
Nothing I have found helps compensate for DCT errors of MPEG-4 compression. Nor will any other Super-resolution, frame averaging or in-painting process. This is because DCT has removed the correlation between adjacent pixels of different pixel blocks. Attempts to use adjacent pixels for enhancement simply lead to a smoothing/blurring effect.
What do you think of this? If you choose to develop my idea, I would like to be an active Alpha/beta tester during its development and receive a licensed copy. Please post your thoughts and comments.
Does any filter already exist that can identify foreground objects and create a new video of just those objects atop a background image (what I called filter #1)?
Does any filter already exist that can reverse DCT errors when only some pixels of a pixel block are lossless (what I called filter #2)?
DC-CPP