Maybe you are going about it backwards. Instead of trying to mask the flat areas, try to mask the not flat areas. For example, things like edges are what you want to preserve right? So if you mask those, its easy to filter the other parts of the image.
|