Yes, it should be very much faster
The short coming with some hash_map implementations is that it forgets the memory allocated to elements when it destructs.
As a workaround you may need to do a MinMaxCache.clear(), etc, somewhere in your code, Autolevels::~Autolevels might be a good place.