VBV will not clamp down the average bitrate of a CRF encode! It can only control bitrate peaks.
Do not expect VBV to save you from having occasionally oversized files. It will not do this! Its sounds like you need to have some shell scripting logic to determine whether a file is actually "too big", and re-encode it at a higher CRF or whole hog two pass as a fallback. Do use VBV set to the profile and level you're using. For example, if you're making an iPod encode at Baseline @ L3, then set the vbv-bufsize and vbv-maxrate to 10000. This will ensure that the encoder will never use more than 10,000,000 bits in any given second, but still allow for instantaneous spikes etc. If you are doing encodes at High @ 4.1, then you can increase this to 62500 for each - as High 4.1 allows for 62.5mbps. You can see the specs for each profile and level on Wikipedia:
http://en.wikipedia.org/wiki/H.264#Levels
Also, I second the notion of using MPlayer's deblocking / denoising filters in place of increasing the deblocker settings in x264. If you can set this up correctly, you will get better overall results, and a sharper overall output.
~MiSfit