Which version of x264 and MeGUI are you using? "--b-pyramid" without any parameter ("none", "strict", "normal") doesn't work with the newest builds at all. I'd suggest you to first update MeGUI to the newer
patched builds, as the original developers have abandoned the program. If you set "--b-pyramid" correctly after updating MeGUI it should work.