The problem has been mentioned in thread
x264 mp4 output borked (rev 295)
I just found an easy way to prevent the extremely large dwRate and dwScale:
at the bottom of your avs script, simply add
assumefps(2997,125) if your clip is @ 23.976fps (in fact, you can also use 24000,1001 to get @ ~23.976024)
EDIT:
If you are going to fix the fps, please use one of the following:
Code:
assumefps(30000,1001) #29.970
assumefps(24000,1001) #23.976
The original code above will produce non-standard fps rate and thus is not recommended. Use the new code instead.
(Although it has been 8 years since I wrote this post, I still want to correct this error because this post is linked from the x264 FAQ post.)