I made another test, and have verified my results again:
Standard compile: 59,412,447 bytes
[00:12:52:000] <-- Transcoding Duration
P4-optimized compile: 59,411,930 bytes
[00:09:15:000] <-- Transcoding Duration
That's a difference of 0.0001% in file size. Half a kilobyte.
Keep in mind that small differences such as this will always exist when you switch around between compilers; no two will ever produce precisely the same output for audio encodes. Further, I've seen the same compile produce consistently different output each time you encode the same file, so this ultra-tiny difference should be no concern whatsoever.