1. You are not losing quality because of the lack of 10 bit hardware decoding. (If you have a HDR display you might miss out on HDR pass-through you could have with newer card & OS. Though I'm not really sure how quality compares to the other madvr output modes.)
2. What video filter are you using for the decoding? Try LAV Video 64 bit in software mode. (We have
a whole thread dedicated to HEVC decoding performance.)