Transfer characteristics is the gamma, essentially, although most standards aren't pure gamma ramps. You can use linear light this way and let the renderer apply your monitor gamma (if any could), although even at 10bit you're looking at a lot of banding at the low end. (And everything ignores it.)
Color primaries are the RGB points that define the gamut, which is mostly only useful if you have a calibrated monitor. (And all but Quicktime ignores it.)
The matrix is the basic conversion between RGB and YUV. This is the only one you need to care about. Since smpte170m=bt470bg=Rec.601 in H.264, it doesn't matter which one you use, the choice is only there if you want to differentiate between NTSC & PAL sources.
For some background info: bt470bg is PAL, bt470m/fcc is NTSC, smpte170m is really old NTSC, smpte240m is a pre-709 HD standard that is rarely if ever seen.
Now, the SD standards are weird as hell, with real color primaries of smpte170m actually being more common on DVD and broadcast than bt470m, so needing accurate colors is a pipe dream (especially for Never Twice the Same Color), but that should get you started.
Starting at page 252 of
the h.264 spec, you can find way more than you probably wanted to know.