Since I have had to do some research and experimenting on this, I thought I would share the results in case it would be helpful to anyone else. If it is already covered better somewhere else, please share the discussion link.

I am not crazy about the default Lilypond serif font (New Century Schoolbook). Here is what you can do to change the default serif, sans serif, and monospace fonts for your Lilypond document.

1. First, check to make sure that the font you want to use is available to Lilypond. You can do that with the following terminal command:
lilypond -dshow-available-fonts x

Then do CTRL+F (or command+F) and search for the font name. I found out after much frustrated experimenting with the steps below that not all fonts on the system were available to Lilypond. For instance, Garamond was not available to Lilypond on my system, but Baskerville (my backup) was. If anyone has an explanation or workaround for this phenomenon, please share it!

2. Lilypond controls default fonts with the Pango library. To change your default fonts, add the following code to your paper block:

myStaffSize = #20
#(define fonts
(make-pango-font-tree "Baskerville"
"Gill sans"
"Luxi Mono"
(/ myStaffSize 20)))

The three fonts to list, in order are your default serif (Baskerville for me), sans serif (Gill sans), and monospaced (Luxi Mono). Replace these names with the system names of your preferred font.

So here is what my whole paper block looks like (note I have also customized the width of the staff):

\paper {
#(set-paper-size "letter")
% between-system-space = 4\cm
line-width = 5\in
indent = 0\in

myStaffSize = #20
#(define fonts
(make-pango-font-tree "Baskerville"
"Gill sans"
"Luxi Mono"
(/ myStaffSize 20)))
}

3. To change the font-family for one verse, add one of the following to your lyrics block:

\override LyricText #'font-family = #'serif (Usually unnecessary because this is default for lyrics.)
\override LyricText #'font-family = #'sans
\override LyricText #'font-family = #'typewriter

NOTE: There is nothing that says you have to have an sans serif font as your default Llilypond sans serif font. So if I defined my default serif font as Baskerville and my default sans serif font as Palatino, and put the above code in my text for the first stanza, then that stanza would be Palatino and all the others would be Baskerville. That particular thing would be odd to do, but I can come up with some scenarios where it might be worthwhile.

4. Much more practical: to change the change the font shape (italics) or font series (weight), use the following lines of code.
\override LyricText #'font-shape = #'italic
\override LyricText #'font-series = #'bold

So if I wanted to change my first stanza text to use my pre-defined sans serif font, and be bold and italicized, here's what my lyricmode block looks like:

verseOne = \lyricmode {
\set stanza = "1."
\override LyricText #'font-family = #'sans
\override LyricText #'font-shape = #'italic
\override LyricText #'font-series = #'bold
Then fol -- lows the text of the first stan -- za.
}


5. For more information about fonts, see this page:
http://lilypond.org/doc/v2.18/Documentation/notation/fonts. Make sure to modify the URL to reflect whatever version of Lilypond you have installed.