U.S. Citizens traveling to and returning directly from a U.S. territory are not considered to have left the U.S. territory and do not need to present a passport.

U.S. territories include:
• American Samoa
• Guam
• Northern Mariana Islands
• Puerto Rico
• Swains Island
• U.S. Virgin Islands

http://www.dhs.gov/xtrvlsec/crossingborders