When you pop an audio CD into your PC's drive, your media player software (iTunes, Windows Media Player, or whatever) downloads this information from an online database.
If you rip the CD to your hard drive in .mp3 or another format, the information for each song (including the album cover graphic) is saved in the file's metadata--data about the data in the file. Most modern data file formats, including musical ones, contain metadata. An .mp3 file's metadata includes the song and album titles, artists, composer, and bit rate (not all fields are filled in on all songs).
The great thing about metadata is that it's part of the file, and therefore travels with it. When you move these files to another computer, or onto an .mp3 player, the titles, artists, and other information travels with it.
The real issue with moving music from one computer to another is digital rights management (DRM). This isn't a problem with .mp3 files, which lack DRM (one reason I prefer them). It could be an issue with something you ripped from a CD, depending on the CD and/or the settings you used to rip it. But it will almost certainly be an issue with.mp4 files purchased off of iTunes, or .wma files from another online retailer. Check the retailer's policies about moving your music.