Setting Up Multi-Channel Sound Cards in OS X
Here is a procedure that will give you a reasonable chance of getting surround sound in Mac OS X out of a sound system hooked up to a multi-channel USB or FireWire sound card. The procedure is cumbersome and if you do anything wrong in any of the steps, you will either get no sound at all, only stereo, or horrible noise. Good luck.
To be honest, if you are serious about hooking up your Mac to a surround sound system, I recommend to buy a standalone 5.1 decoder instead, connect it to your Mac through an optical cable (with a 3.5mm jack adaptor), and set up your media player to pass through the AC3 or DTS stream. This will be much less hassle, it will work in more media players, it will avoid bugs in the few media players that do support multi-channel sound cards, and you can also use the decoder with any other hardware device that has an optical output.
The Procedure
- Log in using an administrator account. This is the kind of account that allows to install stuff. If you don't do this, OS X will silently ignore any changes you make to the multichannel setup. The good news is that you will normally need to do steps 1 through 7 only once. Those settings should be permanently stored even if you switch to a plain user account.
- Go to System Preferences and in the Sound control panel, select the USB audio output device. Turn up the system output volume all the way to the maximum. Do not change the system volume after this, you should only control the volume either from within your media player or on your hardware sound system.
- Open the ‘Audio Midi Setup’ program in your Applications/Utilities folder.

- Open the menu ‘Audio Midi Setup’ → ‘Preferences’. Go to ‘Audio’ and drag the slider almost all the way to the left(1). This is to avoid that your ears or loudspeakers will explode when you play the channel test sound. Do not set the slider all the way to the left, do it as in this image:

- Select the USB card as output device and set the sampling rate to 44100 or 48000 depending on the rate of the audio you want to play (for a movie it's most likely 48000). Set the gain sliders of all channels to 0dB.
- Set the number of channels to the number of loudspeakers in the sound system connected to the sound card (most likely: “6ch-16bit”).

- Click ‘Configure speakers’ → ‘Multichannel’ and choose “5.1 surround” (or whatever setup you have). Click the buttons to test each speaker and tweak the hardware volume controls on your sound system such that all channels sound equally loud. Don't try to tweak the sliders in Audio Midi Setup, not all of them affect the test volume.

Because VLC is pretty much the only media player that is able to correctly output surround sound with a multi-channel sound card, I will only give instructions for VLC here. If you want to use another media player like Plex, nag its developers about support for multi-channel sound cards or re-read the “To be honest” paragraph above.- In VLC, start playing any sound or movie and while it is playing, choose in the menu: “Audio” → “Audio Device” → {your sound card, probably “USB Audio Device”}. If this menu is greyed out, try playing another sound/movie or restarting VLC. You can use a test sound or movie that loops through all surround channels, to see if they are correct.
- Start playing the movie you wanted to watch. Remember, do not touch your system volume: either use the volume control in VLC or on your sound system. If you're lucky, the channels will be correctly balanced. Quite probably though the rear channels and perhaps also the center channel will be too silent or loud. Fiddle with either the controls on your sound system or the sliders in Audio MIDI Setup until it “sounds right”. Be prepared to re-do this for the next movie you play. If you get sick of this, ask the VLC developers to fix this bug, or re-read the “To be honest” paragraph above.
If you get stereo noise instead of the film soundtrack, you probably configured VLC to pass through the undecoded 5.1 stream. Make sure to disable this.
Mind that you will, of course, only get surround from material that is encoded as surround. You will not get surround out of an MP3 or an audio CD, unless you have installed some kind of plug-in that can ‘inflate’ stereo sound to surround, if such thing exists for VLC.
Also mind that this is not everything. Even when ignoring possible bugs in your media player that can cause the channels to be incorrectly balanced, to get really correct surround sound you should make sure that the LFE channel (often mistakenly called “Subwoofer” channel) is amplified with +10dB relative to the other channels. Moreover, you should not just send only the LFE channel to your subwoofer. Ideally you should do ‘bass management’, i.e. strip the bass off all other channels that have loudspeakers that cannot represent deep bass sounds, and re-route it mixed together with the LFE to your subwoofer. With some luck, your sound system already does this for you. But even otherwise the above procedure will produce decent surround sound that will be good enough for most people.
(1): The reason why this slider needs to be used in this strange way can be found elsewhere on my site. Shame on Apple for using a linear slider in an audio setup application.