Anime, like everything else in the modern world, is targeted towards specific audiences, both implicitly by the creators, and overtly by the marketing. Though categories are associated with specific sex and ages, this is not a bar on the anime being enjoyed by people who don't fit into that band. Not only do crossover titles exist, but also categories have a considerable following outside their main demographic. This does not invalidate the usefulness of marking a title as one of the categories, they just give a broad impression of an anime, rather than indicating specific content.
This isn't really an "Audience".
While some anime might feature sexual content they still might not be porn. Thats what this "audience" was added for. Only use it when the content is clearly not intended for minors.
For small children. This should be a very clear distinction in most cases. Simple language and animation, vaguely educational themes, and absolutely nothing objectionable.
For 'the family'. Anything that is deliberately targeted at as wide a range of people as possible. Most common for cinematic and other high budget releases that require a very large audience to be financially viable. Think Ghibli.
For boys. Action, ecchi, or sports, all with a gambatai attitude. People tend to get suckered into thinking that some titles are too explicit to be shounen, in general just because something is full of violence and sexual innuendo it doesn't mean the main audience is young teenage boys. In fact, in probably guarantees it. There are certainly conservative elements in japan, Go Nagai for instance has always been criticised for 'not being suitable', but they don't seem to have the same clamp over the media that the christian right does in america. Shounen series can be vastly popular, and have a habit of running for years.
For girls. Stereotypically romance, but can encompass just as wide a range as shounen. My favourite anime series is shoujo.
For young men. Everything from light romance to action gore-fests. The majority of anime produced (that isn't the shounen mega-series) falls into this category.
How to find the 'target audience'
The readership of the manga
A lot of anime is made off the back of a successful manga - as well as providing the material for story it guarantees a fan base. Obviously it would be possible to take an original work and make a completely different anime out of it, but in practice this doesn't happen as the captive audience is part of the reason it's worth taking the large financial risk. As most successful manga is serialised in magazines aimed at particular demographics, the easiest way to find the target audience for an anime is to see what publication the manga it's based on ran in. If it's a Shounen Jump manga, you can be pretty sure that the anime can be labelled 'Shounen' as well. One possible exception I'll raise here. The Rurouni Kenshin manga was serialised in Shounen Jump from 1994 to 1999. The tv series and movie released from 1996 to 1998, despite being pseudo historical and somewhat more serious than other adventure anime, are quite clearly shounen too. However the ovas, released from 1999 to 2001 seem clearly aimed at the Kenshin fans who, having been reading the manga for 5 years, are perhaps now more interested in the maturer themes, and would be more accurately marked as seinen.
The content of the anime
Basically this is *the* way to tell, but can be deeply misleading if you're not familiar with the conventions. You often see westerners saying seinen must be shoujo, because it has romance in it, or shoujo must be shounen because it's got loads of action, or shounen must be seinen because it's to ecchi for younger boys. In short, a good indicator, but make sure there's some discussion on the off chance you've got the wrong end of the stick.
The 'genre' on anidb, ANN or other english fansites
Generally suffer from the problem indicated above of judging on content, so some kind of discussion is still probably helpful. Though perhaps not always, going by the wikipedia seinen talk page... ( http://en.wikipedia.org/wiki/Talk:Seinen )
What to do with stuff that doesn't fit
Easy. Don't add anything. I still want to work out who the moe style titles are aimed at, judging by #anidb it's be late 20s pervs.