There is a de facto pecking order in each game, such that each character falls a ‘tier’. In some games (e.g. Hokuto no Ken), this means that everyone only ever plays high-tier characters if they expect to win, whereas in other games (e.g. the latest incarnations of Guilty Gear XX), even low-tier characters are viable to play.
Usually, tiers are either descriptive words (‘good’, ‘great’, ‘god’, ‘strong’, ‘demigod’, ‘crap’, ‘Dan’, ‘Sean’, ‘Roll’, etc.) or indicated by grades (which leaves things more open to interpretation; e.g., you may see the normal ‘A’–‘F’, but also plus signs, ‘A’–‘AAA’, ‘S’–‘SSS’, and so forth; usually, anything above ‘A’ indicates that the character is too good for serious play, but not everyone knows about [much less follows] this rule of thumb).
Once upon a time, Shinseisha in Gamest, and Geibunsha in NEO·GEO Freak, and more recently Enterbrain in Arcadia published tier lists (“character ranks” / “chara ranks” in Japanese terminology). In recent times, however, games have rather unofficial player-created tier lists for the most part. Naturally, the disadvantage to this is that opinion plays a big part in unofficial lists, and plagiarism runs rampant. The advantage to this is that the official publications don’t look dumb when the lists have to be corrected due to new discoveries.
It is usually pretty easy to decide who the strongest and weakest characters are, but in between the extremes, there is often dissension among players (and statistically thinking, rightfully so). Moreover, in highly popular games that have been played for a long time, there is almost inevitably going to be some shifting in strategic approaches that changes which tiers characters fall into.
It is, however, a common misconception that a high-tier character can always win; this is not the significance of tiers, which is just an overview of how characters do on the whole. The individual match-ups are what truly matters at times.
In fact, the population of each character’s use will often be skewed in competition such that the tier listing may change significantly (e.g., Makoto in SFIII:3s is theoretically top tier with even distribution; when you consider that nearly everyone picks top or near-top tier in tournaments, though, she gets dropped a little, because she is strong all-around, but not as strong against other characters that are strong all-around).
Beginners and casual players don’t really have to worry about all this all that much. It’s the skilled players that need to consider statistics to get an edge on their competitors.