« Night of the Attack Clowns| Main | Things I Learned At The Beach »

What's Wrong With Internet Rating Systems

| | Comments (3)

Is it just me, or do the current batch of internet tagging and rating systems suck? I've been studying and working in this field for years, and it seems to me there are several major problems with the current crop of tools.

  1. They're Single-Dimensional
    • Can you reduce a big hunk of text to just a yes or no? If so, what question would the yes or no answer? Whether you liked it? Whether you wanted to read it? Do you like everything you want to read? I know I don't. I hate reading about war and death, yet they are important topics and I want to know about them.
    • Can you reduce an article to just a number? Many rating systems seem to think so. How about you? Do you like articles with a 5 better than those with a 3? How about an article with a 100? Is it a lot better than an article rated 90? Or just a little? Is 100 better than a 90 like a 10 is better than a 9? Or is a 100 better than a 90 like an 11 is to a 1?
    • Is there some sort of great numbering system up in the sky for all articles? The implication seems to be that there is. Is your article a 2? Well, I guess it is just not as good as all of those 3s out there. Does that make intuitive sense to you? It doesn't to me.

  2. They Have Limited Categories


    • Even assuming that you can reduce an article to a number, what is the category of the number? There are loads of categories of ratings: interesting, timely, funny, well-written, etc. Some sites, like Slashdot, allow you to have a few categories that you choose when you vote on an article (or "moderate" the article in Slashdot-ese.) Some sites, like Digg, let everybody "vote" on an article (or "digg" an article) but it's not really clear what your vote means, except that it is positive.

    • Why do the sites get to choose what categories are important? Isn't the purpose of article rating systems to help you sort through the tons of information on the net? So why do the sites get to decide which categories are important to you or not?

    • Tagging gives you unlimited categories but no ranking. That's kind of going too far in the other direction. Certainly there can be some middle ground.

  3. They Fall Victim to Crowd Acceleration


    • Over and over again, I've noticed that once an article receives a certain number of votes, a "chain reaction" takes place where a lot more people come and vote on it. Nothing succeeds like success, and in the crowd-rating game, people would much rather say something like "heck yeah!" to other people's opinions than step out on a limb and form their own. Rating systems need to address this, instead of encouraging a kind of "mob rules", which does not necessarily generate good information.

    • This also means the system is easy to game. Want to post an article on Digg and get good ratings? Simply write a really good article (that part cannot be faked) and get a couple dozen of your friends to give you a "boost". I estimate that somewhere around 10-50 diggs the "afterburner" kicks in, and then your ranking is just coasting into the stratosphere. I'm not trying to be cynical here. I have noticed the same thing on most all of the other boards.

    • Re-affirming messages fare better than critical thinking messages. Have something good to say about open source? Then post it on the open source board. Let's be honest: The open source board isn't about an honest discussion of open source, it's about re-affirming the groupthink that open source is great. Same thing for politic blogs, fan blogs, etc. These places have attitudes. This means that the ranking results are not indicative of the material, but rather how much the material corresponds to the general animus of the clan that reads it. This is important to the reader because the same article that is highly ranked on one site may have a completely different ranking on another site.

  4. Users Rate the Article Instead of the Reaction


    • Here's one I bet you haven't though about: we ask the users to categorize and rank the article instead of their reaction to it. This is, as the Yiddish say, facacta. Why? Because you are not reading an article because other people liked it, you are reading it because you think you are going to like it. (whatever the word "like" means) This is due to the group logic I mentioned above, and even more so, the differences that we all have as humans. I can guarantee you that an article that the old ladies' sewing club thought was totally fab is not going to rank high on the list of articles that urban gang members think is dope.

    • Instead of the article, you should be rating your reaction. If it made you laugh, maybe it will make me laugh too. Maybe not. But if I know your reaction and my reaction, I can correlate that information. I might not like "good" articles. But I might always find articles funny that you found funny too.

    • Life is about perceptions, not absolute reality. I like the Three Stooges, my wife does not. The Three Stooges are neither good nor bad, we simply have different tastes. What we really want in a global information organization system is predicted user experience, not some absolute hierarchical ranking of all information known to man. I blame this thinking on letting the programmers get too close to the solution. People are not machines, and treating them like machines is counter-productive.

  5. They're Single Provider


    • The worst thing that is wrong with the current crop of internet rating systems is that they are built for only one site. When Slashdot, or Digg, ranks an article, the purpose is for other readers of Slashdot or Digg NOT to read the article, but to visit the page where the article is discussed. The current ranking system is incestuous in that the sites are trying to encourage their existing readers to read more of their material, instead of encouraging the readers to read the actual articles. How many times have you read a bunch of comments on an article without actually reading the article? (Want to know why you do that? I can tell you. It is because you are looking for all of that "nuanced" information that the rating system lacks BEFORE you spend time with the actual article. You are looking for comraderie with a group instead of new information. You are betting that readers of that site will have similar interests and opinions as yours and are trying to use their impressions as a filter. This is a bet that is seldom a good one!)

    • Let's face it: the current batch of article rating systems is seldom much more that a trick to keep readers on some aggregation site. This is good news for the aggregation sites, but not-so-good news for the poor reader. It would be a lot better if the sites functioned better at ranking, but as we have seen above that's not happening.

    • To be even more blunt, there is an economic reason the system is broken. Sites make money by hooking and keeping readers. Right now, in 2006, there is little economic reason for providers to actually solve the problem of helping readers maximize their internet time. To some degree search engines are the best cross-provider ranking systems we have, and they have ranking systems that are kept secret from the users! Strange world.

  6. They Use Primitive Analysis


    • Is experience additive? If 6 people like an article, is that twice as good as if only 3 people liked it? I don't think so. But most of the ranking systems I've seen simply add up a bunch of numbers. Does this make intuitive sense to you? Why would we want to add anything? Why not average? Or use Standard Deviation?

    • There are surely many more tools to use. After all, we are not electing a president, we are trying to solve complex issues. If I am an American, I accept the votes of my fellow countrymen for president because we share a common interest and value system and because the nature of the question forces a singular answer. The is not true on the net. Quite probably the otherwise. Millions of readers go all over the place. And there doesn't have to be "one" best article. Far from it.

  7. The Focus is on the Site, Not the Reader

  8. What all of this means is that the current system is aligned to continue to provide value to the site providers first, the readers second. Improvements can be expected to be in the best interest of the providers. Web sites are fighting for their lives by providing incentives for users to visit and stay, not go somewhere else and find more agreeable information. If I own a web site, it is more profitable for me if you take a few minutes and read an article and comments that you really don't like that much. As long as you are "rewarded" every now and then with an article that you really like a lot, you'll keep coming back.

    Think about it this way: perhaps you could create a web site that only provided the perfect material for the reader. This material would match the mood and needs of the user, ahead of what even they were consciously aware they needed. Every time they read the site, they would be overjoyed that the material was first rate.

    Now reader B comes along. What are the chances that reader B is going to like the same material? Probably about zero. The current game is not matching the reader to the material, it is maximizing the amount of material that all readers would enjoy. They are not solving the problem for the reader, they are solving the problem for the content provider. There's a difference. It's like night and day. One way is the old mass-media business model. The new way is making the system work for the user, instead of making the user work for his rewards.

    That's my opinion, anyway. I think the net is ready to go to the next level. I hope that I can help out a bit. There are so many great sites and so many great writers and so much information that we all should share. As a community we should be striving towards a more integrated system of collective "thinking" about our world problems, not simply taking the old ways we did things and trying to hammer them into the new format. I like all of the sites I have mentioned, and this article really isn't about them. It's about you.

UPDATE: Some commenters have posted that it's not just the article material, it is the quality of the discussion. This view, paraphrased, is "I don't read the articles for the content, I read for the opportunity to contribute and ability to hear multiple viewpoints"

I think this is a great comment. What it means is that it is not only the article, but the comments that are an integral part of the web content. I meant to make that clear in the story, but by using the term "article" instead of "content" I was probably being too specific.

3 Comments

i did not read any actionable suggestions???

I'm working on some technology to address these issues, sarabas. If any one is interested, feel free to contact me at the email above. There is a solution, and I think people will like it a lot better than the current systems. Some of the current systems were great ideas when they got started, but they lack the ability to really bring the net and the readers together.

Simplicity and approximateness are the *point* of most ratings. If you offer a multi-axial thing where users are supposed to rate five different aspects of the page, they mostly just won't bother with it.

Leave a comment

About this Entry

This page contains a single entry by DanielBMarkham published on May 27, 2006 3:30 PM.

Night of the Attack Clowns was the previous entry in this blog.

Things I Learned At The Beach is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Social Widgets





Share Bookmark this on Delicious

Recent Comments

  • e: Simplicity and approximateness are the *point* of most ratings. If read more
  • Daniel Markham: I'm working on some technology to address these issues, sarabas. read more
  • sarabas: i did not read any actionable suggestions??? read more

Information you might find handy
(other sites I have worked on)





Recently I created a list of books that hackers recommend to each other -- what are the books super hackers use to help guide them form their own startups and make millions? hn-books might be a site you'd like to check out.
On the low-end of the spectrum, I realized that a lot of people have problems logging into Facebook, of all things. So I created a micro-site to help folks learn how to log-in correctly, and to share various funny pictures and such that folks might like to share with their friends. It's called (appropriately enough) facebook login help