Chris Wenham is the Editor-In-Chief of OS/2 e-Zine! -- a promotion from Senior Editor which means he now takes all the blame.

Lash Back! - Join in the ongoing public discussion with our interactive forum. Be frank, be vicious, you can even be anonymous.

- or -

Blast Back! Send a private message directly to Chris Wenham with your thoughts:

Contact OS/2 e-Zine!

Go to a Printer Friendly version of this page

The Great Ratings Controversy

Summary: Oops, after only a couple of issues with it in effect, we've already found a number of problems with the new software and hardware ratings system. Here's what the mistakes were, and how we'll correct them.

While the rest of the OS/2 e-Zine! staff go off and have fun in Chicago this weekend, I'm stuck back here in New York with the job of getting the new issue finished and online. It's a sacrifice, but we know that many of you aren't able to attend either and it wouldn't be fair to hold the 'zine up. But one side-benefit is that the publisher isn't around while I upload this, so I can at last tell you all how awful Falcon Networking's cafeteria food really is ;-)

But the real reason for this letter is to explain the current situation with the new ratings system that we tried out just over a month ago when the e-Zine! changed ownership from its founder, Trevor Smith, to Falcon Networking - the web company owned by Rexx Files columnist Dirk Terrell and which has been hosting OS/2 e-Zine!, the OS/2 Supersite, EDM/2, the Win32-OS/2 project homepage and the Warpstock Homepage for nearly two years now. What we wanted was a way to sum up the crucial points of the programs we reviewed, such as suitability to task for applications and utilities, playability for games, OS/2 Leverage and others. The scores were then averaged and rounded up to give an Overall score.

Two problems occurred, one with the very first issue to use it, embarrassingly enough. We wanted the new ratings system ready for the games issue where we compared Drilling Billy, Galactic Civilizations Gold and Hopkins:FBI. But one problem with ratings categories that are largely subjective is that it takes time to get used to them. There's also the case where one reviewer might give a program a score different from what another writer would, and have different reasons for it. Right away this invited trouble, and we got it. A complaint came in shortly after publication that the OS/2 Leverage score for Hopkins:FBI was too high. In our ratings system, the OS/2 Leverage score gives you an idea of how much the game makes use of OS/2 technologies such as DART, DIVE, multithreading and so on. In the rush we had overlooked the fact that Hopkins used to be a DOS game, now ported to OS/2. A knee-jerk reaction later and I decided to drop Hopkins' OS/2 Leverage score, recalculating the Overall score to be 3.5 instead of 4. But just because a game used to be for DOS, it doesn't mean that it can't be redesigned to take advantage of an advanced platform. Naturally, this score-change did not bode well with PolyEx, who had already began advertising the score of 4 on their web site.

The second problem arose recently when we reviewed MIDI Station Sequencer and Nota Musica Sequencer. While the author of the reviews stated that MIDI Station Sequencer was the far better product, both still got the same Overall score! The reason was partly due to Nota Musica Sequencer's VoiceType features, which boosted its OS/2 Leverage score. That pushed up the Overall score - which is still just a plain average - to the point where it looked as if it was the same as MIDI Station Sequencer. While VoiceType is nifty, a musician would probably prioritize it lower than other features.

After reviewing the situation, I've decided to keep the ratings system, but drop the Overall score from all future reviews (we toyed with the idea of using weighted averages, but thought it would be too complex). I also wish to personally apologize to PolyEx software for my goof up. Hopkins:FBI deserves its full score of 4.0.

If you have some thoughts on the issue yourself, or think you have some ideas for improving the rating system or OS/2 e-Zine! as a whole, drop by our interactive general feedback forum and let us know!