They don't work as well as they could.

Mapmakers should be encouraged to take ratings and reviews into account. Ratings and reviews tell the mapmaker that the map is good or bad or could be improved. If it could be improved, Version 1.0 should be updated with Version 1.1 (minor changes) or Version 2.0 (more significant changes).

However, the current map page system doesn't provide incentives for mapmakers to care as much as they could. If I make a bad Version 1.0 and it's average rating is justifiably low and it has many negative reviews or reviewers bring up problems (unclear, strange, missing connections, ugly, etc.), what in the current system motivates a mapmaker to issue an update? Currently, if I update a bad map, Version 1.0's reviews and ratings remain, as if they are relevant to the updated version(s). And nobody knows which version is being reviewed or rated.

Reviews show who reviewed the map and when. The most important information that should be shown is WHICH VERSION was reviewed! The map page itself should have the title of the map in bigger letters, with the version number below it. Then, when people read a review about Version 1.0, they can look at the name of the map and see that the current version is 2.1, meaning the old review might not apply.

Average ratings are the best quantitative indicator of the quality of the map. The current system doesn't take into account map updates.

This is how the process works:

1. Version 1.0: New map is published. Maybe there are missing connections. Maybe things aren't as interesting as they could be. Initial ratings will be rather low. Players who visit the map page will see the average rating and comments and be less likely to play the map.

2. A responsible mapmaker sees the average rating and negative reviews and decides to improve the map. Version 1.1 or 2.0 is published. The old reviews remain. The average rating reflects what people thought about Version 1.0. Players who visit the map page will see the average rating and comments and will still be less likely to play the map. The map's adoption rate is significantly negatively affected. https://en.wikipedia.org/wiki/Diffusion_of_innovations

3. The map's initial promotion of Version 1.0 flopped or was underwhelming. The improvements in the updated version (Version 1.1 or 2.0) are not clearly reflected in the average rating or reviews. The updates are not promoted, unless the mapmaker uses the forum to do so. The map's adoption rate is significantly negatively affected.

4. If the mapmaker isn't well known, active, influential, or good at template-making, the improved version of the map basically languishes in the netherworld of 1500 underplayed maps. The mapmaker's efforts in updating the map face an uphill or possibly insurmountable battle against the average rating and review system.

Edited 6/4/2016 04:55:48

Possible Solutions:

1. Average ratings could be weighted (as a function of time and/or based on the version number).

2. The map page could show average ratings per version number, with the most recent version clearly marked.

3. Reviews should indicate version number.

4. Players should be more aware that maps have been updated/improved. The "Filter" on https://www.warlight.net/Maps should have a new category: Updated Maps. Basically, maps that have been updated in the last month or so should automatically be placed in this category.

Edited 6/4/2016 04:56:22

I bring this up because when I published my most recent map, I had to think a bit about these problems.

Ideally, I would have simply updated my old map: https://www.warlight.net/Map/6440-Checkerboard-Naval-War.

At first that's what I did. I went into Multiplayer, Design Maps and clicked on the old map. I made a duplicate version and named it Version 2.0. But then I thought, "If I simply update this map, people who visit the map page probably won't play it or will be confused, due to the average rating and reviews."

Old Map Page:

- 32 Ratings
- Average Rating: 2.8
- Positive Reviews: 3/7
- Negative Reviews: 3/7
- Neutral Reviews: 1/7

If I updated the map to Version 2.0, I would have to contend with the negative reviews that discuss things that exist in Version 1.0 but don't exist in Version 2.0.

I would also have to contend with time and the power of promotion. Version 1.0 has 32 reviews after being on WL for four years! Simply updating to Version 2.0 might need another four years before there are an additional 32 reviews! And then what? If Version 2.0's 32 reviews have an average rating of 3.2, the 64 reviews will have an average rating of 3.0. Does that accurately reflect Version 2.0? Not really.

New maps are promoted; updated versions are not. Four years ago, Checkerboard Naval War was promoted on the community page for a longer time than maps are currently promoted. The rate of map production was slower at that time. So most of those 32 reviews were due in large part to the initial promotion and novelty of the map. An updated version of an old map is not promoted at all. And the website and map page have no way of indicating that a map has been updated. I could write in the map description that the map has been updated. But I have already used all the available space there (see https://www.warlight.net/Map/25058-Checkerboad-Naval-War-20) to describe some of the novelties of the map and to provide example templates, which is more necessary.

If the website had a better system for dealing with updated maps, I would have published the new map as Version 2.0 of the old map. Now, WL has an extra map and extra space and pages when it should have only one map. More clutter.

Edited 6/4/2016 04:57:34

Few thoughts on your ideas!

- I understand the issue, but are map ratings really that important, considering the Rate-Farming and lack of players incentive to rate at all? Have you heard about protest votes, comparative voting, second order rating? This is as flawed as it can get.

- Secondly, I already see a way to rig-the ratings. I make a map, let people vote it what-ever they like, and then issue update, and ask my 500 member clan, my 25 alts etc.. to rate it 5 with highly weighted scale. Not exactly more precise system, is it? And as we vote-judge different things the average of rating is not exactly significantly true estimation of maps quality. Quality means to all us so many different things! It is just a average of map liking to personal valuation criteria, nothing objective in nature.

- Thirdly, if I play game in a new map I except it to be perfect, tested and working! If it has mistakes, it obviously ruins a gameplay for me. As we know we all rate different things, visuals, game-play, scale-size-bonus, emotional win-los etc. People can change their ratings, but how often they do that really?

- The version syntax is free to use. I think, each player should update commentary what was changed with each update. Whether they use 1.1 -> 1.2 -> 1.3 or 1.0 -> 2.0 -> 3.0 is unimportant to most players: question is will I know what changed?

- Do you honestly think, that people play someones map just because specific player made it and the quality of map is secondary?

- I am more than sure than there would be players who would update a map without any changes made? Why are the updates important? We all expect one to launch full map at certain point of time, but improving it is just a process and does not need to be emphasized - forum is already full of the map-update-maker spam for commercialize purposes. And who would exactly benefit form this section in Map criteria - I really don´t see any point for that. If I see a mistake in map I always PM and get instant feedback if it has been changed - I think it is more important, that people would write on review for others to see it and later change if it has been fixed! But again, really who reads them expect the maker of a map?

So the bad system should remain unchanged?

I had a suggestion for solution and that is the freedom of rating. Single player game with sudden surrender is almost the freedom of rating but the real freedom would be if rating would be easily available on map page for all maps. It is exploitable but single player with sudden surrender is exploitable too so this remain unchanged.

I dont think that freedom of rating would cause too significant restructuring on map ranks but I think it would generate a new wave of rating activity, rate new maps or rerate updated maps or just think again anothers ratings.

Support the idea of rating freedom!

some sites where something is updated regularly have two ratings shown: a total ratings value, and a ratings since last update value.

Qi brings up a good point here. This is the same problem Steam is having, actually - early reviews can ruin a game's reputation long after the problems brought up in said reviews have been fixed.

some sites where something is updated regularly have two ratings shown: a total ratings value, and a ratings since last update value.

I think this would work very well, but also include a system where you can look at reviews from different versions, such as on the dwarf fortress wiki, where you can press a button and look at the different versions (example: http://dwarffortresswiki.org/index.php/DF2014:Screw_press).

So the bad system should remain unchanged?

Oh c'mon, don't be like that.

- I understand the issue, but are map ratings really that important, considering the Rate-Farming and lack of players incentive to rate at all? Have you heard about protest votes, comparative voting, second order rating? This is as flawed as it can get.

While it's not a large problem, it certainly is still a problem. Otherwise good maps with problems that are fixed in a later version are held down by reviews from the first version. Low reviews means that when I search by 'Average Reviews', I may not see the map at all if I find a good one before I see that one, even if it's better than the first one I found.

- Secondly, I already see a way to rig-the ratings. I make a map, let people vote it what-ever they like, and then issue update, and ask my 500 member clan, my 25 alts etc.. to rate it 5 with highly weighted scale. Not exactly more precise system, is it? And as we vote-judge different things the average of rating is not exactly significantly true estimation of maps quality. Quality means to all us so many different things! It is just a average of map liking to personal valuation criteria, nothing objective in nature.

Could you not do this before, though? If you see a highly reviewed map, try it out and find out it's actually shit, you're still going to rate it badly. Only difference is that the bad reviews come second, not first. Also, there's plenty of ways to weight review score automatically, if need be.

Edited 6/5/2016 20:19:41

I'll admit I didn't read it that well, but from what I skimmed I have this to refute;

Say a terrible map gets terrible reviews to couple with. So, the mapmaker can just go to V 1.1, have the reviews removed while at the same time, just changing a territory, and overall, maintaining the same quality of map he had before. Thus, the system can be gamed even worse than already.

Say a terrible map gets terrible reviews to couple with. So, the mapmaker can just go to V 1.1, have the reviews removed while at the same time, just changing a territory, and overall, maintaining the same quality of map he had before. Thus, the system can be gamed even worse than already.

Have a review system that checks maps before they're published/updated.

Edited 6/5/2016 21:38:41

Interesting problem. All solutions provided so far are clearly exploitable in some way. I think the best solution may be to have reviews and ratings associated with said version, but with a total rating also present, And allow for map searches to cover both total rating and most updated version ratings.