JSA working with knyte is scary....

JSA working with knyte is scary....

Yeah... I'm kind of weirded out that this thread only got attention from (people I would consider) good players.

I was expecting someone on my blacklist to pop in and tell me to do something better with my time. >_<

Nope actually quite impressed with your dedication.
EDIT: Just checked, yes I am on your BL.

Edited 6/17/2015 03:55:32

After a lot of procrastination, I have finally read your thread ;)
I'll be paying more close attention to your posts henceforth :)

I would advise you to break up your posts into small pieces so that it is easier to read, but maybe I knit-pick :P




@JSA
"""Some may be surprised that the 1v1 ladder is not considered as "strategic" as the 2v2 ladder. However, this does not surprise me since I believe team games to have a bigger gap in skill than in 1v1 games. Therefore, there should be more upsets in 1v1 games than team games."""

While I agree with team games requiring more skill than a 1v1, the current 2v2 ladder is heavily influenced by luck due to lack of intel(only 2 starts!) and 16% WR. So I would definitely consider the current 1v1 ladder to be more "strategic" than the current 2v2 ladder.



@ps
"""it's not rocket science, less luck requires more strategy to win."""

I do not agree with that statement entirely. Many players would consider 0%WR to be more strategic than 0%SR(even though it is less deterministic). It is the classic Risk vs Reward problem and I would argue a stronger player would strategize better than a weaker player. However, it is important to analyze this over a significant number of games and not in isolation(where there may be an upset).
With that being said obviously 0%WR is more strategic than 75%WR.

@knyte

"""
I used the rating difference to come up with an expected % of time that the overdog should have won, and compared it to actual results.

So, this relies on the following major assumptions:
-Elo ratings accurately reflect the relative strength of players in terms of how likely they are to win a head-to-head matchup.
-A good Warlight template should have win probabilities that are very close to those predicted by Elo

You can simply subtract the actual overdog win rate from the expected overdog win rate to figure out a template's "bias"- a rating of how likely it makes upsets to happen.
"""



You are performing this analysis over a large sample, but that sample may still not be reflective of the truth. Would it be a fair statement to say that, if you had an infinite number of games on a template, the expected win rate and the observed win rate would converge to be the same value?

That is, the difference that you see, may not actually be bias, but just an inaccuracy due to lack of samples?


For ex- Say I tossed an unbiased coin 10,000 times(large sample). I get 4950 heads and 5050 tails. Does this mean that the probability of heads is 49.5%? Or does this mean that I just haven't observed enough samples. If I had made 100,000 observations it would be closer to 50-50 split.

High strateginess of a game means that luck has relatively smaller effect on the outcome. This will be simply reflected in the higher width of distribution of Elo scores in a population. In chess best player is 2800, worst player is 200, while in lottery everyone will oscillate around 1500.

So just look at the width of final rating distribution for each ladder and that should give you what you are looking for.

@MOTD: That would account for some of the variation but probably not most or all of it. It also wouldn't account for the consistency between the Seasonal Ladders. I forgot my Stats class material but generally with a sample size of 1000, you can expect your data to very closely reflect expected outcomes if it behaves the way it should, theoretically.

The thing is that we don't know what to ground our expectations on- we don't have a system that lets us figured out expected long-term win/loss results on the template itself, only Elo's general model for perfectly strategic scenarios. So, in the end, I end up trying to derive some approximation of the long-term win/loss results on the template itself and compare it to Elo's expected outcomes for a perfectly strategic scenario. There will be some inaccuracy, of course, as I'm comparing experimental data to theoretical data instead of theoretical to theoretical, but it shouldn't make up for the entirety of the bias rating unless every template is perfectly strategic and all these samples are almost consistently inaccurate in the same direction (very improbable).

@TeddyFSB: That's also true. I'm wondering, though, if it only works for lottery-type situations- what about situations where a slightly better player (say, the best and second-best players in 1v1 games) wins all of the time? I think there's a limitation in scenarios where the overdog's win chances are dramatically higher than they should be (so, the other direction). But are those scenarios even unstrategic to begin with- if anything, they'd weigh skill very heavily? I guess I'm really just measuring the probability of upsets (relative to Elo-based predictions) here.

The reason I kind of skipped over that analysis is because I was initially doing this just for CORP Strategic League as a way to weed out bad templates there. Since there's only one 1v1 ladder- not multiple ones on different templates, I wouldn't have been able to perform that analysis there although I have been tracking Elo distributions to make sure the overall ladder isn't luck-based.

In any case, I'll use that analysis to go over tournaments as well. I honestly can't believe I missed that connection.

@ master of the dead

For ex- Say I tossed an unbiased coin 10,000 times(large sample). I get 4950 heads and 5050 tails. Does this mean that the probability of heads is 49.5%? Or does this mean that I just haven't observed enough samples. If I had made 100,000 observations it would be closer to 50-50 split.

there is ways to measure that error e.g. "Binomial test"
lets make a simple calculation based on your numbers
the expected value for tails in your example is 5000 and the mean average error is 50
That means inside the intervall [5050,4950] -times Tails we have an accumulated prabability of 68.3%
and in intervall [5100,4900] -times Tails we have an accumulated prabability of 95.4%
and in intervall [5150,4850] -times Tails we have an accumulated prabability of 99.7%
so if you get outsides these intervalls (especially the last one) the probabillty that your coin was biased gets bigger all the time since its very improbable that you got soch a big deviation from the expected value

Edited 6/17/2015 07:29:59

^ Thanks for that. I was thinking along those lines to come up with some sort of quantitative analysis but I have no idea what the standard deviation here would be so I can't do any sort of t-test-type stuff here.

@ Knyte
2 important Things
1) elo is only a good sculpting method for Player Strengths if ppl don't evolve over time if they do that you have to find a time based exclusion rhythm for old records stopping to matter
2) Beware the Statistic biting you in the back!!! you can't use the data to derive your elo rate than use that elo rate to determine the bias for that same data. The only Bias you get is the accumulated rounding errors in your calculations since elo Ratings are made to get these probability's out.
Sooo the only option is to have a fixed set of ppl with relatively fixed strengths play play 2 tournies the first one to determine their elo rating. The second one to get the bias. Problem here is its nearly impossible for ppl not to learn from a previous experience against a player.
so however much you turn it around its not working.....
Elo (sadly!) is a good method for rating AI's and Mathematicians..... not much more

standard deviation=sqrt(n*p*(1-p))

Darn. :( Thanks.

Alright, so here's the question I began with:

- How do I determine the impact a template has on the likelihood of upsets?

I could simply measure the % of games that ended up in an upset, but a template could end up in 10 games each ending up with a 1490-rated player beating a 1510-rated player and it wouldn't be as worrying as another template that has 10 games with the first 5 involving a 1490-rated player beating a 2000-rated player and the last 5 involving the opposite.

How do I measure the rate of upsets in a way that also accounts for the relative strenghts of the players involved- i.e., weights the upsets based on how unlikely they were to happen?

EDIT: So Corvus and I are having a private convo about this- here's an idea:

Using players' games during a time period to come up with their Elo curve (historical) and use that rating to figure out what sort of bias the template had during a certain game.

So the central challenge here seems to be getting reliable Elo ratings for players to test with.

And actually, I think that tournaments could still be used since we'd be able to come up with players' Elo ratings (reliably) at a given point in time. But that still leaves us with the problem of the template affecting those Elo ratings to begin with.

Edited 6/17/2015 08:07:02

IMO there's nothing more strategic in 16% SR than there is in 0% SR and vice versa
its just that you have to make few adjustments in your attacks(IF NEEDED) to account for luck

with the new updates warlight has made in the analyze attack column its much easier to calculate now and those who were unaware of how it worked earlier can access it with much more ease :)

so basically those who are ignorant have high chances of losing in 16% SR than those(top players) who are not which is an important thing to note because this is somehow misunderstood to be a part of strategy(which i don't like when pro players say it)

^ I agree. The game of Risk is about calculated risk. A small change in the luck factor only adds an equal challenge for both players. The only issue with luck and strategy is how luck modifier changes enable otherwise improbable things to happen (e.g., a player never succeeding at a 3v2 while their opponent always does).

Even hundreds of years later opinions differ still in matter if chess is strategic or psychological game. One should not dvelve into quantitative and qualitative analysis without eliminating first its prejudice.

First of all if you want to measure something (in most cases) make sure you understand what you measure, unless you try to find something new and define it. But in general know what you measure and have a definition beforehand to make it stick, to eliminate options for future interpretation and alteration of results.

Skill, luck, strategy? Are they opposite, co-exist, consist or what?
Good or bad? Unless you do not give any value or clear definition its merely your personal opinion and only dilutes your study/research/method etc.

WR/SR, and luck factor, blind factor?

There is probably no universal definition of strategy, but it does not need to mean that we cant measure it or define it somehow. In general and to simplify I ususally break it up for two: Pure-static strategy and Comparative-Dynamic strategy.
Pure-static Strategy - Chess like gameplay - where all information is known and visible, no random factor unless playing black or white; falls under formal logic, which allows to use in-depth move analysis due to open intel factor- every move has clear and visible effect on further game and can be put into test and under judgement. To sum it down: all possible alternative opponents choises are known to you and their current status of things.
Comparative-dynamic Strategy - More of a general skill how one operates under certain rules. In sense of this partiucal game I would give definition close to that one has to take into considerations all rules, variable factors and changing evironment in closed system. (Closed system: meaning defined map - knowing its not endless; variables: starting positions eiter manual choice or random, fog - which consist blind factor in genral, and rule based variables: luck and WR). To sum it down: Strategy lies in estimating opponents position (calculation risk) under blind factor rules in closed system. [Basic topic-discussion lies what are the factors that give 100% winning advantage to player and if it is true in terms of independent of opponents player choice to avert-prevent it; what measure of rule, luck, random is avert of strategy - if any?]

To give example. Straight round would be definitely Pure Strategy. Weighted random more of second kind as its rule varies and applies differently to every attack and player. Same goes to Fog, automatic starting positions, random wastelands and luck (although here lies the discussion about how big luck and where to draw the line between lottery - hope general logic solves it for now). Of course if we reduce strategy merely on choice, then one of course can claim that buying two lottery tickets will maximise your chance of winning - which in essence is just absurd to define as strategy.

I do not offer full definition or guideline and its merely my opinion although put togethet from various sources and models used in different platforms. Tried to add small WarLight background into it. Basic thing to keep in mind: if you start measuring something just define it, know what you measure and make it logic. Do not leave room for interpretation or mis-results. Mainly: What is strategy to you, or what kind strategy you measure in particular?

Logic is great tool once perfected. In strategy games and analysis one should know how to use, understand and define: system (open-closed), game rules, variables, players-choices, independent-dependent factors. How they supplement each other or actualise in real terms.

Edited 6/17/2015 10:14:26

Returning back to what Corvus said:

1. I've come to agree that the central issue is the unreliability of the Elo ratings. We can't truly know what an "upset" is (or how big an upset is) unless we have reliable Elo ratings for each player. He does have a great idea for getting good baseline Elo ratings, but we'll need some players who're interested in helping us out there.

2. That said, I think the bias ratings are still a lot more than just rounding error. Even when Elo ratings are tainted by a template, the rate of upsets is going to differ significantly. For example, a 2000-Elo rated player (on a lottery template, someone who just got lucky a bunch of times) is still going to lose 50% of games against a 1500-Elo rated player on the same template, but a 2000-Elo rated player on a purely strategic template is going to win a far greater share of games against a 1500-Elo rated player on that template- so even with tainted ratings, we can still meaningfully measure the difference in the probability and magnitude of upsets in a way that's not just accounted for by the Elo ratings. Bad templates still have a greater (or lower) occurrence of upsets, even when just measured in terms of their tainted Elo ratings- simply because of how these templates work.

In other words: what we're actually quantifying is the accuracy of Elo ratings produced by a template. These Elo ratings are predictive- they make claims about what portion of head-to-head matchups two players are going to win in the long run. If the template is bad, it's going to lead to bad Elo ratings- ones that don't actually reflect how likely a player is to win against another player with a certain rating. And we can test that- by looking at the data I'm analyzing and figuring out the rate and magnitude of upsets.

It's like testing a hypothesis- how accurate are the predictions? The hypothesis in this case, of course, is that a template is purely strategic. And the prediction that hypothesis makes is that the outcomes on that template are going to closely reflect Elo-based predictions.

Good template -> reliable Elo ratings -> Elo ratings that are more consistent with actual overdog win rate -> lower bias rating

Bad template -> unreliable Elo ratings -> Elo ratings that are less consistent with actual overdog win rate -> higher bias rating

As far as that "tainting" issue goes, I think it's actually still sound. Perhaps an analysis of a single-template ladder or tournament would actually be more sound based on this logic than an analysis of a single template on a multi-template ladder or tournament, because the Elo ratings in that case would better reflect the impact of the template.

That said, the long-run analysis issue still persists- when we have a consistent Elo rating for each player rather than an Elo curve (which we actually do have in CSL, but right now that's got its own issues), we won't be able to accurately deal with games where the overdog lost because they were just not as good back then as they are now- for example, my first many games on the ladder would probably be considered upsets now since my rating has gone up significantly from then. But in reality, they weren't upsets- I was just a shit player (still am, but used to be even worse back then). There's two solutions to this- either we analyze fewer games or we assign new Elo ratings using a separate process that reflects a player's development. The second is more reliable.

Finally, I think Corvus's process is going to be the most sound solution- we pick 20 or so players, analyze them to get baseline Elo ratings after some realtime games played in certain conditions, and then we use them to test templates. Those baseline Elo ratings would probably be the best way to measure upsets, but there's still the assumption that players have a single skill level across all templates instead of varying skill levels unique to each template. If you find that assumption too big to make, then my original analysis would be the most valid (since all Elo ratings are based on a single template).

I was expecting someone on my blacklist to pop in and tell me to do something better with my time. >_<

If you're hanging out with the right crowd, you won't get that kind of troll.


JSA's the closest thing Warlight has to a historian. If the two of you team up, then maybe you actually will disprove gravity

to answer your other questions knyte :-

is EU 4x5 0% WR just as good as EU 4x4 0% SR?
No!EU 4x5 0% WR simply sucks

Is Rise of Rome too big to be a good 1v1 map?Is it a good 2v2 map then?
it depends from person to person whether they like it or not.i personally like RoR 3v3,4v4 and 5v5
i hate the idea of 1v1 but if its a coin game,any time dude :)

Are Poon Squad's settings really that bad?
consider my 1st answer.now multiply it by 1000

How do I explain to someone that Guiroma 1v1 is actually not a "really weird and bad template" but in fact a solid and well-tested strategic 1v1 template?
don't.if they don't realize it after playing 4/5 games then don't even bother

Edited 6/17/2015 14:08:53

can I get a link to the guiroma template?