dota2-banner
All News
article-headline
Dota 26 years agoGosu "GosuGamers" Gamers

Pain and Gain: Analyzing the new Re-calibration System


Image courtesy of Deviant Art artist ANG-angg 

A new ranked season is finally upon us, and with it, an opportunity to apply what you've learned this past season to achieve that shiny new Medal. The past six months have tested your resolve, demanded your proficiency, and beckoned your time. The journey was long and not always fair but your overall performance from calibration through-to-the end of the season was appropriate (generally speaking). That being said, the prior ranked system was not without its faults - leaving room for unscrupulous players to exploit various metrics to augment their performance. Valve decided to address communal concerns regarding such exploitation by revamping their re-calibration code for this season. Consequently, the disproportionate fix has spawned a new host of issues - far more detrimental than the initial cause of concern. 

For players committed to rising in rank, re-calibrating evokes an inexorable feeling of anxiety surrounding the uncertainty of outcome, an outcome which largely determines their Dota experience for the next six months. With the new re-calibration system in effect, the aforementioned players have good reason to be anxious this season, as the early results of the “substantial change in rank” Valve promised have been substantial indeed…

Adding Fuel to the Fire


Image courtesy of dotageeks.com

Before we begin analyzing the new system, let’s clarify the changes made with this Reddit response by Jeff Hill, Software Developer at Valve.

As we can see, the factors comprising the re-calibration code for this season remove calibration credit for individual player performance in matches - derived from the variables mentioned above. While excluding in-game performance from the re-calibration formula directly removes the ability for players to abuse certain heroes/items to inflate their score, it also substantially diminishes the impact players now have on their MMR. Thus, ranked players are now subjected to an arbitrary system predicated upon the mismatched pairings it generates. Players that perform well despite lower pairings can look forward to tremendous reductions in MMR should they lose a calibration match under such circumstances. The dire consequence, and subsequent misery of losing a calibration match, conjures a profound resentment for the obvious shortcomings of the new system in place. Inversely, players matched against enemy teammates far above their skill bracket rewards a generous MMR increase should they overcome staggering odds. 

Match outcomes during re-calibration have devolved into little more than a game of high-stakes probability, in which MMR can fluctuate as dramatically as +/- 800-1000 in certain cases. The re-calibration system this season conveys a total lack of regard for players who spent the past six months in the MMR trenches to then have their efforts slighted by ten overly sensitive matches. 

Match quality as a whole may potentially suffer in the wake of skewed re-calibrations, as the correlation between rank and skill has been disrupted, adversely impacting higher-skilled players the most by relegating them to lower brackets. The matchmaking system's historical tendency to facilitate wins and losses in streaks makes surmounting lower skill brackets a stagnant grind for even high-skill players, in which the average win bears little significance in the face of overwhelming loss. At the moment, the average post-calibration win (within the context of even pairings) often results in a 1-2% net gain towards your next Medal's rank (if any at all), while losing can reduce progression tenfold. 

Following the initial backlash of the June 6th re-calibration release, Valve responded with a vague update on June 8th, assuring players they would be correcting the overly sensitive ranked gains and losses that occurred.

While players on the winning end of the system lamented the update, others took comfort in the notion their updated medal would better coincide with their overall performance prior to calibration. What followed was a further reduction in rank for many players attributable to MMR rescaling. Perhaps the level of confusion surrounding questionable placements would be better understood if Valve provided an in-depth breakdown of how much MMR was gained or lost in each calibration match. Based on anecdotal evidence shared on /r/DotA2's Recalibration thread, there appears to be a grave disparity of outcome for players, despite similar circumstances.

There have been numerous claims raised by players that won 7-9 of their 10 calibration matches, to then go on and lose hundreds of MMR despite higher win percentages. Some players speculate that certain calibration matches could have been worth as much as 250 MMR prior to the June 8th update. While any speculation along these lines warrants a healthy dose of skepticism, the deductions drawn from demonstrated results support the theory. If such claims are true, then players can do little more than hope that another update designed to correct the inaccuracies of the first update, will do so correctly.

Where do we go from here?


Image courtesy of dotabuff.com

Now that we've identified the numerous issues of this season's re-calibration, what can be done to avoid them in the future?

First and foremost, ten calibration matches is much too small a sample size to draw upon. As mentioned earlier, players routinely go on extended win and loss streaks before, during, and after re-calibration. The argument that win/loss streaks are entirely attributable to player performance holds little weight within a system that no longer accounts for individual performance. At the very least, overall calibration should consist of anywhere between 25-30 matches (erring on the side of 30). This seems like a lot, and it is, however more matches equates to a more balanced representation of player proficiency. Adding more re-calibration matches to the mix could curb the severity associated with winning or losing a match, matchmaking inadequacies, and seeding related overcompensations.       

The next suggestion is one I have alluded to thus far - bringing performance metrics back into the equation! I do not want to diminish the negative aspects of exploitation and support Valve's decision to address it, however ignoring player performance entirely is not the answer. In a matchmaking system that can leave players feeling restricted at times, exceptional performance is imperative to re-allocate players to more appropriate skill brackets. If anything, individual performance should be the driving force behind MMR, especially within the solo-queue environment. Kills, deaths, assists, hero damage/healing, wards placed, and net worth, are all valid indicators of how players are performing in their roles. While item and ability indicators leave room for abuse, perhaps the correct response is to reduce their impact on MMR, instead of removing it entirely. 

With the current re-calibration system in place, it is more important for players to advantageously attune to the matchmaking system than it is to perform well during their match. Players can do so by queuing during peak hours for their time zone (larger matchmaking pool to select from) and by avoiding re-queuing instantly after a loss (waiting 5-10 minutes) to avoid running into the same group of players - especially during off-peak hours when matchmaking imbalance is more probable. My intention with this advice is not to squander the value of performance, only to bolster its impact given the circumstances.

In Closing

It's important to recognize the level of difficulty associated with balancing one the most complex games ever conceived, and historically Valve has done a great job of doing so. Unfortunately, the increased prevalence of updates over the past few months places immense pressure on developers to come up with fast solutions to multi-faceted problems. This reality has caused a notable shift in design choice and implementation - often feeling more impulsive than sensible. Perhaps the drastic results of the new re-calibration system could have been avoided with a bit more time and testing. The challenge that compels the Dota experience is nothing without balance - adjust accordingly.   

All Esports

Entertainment

GosuBattles

Account