【Research Paper】The Accuracy of Criteria in the Performance Evaluation : An Examination of the Subjective Evaluation in Figure Skating


In this post, I’d like to introduce a Japanese research paper related figure skating.

In Japan, though figure skating is one of the most famous sport, there are not many research papers and researchers related figure skating and almost all research papers were written in only Japanese. So I’m going to try to introduce them in English to foreigners!!

I’m not accustomed to write articles in English. If you have any question ,please comment me!! (my self introduction page is here!!)

Today’s topic is “Whether or not bias exists in figure skating score”.

Paper information


The Accuracy of Criteria in the Performance Evaluation : An Examination of the Subjective Evaluation in Figure Skating.


Norifumi Yukutake (Nihon University)

Reona Fujino (Keio University)


The journal of Japan Economic Research Vol.76 68-92




In this paper, author’s main motivation is to clarify “the possibility of existence of bias among the skating groups”.

Maybe most of you know, in figure skating competitions, skaters are divided into groups of six in order of their performance(Ranking).Therefore, there is a high possibility that skaters with higher abilities gather in the latter half of groups.

This means that if judges evaluate them based on the expectations that latter half groups skaters must perform well, there is a possibility of significant differences of evaluation depending on the groups which skaters were assigned to even if they have almost equal abilities.

They tried to make it clear using “Regression Discontinuity Design”(RDD) which is one of the statistical method and corrected data from 14 competitions (11 World championships, 3 Olympic games / only Men and Women Single).

To come to the point, no bias was observed in terms of TES(Total Elements Score/more objective) but PCS(Program Components Score/more subjective).

Specifically, based on the first group, second group skaters get 2.001(Men’s free skating) , 1.912(Women’s free skating),third group skaters get 2.001×2 (Men) , 1.912×2(Women), forth group skaters get 2.001×3(Men) , 1.912×3(Women) point.

As a result of estimating the score excluding the bias, it was shown that the final rank change for some skaters.

Detailed method

As I mentioned, in figure skating competitions, skaters are divided into groups of six in order of their performance(Ranking).Therefore, even if there is no bias, there is a difference in the average score among the groups.

But, if the latter half of the group skaters benefit from bias,the difference of average score among groups is considered to expand more than their ability.(Fig.1)

In other words ,if such bias exist, discontinuities occur at the group boundaries (6th and 7th, 12th and 13th, 18th and 19th).]The difference of their ability is not so large, but they are in different groups.

So, authors use RDD method to clarify this discontinuities.

They did two approaches.

  1. Assume that the discontinuity is at the boundary.(6th and 7th, 12th and 13th, 18th and 19th)
  2. Assume that the discontinuity is in the group.(5th and 6th, 7th and 8th, 11th and 12th, 13th and 14th, 17th and 18th, 19th and 20th)

Base model

Authors proposed this evaluation function.

Si : Score

Di : Group Dummy (in the case of “Assume the discontinuity is at the boundary”, if skaters rank is 6th, 12th, 18th = 1)

SPC: Short Program score

SBF:Season Best score

TD: Competitions Dummy


u:Error term


11 World Championships ,Men / Women Single (2005~2015)

3 Olympic games, Men / Women Single (2006, 2010, 2014)

Full sample size: Men 332, Women 329

Boundary skaters : Men 80, Women 80

Estimate Result

The result is here.(Writer’s translation)

About “Boundary“, “group dummy” was significant at 1% level in “PCS” in terms of Men and Women. But, “Base Value” and “GOE” were nonsignificant.

About “In group“, “group dummy” was nonsignificant in “PCS”, “Base Value”, “GOE”.

This implies that the PCS may be increased regardless of the skater’s ability only by belonging to the later group.

*** 1% level, **5% level, *10% level

*** 1% level, **5% level, *10% level

On average, if skaters are belong to one next group, they are able to get 2.001 point for Men free and 1.912 point for Women free.

As of 2004, Findlay, L. C. and D. M. Ste-Marie clarified that such bias existed in not only “Presentation” (Like PCS) but also “Technical merit” (Like TES).

(“A Reputation Bias in Figure Skating Judging, “Journal of Sport and Exercise Psychology, Vol.26, pp.154-166.)

Therefore, it can be said that the change of the scoring system contributed to the alleviation of the bias at least Technical score.


I used to play figure skating and also participated in competitions.At that time, there was an tacit recognition that latter group skating order is more advantageous.So, this research paper is really interesting for me!!

And,I’m glad to know that the change of the scoring system may have reduced the bias on technical score!!

I’m so sorry for my bad English skill, but I try to continue to introduce Japanese interesting research paper to foreigners!!

If you have any question, please comment me.


  • このエントリーをはてなブックマークに追加