UEC e-Bulletin
December 2016 Issue
Topics

Evaluating the evaluators: Analysis of big data for peer assessment

Assistant Professor Masaki Uto and Professor Maomi Ueno are pursuing research in areas including machine learning, artificial intelligence, information science, statistics, and educational technology, at the Department of Computer and Network Engineering, Graduate School of Informatics and Engineering, University of Electro-Communications.

"My research is based on Bayesian statistics," says Uto. "I construct and optimize statistical models such as the latent variable model and use machine learning and artificial intelligence to analyze big data for applications including e-Testing, e-Learning."

Recent research includes the development of 'massive rating technology' with the inclusion of modelling parameters accounting for unique characteristics of reviewers for assessing the accuracy and reproducibility of data collected for rating of on line shopping; replies to questionnaires; quality control of cloud sourcing; peer review of reports for massive open online courses; and evaluation of university entrance exams based on written essays.

"The important point in this research is estimating the 'true score' of such ratings," explains Uto. "We incorporate reviewer characteristic parameters in our item response model (latent variable model)."

Uto and colleagues have been successful in assessing a wide range of reviewers using this approach [1]. Importantly, the UEC researchers are collaborating with partners on developing practical applications of their 'rating technology'. Recent examples of real-life applications are:

  • - e-Testing (Computer based testing) [2] with the Benesse Holdings, Inc. and the Information-technology Promotion Agency (IPA).
  • - English language exams with the Eiken Foundation of Japan.
  • - Medical examinations with Common Achievement Tests Organization (CATO)
  • - Selecting reviewers for specific assessments with Recruit Career Co., Ltd.
Further information
Publication

Masaki Uto and Maomi Ueno, Item Response Theory for Peer Assessment, IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES, 9, 157, (2016)
Maomi Ueno Lab, UEC
http://www.ai.lab.uec.ac.jp/index-e/
Masaki Uto
https://sites.google.com/site/utomasakieng/

Evaluating the evaluators
Evaluating the evaluators
Evaluating the evaluators
Assistant Professor:
Masaki Uto (Ph. D. from University of Electro-Communications 2013/09)
Current research areas:
Bayesian Statistics, Machine Learning, Big Data, Data Science, Bayesian Natural Language Processing, Artificial Intelligence, e-Testing, e-Learning, Information Systems
Current research subjects:
Rating big data analysis, Bayesian network, argument mining, e-Testing,
Personal website
https://sites.google.com/site/utomasakieng/