DeepSkill: A methodology for measuring teams’ skills in massively multiplayer online games

Abstract

The game industry is witnessing a significant trend of players toward massively multiplayer online games (MMO). Players are keen on forming teams and cooperating/competing in these games. Real-time measurement of players’ performance is one of the subjects of researchers’ attention to dynamically adjust the game difficulty and immerse players in the game. However, our extensive studies show that real-time measuring of teams’ skill levels has received much less attention. In this paper, a general real-time method called DeepSkill is proposed to measure the MMOs teams’ skills directly using players’ gameplay raw low-level data. The proposed method, which is based on the evidence-centered assessment design, was tested under six different configurations using popular machine learning techniques, including deep neural network (DNN), extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), CatBoost, random forest (RF), and linear support vector regression (LinearSVR). According to the results, the proposed method provides accurate skill estimations and expertise level classifications. Specifically, Deepskill’s DNN-based evidence model provided the lowest mean absolute error of 0.09 in team skill estimation. Additionally, the proposed method achieved an accuracy of 0.973 in classifying the teams’ expertise level for the expert-novice classification task. Furthermore, a cost-effectiveness analysis was performed on the two top-performing evidence models. The LightGBM-based evidence model yielded the best results in both training and prediction phases in terms of low resource consumption alongside considerable accuracy. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023.