PIANO MUSIC PERFORMANCE EVALUATION SYSTEM
Piano performance analysis, evaluation, and explanation
For a piece piano performance by a player, why would we think it is a very good performance? If we think this performance is good, how can I play like him or her? We have designed features for professional labelers to label more than a few thousand piano performance segments.
This is a Multi-disciplined collaboration between the Graduate School of Data Science (Prof. Wen-Syan Li), Dept CSE from the School of Engineering (Prof. Seung-won Hwang), the School of Music (Prof. Jong Hwa Park & Dr. Jeong Mi Park from Seoul National University, and Tongji University Shanghai International College of Design and Innovation(Prof. Xiaohua Sun).
Please follow the link to contact us for acquiring the music performance dataset.
Motivation
For music performance, it is not fair to say only one interpretation is 'right'; even for great musicians, they make different interpretations.
However, we still have some 'preferred' performances, or called to be great. Then, what divides these performances to be great or not?
In this point of view, we might know how difficult it is to teach students in music performing, since tutors cannot say any interpretation is canon.
Especially at a high level, it is not just important to make no mistakes but have to make expressive performances. Then how should a tutor teach student to perform in a proper way, especially in this online era?
There are many interpretations for Bach's Goldberg Variations(BWV 988), length of 38 min to 90 min- especially, Glenn Gould himself made a totally different performance.
GOALS
Our main goals are:
Explain the differences of expressive piano styles.
Rank the performance of expressive piano playing.
Giving visualization of performance including where/what to fix.
For that, we are using MusicXML score data, audio data with recordings of professional musicians.
Scenario for project is this:
First, we make model with recordings of professional musicians to the score.
Second, use XAI for explaining styles, and get style differences from the model.
Lastly, use recordings to extract style(of the student), evaluation for style quality of the recording, and visualization for correction.
For the deliverables of the project, we will get:
Visualization software for educational purpose
Open-source code
Paper publication at conferences
People
As this is interdisciplinary study, instructors & students of various schools joined for the project.
SNU Graduate School of Data Science(GSDS), SNU Department of Computer Science and Engineering(CSE), SNU College of Music work together.
Also people from KAIST mac lab(made VirtuosoNet) helped for basic knowledge.
Faculty:
Prof. Wen-Syan Li
SNU GSDS
Prof. Seung-won Hwang
SNU CSE
Prof. Jong Hwa Park
SNU College of Music
Professional Pianist
Dr. Jeong Mi Park
SNU College of Music
Neuroscience / Psychology of Music
Prof. Sun Xiaohua
Tongji University
Shanghai International College of Design and Innovation
Students:
Jisoo Park
SNU GSDS (Master's)
Background in Mathematics & Music
Integrated perspective of AI & Music
Jisoo Jang
SNU GSDS (Doctor's)
Background in Economics
Data Analytics & Model revision
Jongho Kim
SNU CSE / Artificial Intelligence
(Integrated Ph.D's)
Language and Data Intelligence Lab
Model training & evaluation
Ahyeon Choi
SNU Converge Science & Technology (Doctor's)
Music and Audio Research Group
Audio feature analysis & labeling evaluation
Also, until summer of 2022, two exchange students worked together for this project.
Bjørn Are Therkelsen
Norwegian University of Science and Technology (NTNU) IDI
Angela Ng Weihan
SNU Music School / University of Glasgow
Our Works
Demo
1stver.MP4 사본Our demo shows the simple usage of our final application.
Suppose a teacher teaches a student to play piano. The teacher plays piano first. While playing, the monitor shows performance with midi bars. Another monitor will show the result: we can set the instructor's version as ideal. If not, we can use the default one.
After that, the student plays, and the application shows the result. The purple bar is for the user(student), the yellow bar is for ideal performance to compare. Categorized labels, each representing style is shown, also in a spider chart. Users can also choose dimensions to show, to show a detailed graph with fewer labels.
With this application, the teacher can explain differences in students' performance easily, with details using an exact style label. Students alone can also play again and compare with previous practice.
Data
We used segmented(into a certain number of measures(bars)) midi data of piano performances with labels, which is collected with crowdsourcing. For labeling, qualified music experts worked.
With the labels, music experts of this project worked for the setting. As a result, we got 28 labels to describe music style.
In total, we used Schubert's one of last Piano Sonata(D960), and the number of labeled segments is 847.
If you want to use our dataset, and for more information on it, look at the data set description below:
Model
Preprocessing of recorded performance is needed: here we used Performance Error Detection from Nakamura, to get performance-score aligned. Also got ground truth for crowdsourced data with apex of probability density function of results.
RNN-based GRU / LSTM model is used to get output of styles scorings. With this, we got good correlations between ground truth and predicted value(best 0.787).
With these result, we learned that combining music domain, data, and AI technology is much complicate than first thought. Especially, technical idea and domain knowledge(and vice versa) are hard to meet.
With the lessons learned from works before, we tried to compensate our works and set new topics to discover, with our overall deliverables.
Current Topics
Data
Label, mid-level perceptual, & aggregated feature analysis
Validation and interpretation of the correlation between
• features & features
• features & mid-level aggregated features
Selection of music and analysis of the benefit of training/explanation (i.e. active learning)
Preparation of music and contract for labeling outsourcing
Model
Validation and improvement of model training
Validation and explanation of inference results
3. System
Demo system and usability study with SNU, School of Music and Tongi University, School of Design
See More:
Existing works
Score-Performance Alignment (E. Nakamura et al.)
VirtuosoNet (KAIST mac lab- D. Jeong et al.)
Presentation at SNU Data Science Seminar (6/15/2022)
https://gsds.snu.ac.kr/category/board-50-GN-lOTnwSj3-20210406112509/
XAI_presentation v10.pptxIMG_3195.MOV의 사본Contact
For more information, contact to Jisoo Park of SNU GSDS.
Especially if you need data we collected, please look at our data description page first and contact us.
