J. Kerobo, I. Bukvic, “Physiologically-Enhanced Emotional MIDI: Integrating Biometric Feedback with Symbolic Music Generation,” Recent Advances in Deep Learning Applications, 2026, [in-review]
Abstract
This paper explores the interplay between affect and musical engagement, drawing on psychological theories and empirical studies. We propose extending this exploration into telematic music, investigating the potential for creating an emotional AI collaborator. The study aims to compile a dataset for machine learning by converting physiological responses, self-reported emotions, and musical passages. That data will enable the generation of symbolic music by AI. The relationship between affect (emotional experience) and musical engagement is explored across psychology, neuroscience, and musicology, offering insights into how music evokes emotions and engages individuals cognitively. This exploration involves a social-psychological inquiry into human perceptions and feelings regarding musical engagement, as well as a deeper investigation into the physiological and neuroscientific responses triggered by exposure to musical stimuli. The project aims to emphasize the integration of physiological sensing (biometric feedback) with emotional MIDI data for music generation. It maintains the core focus on symbolic music generation while highlighting the addition of physiological measurements as a key enhancement to the emotional classification process. This involves developing a module for real-time emotion classification and incorporating it into the symbolic music generation pipeline. The continuous-concatenated model demonstrated superior performance with an NLL of 0.7101 and a Top-1 accuracy of 0.8208, outperforming state-of-the-art discrete-token models. Integrating AI, machine learning, and human feedback is proposed to deepen understanding and enable continuous measurement of training data, thereby advancing music information retrieval knowledge and enhancing emotional expression, collaboration, and engagement in musical performance.
Bibliography
[1] T. Kim, M. Chung, E. Jeong, Y. S. Cho, O.-S. Kwon, and S.-P. Kim, “Cortical representation of musical pitch in event-related potentials,” Biomed. Eng. Lett., vol. 13, no. 3, pp. 441–454, Aug. 2023, doi: 10.1007/s13534-023-00274-y.
[2] S. Wang, T. Wang, N. Chen, and J. Luo, “The preconditions and event-related potentials correlates of flow experience in an educational context,” Learn. Motiv., vol. 72, p. 101678, Nov. 2020, doi: 10.1016/j.lmot.2020.101678.
[3] W. James, “II.—What is an Emotion?,” Mind, vol. os-IX, no. 34, pp. 188–205, Apr. 1884, doi: 10.1093/mind/os-IX.34.188.
[4] L. F. Barrett, How emotions are made: The secret life of the brain. in How emotions are made: The secret life of the brain. Boston, MA: Houghton Mifflin Harcourt, 2017, pp. xv, 425.
[5] T. Dalgleish, “The emotional brain,” Nat. Rev. Neurosci., vol. 5, no. 7, pp. 583–589, Jul. 2004, doi: 10.1038/nrn1432.
[6] J. Dewey, “The theory of emotion: I: Emotional attitudes,” Psychol. Rev., vol. 1, no. 6, pp. 553–569, 1894, doi: 10.1037/h0069054.
[7] J. Dewey, “The theory of emotion,” Psychol. Rev., vol. 2, no. 1, pp. 13–32, 1895, doi: 10.1037/h0070927.
[8] M. S. George et al., “Vagus nerve stimulation therapy,” Neurology, vol. 59, no. 6_suppl_4, pp. S56–S61, Sep. 2002, doi: 10.1212/WNL.59.6_suppl_4.S56.
[9] T. Chin and N. S. Rickard, “Emotion regulation strategy mediates both positive and negative relationships between music uses and well-being,” Psychol. Music, vol. 42, no. 5, pp. 692–713, Sep. 2014, doi: 10.1177/0305735613489916.
[10] A. M. Croom, “Music practice and participation for psychological well-being: A review of how music influences positive emotion, engagement, relationships, meaning, and accomplishment,” Music. Sci., vol. 19, no. 1, pp. 44–64, Mar. 2015, doi: 10.1177/1029864914561709.
[11] L. Deckers, Motivation: Biological, Psychological, and Environmental, 6th ed. New York: Routledge, 2022. doi: 10.4324/9781003202646.
[12] F. D’Hondt et al., “Early Brain-Body Impact of Emotional Arousal,” Front. Hum. Neurosci., vol. 4, p. 33, Apr. 2010, doi: 10.3389/fnhum.2010.00033.
[13] C. M. Ghetti, “Active Music Engagement with Emotional-Approach Coping to Improve Well-being in Liver and Kidney Transplant Recipients,” J. Music Ther., vol. 48, no. 4, pp. 463–485, Dec. 2011, doi: 10.1093/jmt/48.4.463.
[14] J. J. Gross and L. F. Barrett, “Emotion Generation and Emotion Regulation: One or Two Depends on Your Point of View,” Emot. Rev., vol. 3, no. 1, pp. 8–16, Jan. 2011, doi: 10.1177/1754073910380974.
[15] E. L. Johnsen, D. Tranel, S. Lutgendorf, and R. Adolphs, “A neuroanatomical dissociation for emotion induced by music,” Int. J. Psychophysiol., vol. 72, no. 1, pp. 24–33, Apr. 2009, doi: 10.1016/j.ijpsycho.2008.03.011.
[16] A. Lamont, “Emotion, engagement and meaning in strong experiences of music performance,” Psychol. Music, vol. 40, no. 5, pp. 574–594, Sep. 2012, doi: 10.1177/0305735612448510.
[17] P. J. Lang, “The Varieties of Emotional Experience: A Meditation on James-Lange Theory.,” Psychol. Rev., vol. 101, no. 2, pp. 211–21, 1994.
[18] J. McCormack, T. Gifford, P. Hutchings, M. T. Llano Rodriguez, M. Yee-King, and M. d’Inverno, “In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, in CHI ’19. New York, NY, USA: Association for Computing Machinery, May 2019, pp. 1–11. doi: 10.1145/3290605.3300268.
[19] R. Schlagowski et al., “Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, in CHI ’23. New York, NY, USA: Association for Computing Machinery, Apr. 2023, pp. 1–16. doi: 10.1145/3544548.3581162.
[20] R. S. Lazarus, Emotion and adaptation. in Emotion and adaptation. New York, NY, US: Oxford University Press, 1991, pp. xiii, 557.
[21] C. D. Wickens, W. S. Helton, J. G. Hollands, and S. Banbury, Engineering Psychology and Human Performance, 5th ed. New York: Routledge, 2021. doi: 10.4324/9781003177616.
[22] C. Bernard, An Introduction to the Study of Experimental Medicine. Courier Corporation, 1957.
[23] N. Schneiderman, G. Ironson, and S. D. Siegel, “STRESS AND HEALTH: Psychological, Behavioral, and Biological Determinants,” Annu. Rev. Clin. Psychol., vol. 1, pp. 607–628, 2005, doi: 10.1146/annurev.clinpsy.1.102803.144141.
[24] W. B. Cannon, Bodily changes in pain, hunger, fear, and rage. D. Appleton and Company, 1915.
[25] W. B. Cannon, “The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory,” Am. J. Psychol., vol. 39, no. 1/4, pp. 106–124, 1927, doi: 10.2307/1415404.
[26] H. Selye, The stress of life, Rev. ed. in The stress of life, Rev. ed. Oxford, England: Mcgraw Hill, 1978, pp. xxvii, 515.
[27] R. E. S. Panda, R. Malheiro, B. Rocha, A. P. Oliveira, and R. P. Paiva, “Multi-Modal Music Emotion Recognition: A New Dataset, Methodology and Comparative Analysis,” in 10th International Symposium on Computer Music Multidisciplinary Research (CMMR 2013), 2013, pp. 570–582. Accessed: Feb. 28, 2024. [Online]. Available: https://estudogeral.uc.pt/handle/10316/94095
[28] L. Ferreira and J. Whitehead, “Learning to Generate Music With Sentiment,” presented at the Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-8, 2019, 2019. Accessed: Feb. 28, 2024. [Online]. Available: http://archives.ismir.net/ismir2019/paper/000045.pdf
[29] H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y.-H. Yang, “EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation,” Aug. 03, 2021, arXiv: arXiv:2108.01374. doi: 10.48550/arXiv.2108.01374.
[30] S. Sulun, M. E. P. Davies, and P. Viana, “Symbolic Music Generation Conditioned on Continuous-Valued Emotions,” IEEE Access, vol. 10, pp. 44617–44626, 2022, doi: 10.1109/ACCESS.2022.3169744.
[31] M. Cook, “Telematic Music: History and Development of the Medium and Current Technologies Related to Performance,” Bowling Green State University, 2015. Accessed: Oct. 05, 2024. [Online]. Available: https://etd.ohiolink.edu/acprod/odb_etd/etd/r/1501/10?clear=10&p10_accession_num=bgsu1447261468
[32] I. Bukvic, “L2Ork » Linux Laptop Orchestra.” Accessed: Apr. 03, 2024. [Online]. Available: http://l2ork.music.vt.edu/main/
[33] I. Bukvic, “L2Ork » L2Ork Tweeter.” Accessed: Mar. 06, 2024. [Online]. Available: https://l2ork.music.vt.edu/main/make-your-own-l2ork/tweeter/
[34] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” Dev. Psychopathol., vol. 17, no. 3, pp. 715–734, 2005, doi: 10.1017/S0954579405050340.
[35] J. A. Russell, “A circumplex model of affect,” J. Pers. Soc. Psychol., vol. 39, no. 6, pp. 1161–1178, 1980, doi: 10.1037/h0077714.
[36] “MIDI.org – Expanding, promoting, and protecting MIDI technology for the benefit of artists and musicians around the world.” Accessed: Oct. 05, 2024. [Online]. Available: https://midi.org/
[37] S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan, “This time with feeling: learning expressive musical performance,” Neural Comput. Appl., vol. 32, no. 4, pp. 955–967, Feb. 2020, doi: 10.1007/s00521-018-3758-9.
[38] C.-Z. A. Huang et al., “Music Transformer,” ArXiv180904281 Cs Eess Stat, Dec. 2018, Accessed: Apr. 15, 2022. [Online]. Available: http://arxiv.org/abs/1809.04281
[39] C.-Z. A. Huang et al., “The Bach Doodle: Approachable music composition with machine learning at scale,” Jul. 14, 2019, arXiv: arXiv:1907.06637. doi: 10.48550/arXiv.1907.06637.
[40] C.-Z. A. Huang, T. Cooijmans, A. Roberts, A. Courville, and D. Eck, “Counterpoint by Convolution,” Mar. 17, 2019, arXiv: arXiv:1903.07227. doi: 10.48550/arXiv.1903.07227.
[41] “MAX30102 Datasheet and Product Info | Analog Devices.” Accessed: Oct. 30, 2024. [Online]. Available: https://www.analog.com/en/products/max30102.html
[42] “Grove – GSR sensor.” Accessed: Apr. 28, 2025. [Online]. Available: https://www.seeedstudio.com/Grove-GSR-sensor-p-1614.html
[43] S. Bakhri, E. Rosiana, and R. C. Saputra, “Design of Low Cost Pulse Oximetry Based on Raspberry Pi,” J. Phys. Conf. Ser., vol. 1501, no. 1, p. 012003, Mar. 2020, doi: 10.1088/1742-6596/1501/1/012003.
[44] M. H. Chyad, S. K. Gharghan, H. Q. Hamood, A. S. H. Altayyar, and S. L. Zubaidi, “A sleep apnea system based on heart rate and SpO2 measurements: Performance validation,” AIP Conf. Proc., vol. 2804, no. 1, p. 040001, Sep. 2023, doi: 10.1063/5.0157203.
[45] S. I. Purnama, M. A. Afandi, R. A. Rochmanto, and D. Prasetyo, “Implementation and Evaluation of Prototype Photoplethysmography for Healthy Person-Based Internet of Things,” in Proceeding of the 3rd International Conference on Electronics, Biomedical Engineering, and Health Informatics, T. Triwiyanto, A. Rizal, and W. Caesarendra, Eds., Singapore: Springer Nature, 2023, pp. 219–235. doi: 10.1007/978-981-99-0248-4_16.
[46] S. F. A. Razak, Y. J. Wee, S. Yogarayan, S. N. M. S. Ismail, and M. F. A. Abdullah, “Real-time monitoring tool for heart rate and oxygen saturation in young adults,” Bull. Electr. Eng. Inform., vol. 13, no. 2, Art. no. 2, Apr. 2024, doi: 10.11591/eei.v13i2.6371.
[47] “Grove Base Hat for Raspberry Pi – 24-Pin GPIO maintain, Grove interface for 3× I2C, 1× UART, 6× Digital, 4× Analog, SWD Debug interface, 1x PWM Port.” Accessed: Apr. 29, 2025. [Online]. Available: https://www.seeedstudio.com/Grove-Base-Hat-for-Raspberry-Pi.html
[48] S. Banik, A. Adithya, B. Mitra, S. Sen, S. Saha, and S. Ghosh, “PResCon: Physiological Response Augmented Continuous Emotion Annotation Tool,” in 2025 17th International Conference on COMmunication Systems and NETworks (COMSNETS), Jan. 2025, pp. 1374–1376. doi: 10.1109/COMSNETS63942.2025.10885663.
[49] J. P. D. Serrano, J. M. A. Soltez, R. K. C. Pascual, J. C. D. Castillo, J. L. Torres, and F. R. G. Cruz, “Portable Stress Level Detector based on Galvanic Skin Response, Heart Rate, and Body Temperature,” in 2018 IEEE 10th International Conference on Humanoid, Nanotechnology, Information Technology,Communication and Control, Environment and Management (HNICEM), Nov. 2018, pp. 1–5. doi: 10.1109/HNICEM.2018.8666352.
[50] J. Moinard, M. Ceccarelli, and M. Russo, “Design and Testing of a Wearable System for Monitoring Car Drivers,” Appl. Sci., vol. 15, no. 4, Art. no. 4, Jan. 2025, doi: 10.3390/app15041930.
[51] M. Adkisson, J. C. Kimmell, M. Gupta, and M. Abdelsalam, “Autoencoder-based Anomaly Detection in Smart Farming Ecosystem,” in 2021 IEEE International Conference on Big Data (Big Data), Dec. 2021, pp. 3390–3399. doi: 10.1109/BigData52589.2021.9671613.
[52] S. Sontowski et al., “Cyber Attacks on Smart Farming Infrastructure,” in 2020 IEEE 6th International Conference on Collaboration and Internet Computing (CIC), Dec. 2020, pp. 135–143. doi: 10.1109/CIC50333.2020.00025.
[53] S. Banik, S. Sen, S. Saha, and S. Ghosh, “Improving Continuous Emotion Annotation in Video Platforms via Physiological Response Profiling,” in 2024 12th International Conference on Affective Computing and Intelligent Interaction (ACII), Sep. 2024, pp. 370–377. doi: 10.1109/ACII63134.2024.00047.
[54] Z. Balogh, K. Fodor, M. Magdin, J. Reichel, J. Kopják, and Š. Koprda, “Feartherapy: Assessing the Impact of Therapeutic Games in Virtual Environments Through Physiological State Measurements,” Mar. 12, 2025, Social Science Research Network, Rochester, NY: 5170830. doi: 10.2139/ssrn.5170830.
[55] D. Eckhoff, J. Schnupp, P. C. Wong, and A. Cassinelli, “When Flames Feel Real in Augmented Reality: Effects of Plausibility and Placement of Virtual Flames on the Burning Hand Illusion and Physiological Responses,” in 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar. 2025, pp. 63–71. doi: 10.1109/VR59515.2025.00031.
[56] A. Walczak, J. Dominiak, M. P. Woźniak, W. Walczak, and K. Grudzień, “ExpreSense: Designing Interactive Necklace to Support Identity Expression Using Biometric and Ambient Input,” in Proceedings of the Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction, in TEI ’25. New York, NY, USA: Association for Computing Machinery, Mar. 2025, pp. 1–8. doi: 10.1145/3689050.3706012.
[57] A. Kargarandehkordi, S. Li, K. Lin, K. T. Phillips, R. M. Benzo, and P. Washington, “Fusing Wearable Biosensors with Artificial Intelligence for Mental Health Monitoring: A Systematic Review,” Biosensors, vol. 15, no. 4, Art. no. 4, Apr. 2025, doi: 10.3390/bios15040202.
[58] Sudimanto, I. Herwidiana Kartowisastro, A. Trisetyarso, and W. Budiharto, “Pain Classification Using Discrete Wavelet Transform Feature Extraction and Machine Learning Techniques,” IEEE Access, vol. 13, pp. 45912–45922, 2025, doi: 10.1109/ACCESS.2025.3545869.