Justin Kerobo

Home > J. Kerobo, I. Bukvic, “Real-Time Human Classified Emotional MIDI Dataset Integration for Symbolic Music Generation,” ICMLA 2024, 2024, Miami, Florida.

J. Kerobo, I. Bukvic, “Real-Time Human Classified Emotional MIDI Dataset Integration for Symbolic Music Generation,” ICMLA 2024, 2024, Miami, Florida.

Feb 2026 16

Abstract

This paper explores the interplay between affect and musical engagement, drawing on psychological theories and empirical studies. We propose extending this exploration into telematic music, investigating the potential for creating an emotional AI collaborator. The study aims to compile a dataset for machine learning by converting physiological responses, self-reported emotions, and musical passages. That data will allow for the generation of symbolic music from AI. The relationship between affect (emotional experience) and musical engagement is explored across psychology, neuroscience, and musicology, offering insights into how music evokes emotions and engages individuals cognitively. This exploration involves a social-psychological inquiry into human perceptions and feelings towards musical engagement and a deeper investigation into the physiological and neuroscientific responses when exposed to musical stimuli. The project aims to create a new MIDI dataset with real-time emotional data captured from L2Ork Tweeter, a telematic musicking software platform, and integrate it into the symbolic music generation process. This involves the development of a module for real-time emotion classification and incorporation into the symbolic music generation pipeline. Integrating AI, machine learning, and human feedback is proposed to deepen understanding and enable continuous measurement of training data, thereby advancing music information retrieval knowledge and enhancing emotional expression, collaboration, and engagement in musical performance.

Bibliography

  • T. Kim, M. Chung, E. Jeong, Y. S. Cho, O.-S. Kwon, and S.-P. Kim, “Cortical representation of musical pitch in event-related potentials,” Biomedical Engineering Letters, vol. 13, no. 3, pp. 441–454, Aug. 2023. [Online]. Available: https://doi.org/10.1007/s13534-023-00274-y
  • S. Wang, T. Wang, N. Chen, and J. Luo, “The preconditions and event-related potentials correlates of flow experience in an educational context,” Learning and Motivation, vol. 72, p. 101678, Nov. 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0023969020301715
  • W. James, “II.—What is an Emotion?” Mind, vol. os-IX, no. 34, pp. 188–205, Apr. 1884. [Online]. Available: https://doi.org/10.1093/mind/ os-IX.34.188
  • L. F. Barrett, How emotions are made: The secret life of the brain, ser. How emotions are made: The secret life of the brain. Boston, MA: Houghton Mifflin Harcourt, 2017, pages: xv, 425.
  • T. Dalgleish, “The emotional brain,” Nature Reviews Neuroscience, vol. 5, no. 7, pp. 583–589, Jul. 2004, publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/nrn1432
  • J. Dewey, “The theory of emotion: I: Emotional attitudes,” Psychological Review, vol. 1, no. 6, pp. 553–569, 1894, place: US Publisher: Macmillan & Company.
  • ——, “The theory of emotion,” Psychological Review, vol. 2, no. 1, pp. 13–32, 1895, place: US Publisher: Macmillan & Company.
  • M. George, Z. Nahas, D. Bohning, F. Kozel, B. Anderson, J.-H. Chae, M. Lomarev, S. Denslow, X. Li, and C. Mu, “Vagus nerve stimulation therapy,” Neurology, vol. 59, no. 6 suppl 4, pp. S56–S61, Sep. 2002, publisher: Wolters Kluwer. [Online]. Available: https://www.neurology.org/doi/10.1212/WNL.59.6 suppl 4.S56
  • T. Chin and N. S. Rickard, “Emotion regulation strategy mediates both positive and negative relationships between music uses and well-being,” Psychology of Music, vol. 42, no. 5, pp. 692–713, Sep. 2014, publisher: SAGE Publications Ltd. [Online]. Available: https://doi.org/10.1177/0305735613489916
  • A. M. Croom, “Music practice and participation for psychological well- being: A review of how music influences positive emotion, engagement, relationships, meaning, and accomplishment,” Musicae Scientiae, vol. 19, no. 1, pp. 44–64, Mar. 2015, publisher: SAGE Publications Ltd. [Online]. Available: https://doi.org/10.1177/1029864914561709
  • L. Deckers, Motivation: Biological, Psychological, and Environmental, 6th ed. New York: Routledge, Mar. 2022.
  • F. D’Hondt, M. Lassonde, O. Collignon, A.-S. Dubarry, M. Robert, S. Rigoulot, J. Honore´, F. Lepore, and H. Sequeira, “Early Brain-Body Impact of Emotional Arousal,” Frontiers in Human Neuroscience, vol. 4, p. 33, Apr. 2010. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2859881/
  • C. M. Ghetti, “Active Music Engagement with Emotional-Approach Coping to Improve Well-being in Liver and Kidney Transplant Recipients,” Journal of Music Therapy, vol. 48, no. 4, pp. 463–485, Dec. 2011. [Online]. Available: https://doi.org/10.1093/jmt/48.4.463
  • J. J. Gross and L. F. Barrett, “Emotion Generation and Emotion Regulation: One or Two Depends on Your Point of View,” Emotion review, vol. 3, no. 1, pp. 8–16, Jan. 2011. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3072688/
  • E. L. Johnsen, D. Tranel, S. Lutgendorf, and R. Adolphs, “A neuroanatomical dissociation for emotion induced by music,” International Journal of Psychophysiology, vol. 72, no.  1,  pp.  24–33,  Apr.  2009.  [Online].  Available:  https://www.sciencedirect.com/science/article/pii/S016787600800799X
  • A. Lamont, “Emotion, engagement and meaning in strong experiences of music performance,” Psychology of Music, vol. 40, no. 5, pp. 574–594, Sep. 2012, publisher: SAGE Publications Ltd. [Online]. Available: https://doi.org/10.1177/0305735612448510
  • P. J. Lang, “The Varieties of Emotional Experience: A Meditation on James-Lange Theory.” Psychological Review, vol. 101, no. 2, pp. 211–21, 1994. [Online]. Available: https://eric.ed.gov/?id=ej492258
  • J. McCormack, T. Gifford, P. Hutchings, M. T. Llano Rodriguez, M. Yee-King, and M. d’Inverno, “In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ser. CHI ’19. New York, NY, USA: Association for Computing Machinery, May 2019, pp. 1–11. [Online]. Available: https://doi.org/10.1145/3290605.3300268
  • R. Schlagowski, D. Nazarenko, Y. Can, K. Gupta, S. Mertes, Billinghurst, and E. Andre´, “Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23. New York, NY, USA: Association for Computing Machinery, Apr. 2023, pp. 1–16. [Online]. Available: https://dl.acm.org/doi/10.1145/3544548.3581162
  • R. S. Lazarus, Emotion and adaptation, ser. Emotion and adaptation. New York, NY, US: Oxford University Press, 1991, pages: xiii, 557.
  • C. D. Wickens, W. S. Helton, J. G. Hollands, and S. Banbury, Engineer- ing Psychology and Human Performance, 5th ed. New York: Routledge, Sep. 2021.
  • C. Bernard, An Introduction to the Study of Experimental Medicine. Courier Corporation, Jun. 1957, google-Books-ID: xuwhDgAAQBAJ.
  • N. Schneiderman, G. Ironson, and S. D. Siegel, “STRESS AND HEALTH: Psychological, Behavioral, and Biological Determinants,” An- nual review of clinical psychology, vol. 1, pp. 607–628, 2005. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2568977/
  • W. B. Cannon, Bodily changes in pain, hunger, fear, and rage. D. Appleton and Company, 1915, google-Books-ID: uyIJGWZQSpwC.
  • ——, “The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory,” The American Journal of Psychology, vol. 39, no. 1/4, pp. 106–124, 1927, publisher: University of Illinois Press. [Online]. Available: https://www.jstor.org/stable/1415404
  • H. Selye, The stress of life, Rev. ed, ser. The stress of life, Rev. ed. Oxford, England: Mcgraw Hill, 1978, pages: xxvii, 515.
  • R. E. S. Panda, R. Malheiro, B. Rocha, A. P. Oliveira, and R. P. Paiva, “Multi-Modal Music Emotion Recognition: A New Dataset, Methodology and Comparative Analysis,” in 10th International Symposium on Computer Music Multidisciplinary Research (CMMR 2013), 2013, pp. 570–582, accepted: 2021-04-02T19:16:54Z. [Online]. Available: https://estudogeral.uc.pt/handle/10316/94095
  • L. N. Ferreira and J. Whitehead, “Learning to Generate Music With Sentiment,” Mar. 2021, arXiv:2103.06125 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2103.06125
  • H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y.-H. Yang, “EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation,” Aug. 2021, arXiv:2108.01374 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2108.01374
  • S. Sulun, M. E. P. Davies, and P. Viana, “Symbolic Music Generation Conditioned on Continuous-Valued Emotions,” IEEE Access, vol. 10, pp. 44 617–44 626, 2022, conference Name: IEEE Access. [Online]. Available: https://ieeexplore.ieee.org/document/ 9762257/citations?tabFilter=papers#citations
  • M. Cook, “Telematic Music: History and Development of the Medium and Current Technologies Related to Performance,” Ph.D. dissertation, Bowling Green State University, 2015. [Online]. Available: https://etd.ohiolink.edu/acprod/odb etd/etd/r/1501/10?clear= 10&p10 accession num=bgsu1447261468
  • I. Bukvic, “L2Ork  Linux Laptop Orchestra,” Jan. 2009. [Online]. Available: http://l2ork.music.vt.edu/main/
  • ——, “L2Ork L2Ork Tweeter,” Jan. 2020. [Online]. Available: https://l2ork.music.vt.edu/main/make-your-own-l2ork/tweeter/
  • J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology,” Development and psychopathology, vol. 17, no. 3, pp. 715–734, 2005. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2367156/
  • J. A. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, vol. 39, no. 6, pp. 1161–1178, 1980, place: US Publisher: American Psychological Association.
  • “MIDI.org – Expanding, promoting, and protecting MIDI technology for the benefit of artists and musicians around the world.” [Online]. Available: https://midi.org/
  • S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan, “This time with feeling: learning expressive musical performance,” Neural Computing and Applications, vol. 32, no. 4, pp. 955–967, Feb. 2020. [Online]. Available: https://doi.org/10.1007/s00521-018-3758-9
  • C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, I. Simon, Hawthorne, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music Transformer,” Dec. 2018, arXiv:1809.04281 [cs, eess, stat]. [Online]. Available: http://arxiv.org/abs/1809.04281
  • C.-Z. A. Huang, C. Hawthorne, A. Roberts, M. Dinculescu, J. Wexler, L. Hong, and J. Howcroft, “The Bach Doodle: Approachable music composition with machine learning at scale,” Jul. 2019, arXiv:1907.06637 [cs, eess, stat]. [Online]. Available: http://arxiv.org/ abs/1907.06637
  • C.-Z. A. Huang, T. Cooijmans, A. Roberts, A. Courville, and D. Eck, “Counterpoint by Convolution,” Mar. 2019, arXiv:1903.07227 [cs, eess, stat]. [Online]. Available: http://arxiv.org/abs/1903.07227
Real_Time_Human_Classified_Emotional_MIDI_Dataset_Integration_for_Symbolic_Music_Generation__Main_ICMLA_FinalDownload