Enhancement of Sound Function (Cooperation with Sound Generative AI)

In Suite Chidori Engine^®, we are improving the sound function, particularly by developing a cooperation with sound generative AI. Sound generative AI is a technology that uses machine learning algorithms to automatically create and synthesize new audio content, such as music, voices, sound effects, and environmental sounds. Well-known examples of this technology include Suno AI and Udio.

In current game development, it is common practice to either create your own background music and sound effects or select them from existing libraries. Recently, it has become possible to use voices and music generated by generative AI. In Suite Chidori Engine^®, we are considering collaborating with Sound Generative AI on an API to develop a function that generates music in real time based on the emotions and facial expressions of the scenes and characters. This feature enables more flexible sound production, allowing you to instantly switch music to suit the situation in the game.

Specifically, introducing the functions shown in Table 1 is under consideration.

Function	Contents	Techniques
AI API connection	Calls HTTP/REST for generative AI voice synthesis service	Parameter transfer through C++ library/JSON
Scene and character information acquisition	Monitors events and conditions inside the engine	Event handler + Condition management class
Generation instruction generation	Scene/feeling -> Music style mapping (tempo, key, genre)	C++ implementation / JSON specification
Extended playback	Expanded sound capabilities to read a variety of formats	C++ library
Cache	Caches generated music for the same instruction	LRU cache, file system

Table 1 Function overview

The setting can be done in C++ or JSON as follows: Describe the character's facial expression, set the overall atmosphere, and provide a brief summary of the scene in "atmosphere" to automatically generate music that matches the scene. You can also set detailed settings for each scene using "music_style".

The sound generative AI collaboration function operates in the following order.

Table 2 below provides a simple comparison with traditional game development environments.

Item	Conventional issue	After AI integration
Flexibility	Preset BGM	Music changes instantly according to scenes and feelings.
Work efficiency	Requires time for music production and editing.	Automatically generates music with just instructions
User experience	A lot of similar background music	Individual productions enhance immersion.
Maintainability	Increases version management and resources	It only calls the AI when necessary and reuses it in the cache, which can be saved to a file.

Table 2 Comparison with conventional game production environment

In the past, creating or requesting sound, embedding completed data, and assigning indexes for specific scenes took time and effort, even if this was reduced with middleware. However, this feature will make game development easier and faster.

Currently it supports WAV (PCM44, 100Hz and 16-bit Stereo) and OGG (Vorbis Codec). We are currently expanding the functionality to support codecs such as FLAC (192,000Hz and 24-bit Stereo) etc. We are also developing a feature that will allow users to manually enable or disable resampling.

This topic introduced a function that, through collaboration with a sound generative AI currently under development, will enable the automatic generation of music that matches emotions and facial expressions in real time. Since instructions can easily be given in C++/JSON format, the burden on developers is expected to be significantly reduced. Please keep an eye out for future updates of Suite Chidori Engine^®.