Researchers have developed an innovative system that generates personalized storybooks using generative artificial intelligence and home IoT technology to assist children in language learning. Their findings were published in the *Proceedings of the CHI Conference on Human Factors in Computing Systems* and showcased at the ACM CHI Conference, earning an “Honorable Mention Award” for being among the top 5% of submissions.
Language development is critical for children’s cognitive and academic growth, peer interactions, and overall social development. Regular evaluation of language progress and timely interventions are essential. Traditional methods, often relying on standardized vocabulary lists and pre-made storybooks, fail to address the diverse environments in which children grow up, leading to variations in vocabulary exposure.
To overcome these limitations, the researchers created an educational system tailored to each child’s unique environment. Home IoT devices capture and monitor the language children hear and speak daily. Using speaker separation and morphological analysis, the team examined the vocabulary children were exposed to, the words they spoke, and those they heard but did not vocalize. They assessed each word based on key factors relevant to speech pathology.
Employing advanced generative AI technologies like GPT-4 and Stable Diffusion, the team produced custom children’s books integrating target vocabulary for each child. This personalized approach combines speech pathology theory with practical expertise, creating an effective language learning system. The system allows for individualized weighting of factors and flexible vocabulary selection criteria, continuously updating the storybooks and vocabulary in response to changes in the child’s development and environment.
After testing the system in nine families over four weeks, results showed effective vocabulary learning, demonstrating its applicability beyond therapy rooms. The project was a collaboration between POSTECH’s Department of Computer Science and Engineering and Ewha Womans University’s Department of Communication Disorders.
Lead author Jungeun Lee from POSTECH stated, “We addressed the limitations of traditional approaches by using generative AI to create customized guides tailored to individual needs.” Professor Inseok Hwang, the corresponding author, added, “Our interdisciplinary research developed a personalized language stimulation system integrating AI technology with speech pathology theory, encouraging educators to respect and incorporate diverse learning environments and goals.”
Co-author Professor Dongsun Yim emphasized, “Our work demonstrates the potential for personalized language support services, showcasing the ability to tailor vocabulary extraction and linguistic stimuli for children in varied environments.
