Storytelling AI

Transforming plain text into immersive multilingual videos
Watch Demo
Storytelling AI Thumbnail

Behind Storytelling AI

Storytelling AI is a bridge between languages, cultures, and stories. Born from the desire to connect diverse communities and break down language barriers, this project aims to bring writers from around the globe to Spanish-speaking audiences.

By seamlessly translating stories from any language into Spanish and crafting captivating narrated videos, I celebrate the diversity of human expression and ensure that every story has the opportunity to be shared and appreciated, regardless of its origin.

Demo Video

Video demonstration of the app's output when encountering an entry from a Thriller genre story:

Gallery

Audio Correction Interface
Fine-tune generated audio for perfect narration
Audio Correction STAI
Story Details Panel
View and edit story metadata and settings
Story Details STAI
Suspense Analysis
AI-powered sentence-level suspense detection
Suspense STAI
Audio Management
Organize and preview generated audio segments
Audios STAI
×
Gallery image full size

Multilingual Narrative Storytelling

Storytelling AI is for now a locally-operating web app that utilizes AI technologies such as OpenAI GPT-4, Whisper, Google Cloud Translation API, XTTS, Stable diffusion XL, and Mistral 7B.

It translates any language story into Spanish, generating narrated videos with music, sound effects, images, and automated summaries. The app also performs sentence-level suspense analysis using SpaCy. Developed with Streamlit, Python and JSX.

Architecture

Storytelling AI Architecture

Technical Overview

Click to explore the system architecture

The architecture of this web application involves receiving the story text through a Streamlit web interface developed in Python. The text is translated from any language to Spanish using the Google Cloud Translation API. Then, the translated text undergoes automated narration using OpenAI GPT-4 and XTTS models, enriched with images from Stable Diffusion XL.

The Mistral 7B model alongside SpaCy are used for natural language processing ( NLP) to dynamically place sound effects in the narration. Finally, the generated elements are combined into a video with music using After Effects Scripting, providing users with a seamless and immersive storytelling experience.

The project's source code is currently not available to the public. I'm working on improving it by restructuring, adding comments, and documenting it, after which it will be made accessible to everyone.

Project Journey

Concept Development

Identified the need for cross-language storytelling tools and began researching AI translation capabilities

Prototype Creation

Built initial text-to-speech pipeline with basic translation functionality

AI Integration

Incorporated advanced AI models for improved translation, narration, and image generation

UI Development

Created intuitive interface for story input, editing, and output generation

Current Status

Functional local application with ongoing improvements and documentation