Last week, OpenAI surprised the world by launching a new artificial intelligence model called Sora. is capable of generating high-resolution video clips from text prompts.
However, these video clips are essentially clever silent films. Now, ElevenLabs, an AI-based voice cloning startup, took the experience a step further by adding background sounds to the videos created by Sora.
ElevenLabs is known for its AI-powered text-to-speech software and an automatic synchronization tool that can translate video speech into over 20 languages “Maintains the original tone and style of the voice.”
Now the company is working on something new viz can generate sounds to accompany seemingly silent videos based on user-provided scene descriptions.
To demonstrate its capabilities, Elevenlabs applied it to some Sora-generated content.“We used text prompts like ‘waves crashing’, ‘metal striking’, ‘birdsong’ and ‘racing car engine’ to generate audio that we overlaid on top of some of our favorite clips from the OpenAI Sora announcement.”the company explained in a blog post.
The exact details of ElevenLabs’ sound effects have not yet been revealed, but the demo shows a series of video clips generated by Sora, accompanied by quite realistic background sounds: from footsteps on a busy street to the hum of the city to beeps and mechanical hums Robots from the future. All of this apparently comes from text-to-audio prompts.
There’s no word yet on when we can expect the sound effects technology to be available, but those interested in learning more can sign up here.