TTS audio to video trick using FFmpeg

TTS (Text-to-Speech)
#1

Slightly “off topic”, but hope that won’t be a concern as there is a (tenuous) connection :slightly_smiling_face:

Just thought I’d share a trick for turning TTS audio (or any audio) into a nice little spectrogram video using FFmpeg - this means it’s then easy to share on Twitter (which doesn’t take straight audio uploads otherwise) or for other similar uses.

The result can be seen here: https://twitter.com/nmstoker/status/1276569419267952643

There are a whole load of ways to create the video “image” part besides using a spectrogram (it’s probably best to Google it if you want to explore other things like using a static image etc; I’m not an FFmpeg guru!)

This technique uses the audio to create the spectrogram which it merges with the audio.

So that it works on Twitter the output audio needs to be transcoded to AAC, that happens by default but to have some control over the quality you can make adjustments to the 128k parameter below (I haven’t yet figured out if Twitter keeps the audio or resamples it after upload)

Command is all one line:

ffmpeg -i your_input_audio.wav -filter_complex "[0:a]showspectrum=s=320x240:color=magma,format=yuv420p[v]" -map "[v]" -map 0:a -c:a aac -b:a 128k output_video_file.mp4

If you want a smaller file with just a blank video you can use something like this:

ffmpeg -f lavfi -i color=c=0x0a84ff:s=320x240 -i your_input_audio.wav -c:v libx264 -tune stillimage -pix_fmt yuv420p -shortest -c:a aac -b:a 128k output_video_file.mp4

Details on the showspectrum options here:
http://www.ffmpeg.org/ffmpeg-filters.html#showspectrum-1

Details on the colour options for second example here:
https://ffmpeg.org/ffmpeg-utils.html#Color
(with 0x0a84ff in second example inspired by colours here)

Details on requirements / limitations for uploading video to Twitter here: https://help.twitter.com/en/using-twitter/twitter-videos

4 Likes
#2

This very cool! --------

1 Like
#3

Thanks for sharing, this is great. I couldn’t believe how hard it is for people to play an audio, but somehow they all manage to play video :slight_smile:

1 Like
#4

what version ffmpeg do you use ? It raised an argument error on my side.

#5

I’ve got 4.2.3 installed - not quite the latest (which seems to be 4.3, as here: http://www.ffmpeg.org/download.html)

My command line gives:

ffmpeg version n4.2.3 Copyright © 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)

I saw you’d put a black & white version on Twitter, so did you get the argument error resolved or was it still causing difficulties with the “color” argument?

I did just redo it with a copy-paste from the forum to run it again now (to rule out some corruption of the text I posted) and it does seem to work okay for me, but I can investigate further later (am still “at” work, just on a brief coffee break :slightly_smiling_face:)