// HACKER NEWS — CYBERSECURITY
Automating Immersive Reading
How Storyteller’s forced alignment algorithm works.
All children, except one, grow up. They soon know that they will grow up, and the way Wendy knew was this. One day when she was two years old she was playing in a garden, and she plucked another flower and ran with it to her mother. I suppose she must have looked rather delightful, for Mrs. Darling put her hand to her heart and cried, 'Oh, why can't you remain like this for ever!' This was all that passed between them on the subject, but henceforth Wendy knew that she must grow up. You always know after you are two. Two is the beginning of the end.
Storyteller is now a sprawling ecosystem of software, with a full stack web application, native apps for Android and iOS, KOReader plugins, and upcoming macOS, watchOS, and tvOS apps. We're alpha and beta testing v3 releases of the above, which will bring updated UIs, a huge suite of new library management features, and more. We're really excited to show you what we've been working on!
But at the core of Storyteller is its alignment algorithm. Storyteller can take an ebook and an audiobook that you provide it and align them, finding where each word of the ebook is spoken in the audiobook. It does this automatically, without any input from you. Then it uses the EPUB specification's built-in audio synchronization system, called Media Overlays, to embed the audio and synchronization info into the EPUB. That lets you read your book immersively, with your reader app highlighting each sentence as it's narrated by the narrator, just like in the demo above (which is using real Storyteller alignments!).
Initially, that's all Storyteller was: one Python script. It took one audiobook file and one ebook file, and it output a new ebook file with audio synchronization metadata. At the time, there were a very small number of ereader apps (and zero ereader devices) that could consume these files, which used the EPUB Media Overlay specification for their functionality. I would run the script on my computer, copy the resulting EPUB to my phone, and then use BookFusion's fledging Media Overlay support to read and listen to my books.
Back then, I was completely unfamiliar with the field of forced alignment that I was unwittingly sprinting into headlong. My first alignment algorithm was a wobbly construction of clunky, nested while loops. I felt like I was stumbling through the dark, aware that there was probably light at the end, but unable to see it.
Still, I found a few insights along the way. Existing forced aligners, even ones designed for this specific task of aligning ebooks and audiobooks, struggled with a few challenges common to books, and even my first fledgling attempt handled these (to varying degrees of success):
Ebooks and audiobooks may (and often do) have different chapter orders. For example, content that would be considered frontmatter in an ebook, like a dedication, may be read at the end of the audiobook instead, since audiobooks often attempt to start with the content immediately.
When I sat down to write this book on a whim, I had no idea where the whole project would end up going…
Each format will almost certainly have chapters that the other is missing entirely. Appendices, forewards, tables of contents — these are all almost always skipped in audiobooks. And audiobooks often have small chapters that don't exist in ebooks, as well.