The missing part of existing Audio editors.
Parts of speech and why the others were not good enough
This all began a long time ago when I ran a youtube channel making coding tutorials. It was nothing serious, just simple algorithms, setup & installation videos, repeating labwork into a small video to help me with the exam. Given that I'm not the best speaker nor a conversationalist, the videos would usually operate loosely on a script with a bit of improv. Which for anyone who's done this before, knows that this opens room for a lot of error. Constantly redoing parts, repeating sentences, cutting out all the 'ums and ahhs', annoying breathing or other mouth sounds slipping through. I could make an endless list of all the things that went wrong.
audacity and other audio editors
The problem isn't that these hiccups happen when recording voice-overs. It's that existing audio editors are unapologetic when you make mistakes. Try re-listening to an hour long recording and fixing a small 10 second segment that you didn't quite like. For the non sound engineer or audacity enthusiast, this becomes a real pain point. Now scale that problem times 100. You have all of these mistakes hovering all over your project timeline, things you didn't notice the first time around. Now we can either exhaustively go through each of these one by one and try our best to repair or impute them, or we can just start over from scratch(something I've done a lot). We could also check each recording is ok before moving onto the next, but this requires messing with the audio timeline constantly, which gets annoying with long-form narrations.
Now for me, neither of these are acceptable solutions. I've tried hopping and flipping between different editors: audacity, kdenlive, shotcut, imovie etc,. I really tried to make some of them work, but none of them quite scratch that itch that I wanted to fix.
The Talkback workflow
What I was really looking for this whole time was to emulate the environment of a studio recording booth, also known as, talkback. Where a musician in a recording booth works with a moderator that records individual segments or chunks one by one. Each segment is carefully recorded in isolation until its perfect. The moderator listens to the segment and gives real-time feedback to the singer telling them what to change, timing, more emphasis here, change your tone and pitch there etc,. Allowing segments to pass only when its perfected.
You can emulate this workflow / loop using existing software, but it's not perfect. Editors are not designed for this workflow. I've tried forcing Audacity to behave this way for a long time and essentially gave up. The problem is that you have to fight with the editor to do things in a talkback way. Which adds mental exhaustion, and when you have a high quantity of repetitions, this exhaustion starts to compound over time, especially for long-form narrations which require a lot of work.
Designing the Backend for Parts of speech
Instead of fighting a traditional non-linear editor, I designed the backend of Parts of Speech to strictly enforce the talkback workflow as its core functionality. This no longer makes an open-ended continuous timeline the base, but the data model organizes audio into a Project containing a sequential array of manageable Segments. Each segment is just a self-contained wrapper around audio samples.
So now the talkback workflow is baked directly into the architecture through a finite state machine. The app only holds these finite modes: Idle, Recording, and Reviewing. Now, when you start recording, the incoming microphone data isn't written directly to the project timeline, it's captured in a temporary buffer, which acts as an uncommitted Segment.
The actual workflow
The moment you stop recording, the state explicitly shifts into Reviewing. You are forced to evaluate the take before moving on, Approve or Reject. If you stumble over a word, you dispatch a Reject command, which simply clears the current buffer and returns you to Idle. If the take is perfect, the Approve command commits the current segment into the main timeline of the Project.
Because the app operates on discrete chunks rather than a continuous waveform, repairing past mistakes becomes computationally simple, as well as being frictionless for the end user. If you re-listen to your project and find a flaw in a previous take, you don't need to surgically slice out the bad audio on a timeline. The backend handles this by just retrying the segment. When you re-record and approve, the new segment cleanly overwrites that previous segment in the array, without the user scrambling around with the audio timeline and fitting things together. You can also squeeze forgotten lines between existing segments by just inserting between them.
By strictly separating the audio data into isolated segments and tightly controlling the state transitions, the app naturally guides you through a studio-like recording loop, except you are both the musician and the moderator.
Couldn't this just be a plugin?
This could in theory just be a package you add on top of Audacity or kdenlive. The existing non-linear functions are already present within their architecure, and you could emulate the same state machine loop on top of these editors. However I find that there is an obvious and clear separation of concerns. There is a large difference between getting the audio right and making the audio sound good. Having both of things in the same place can actually get quite messy. Imagine applying your noise reduction, eq, compressor, limiter and then realizing you want to do a take again, ten or twenty takes again?
You can fairly argue that this is the users fault and not the software. However, grouping two different use-cases in the same place will naturally lure the average user into this pitfall. Not everyone is an audio-expert or a sound-engineer. I find that you can avoid this problem entirely by splitting the application in two:
Get the audio right first, then apply your effects. No room for mistakes.