Hearing the Room Across Language: Building Catch’s Realtime iPhone Interpreter

Learn how to build Catch, a native iPhone app for real-time, continuous interpretation of lectures and talks. Discover the journey from web app to iOS, including testing and beta launch challenges.

Swift SwiftUI AVFoundation WebSockets AVAudioEngine AVAudioSession realtime speech interpretation APIs local iOS persistence

Overview

I’ve been building Catch, a native iPhone app that helps people follow talks, lectures, meetings, panels, classes, and tours in a language they do not fully understand. Catch’s current Listen Mode captures room audio, turns it into live interpreted output, plays interpreted audio through headphones when available, shows transcript text on screen as backup and saves the transcript for review afterward.

I initially built Catch for myself so I could better follow conferences and presentations in Chinese. As I kept working on it, I realised that live interpretation can help people join events, talks, meetings, and community spaces they might otherwise avoid because of language barriers. For someone living in another country, or regularly moving between languages, the goal is to make participation feel more possible instead of leaving them reading, guessing, or sitting outside the conversation.

Catch differs from products like Google and Apple Translate because it is specifically designed for long-form settings where you need continuous and simultaneous interpretation for extended periods of time.

Tech stack

Swift

Swift is an open-source, compiled programming language: it's fast, safe, and expressive for building apps across all Apple platforms and beyond.

Swift is a powerful, multi-paradigm language, originally launched by Apple in 2014 to replace C-based languages for their ecosystem. It was engineered for performance, leveraging the LLVM compiler to produce lightning-fast, optimized machine code. The design prioritizes safety: features like Automatic Reference Counting (ARC) and explicit handling of optionals eliminate entire classes of common bugs (e.g., null pointer dereferencing). Now open-source under the Apache 2.0 license, Swift (current stable version 6.2.1) is the primary language for iOS, macOS, and watchOS development, with growing adoption for server-side and Linux applications.

https://www.swift.org

View projects
SwiftUI

SwiftUI is Apple’s declarative framework for building native user interfaces across all Apple platforms with a single, unified codebase.

This is Apple’s modern UI toolkit, leveraging the power of Swift to deliver a declarative syntax: you state what your UI should do, not how. It’s a unified system, meaning you write one set of code to deploy on iOS, iPadOS, macOS, watchOS, and tvOS (plus visionOS). Development is streamlined via deep integration with Xcode, offering features like real-time previews and automatic synchronization between the UI and app state. For existing projects, SwiftUI offers seamless interoperability with older frameworks like UIKit and AppKit, enabling incremental adoption without a full rewrite.

https://developer.apple.com/swiftui/

View projects
AVFoundation

Apple’s primary framework for high-performance audio and video capture, editing, and playback across all hardware platforms.

AVFoundation provides the essential toolkit for managing time-based media on iOS, macOS, watchOS, and tvOS. It offers granular control over device hardware (cameras and microphones) and media assets (MPEG-4 files and HLS streams). Developers use specific classes like AVPlayer for playback, AVCaptureSession for real-time input, and AVAsset for inspecting metadata. The framework handles complex tasks: multi-track audio mixing, frame-by-frame video processing, and hardware-accelerated encoding (HEVC and ProRes). It manages low-level synchronization and buffer handling so you can focus on building professional-grade media applications.

https://developer.apple.com/documentation/avfoundation

View projects
WebSockets

Secures a persistent, full-duplex TCP connection via a single HTTP handshake, delivering low-latency, bidirectional data streaming for real-time applications.

WebSockets establishes a continuous, two-way communication channel: a critical shift from the request/response cycle of HTTP/1.1. The connection initiates with an HTTP `Upgrade` handshake, switching the protocol to `wss` (secure) on port 443. This persistent link minimizes network overhead, eliminating the need for inefficient HTTP polling. The protocol, standardized as IETF RFC 6455 in 2011, ensures near-instant data transfer: vital for high-performance use cases (e.g., live chat, collaborative editing, financial market data).

https://websocket.org

View projects
AVAudioEngine

Apple's high-level, objective-C based audio engine for constructing complex signal processing chains through a dynamic graph of audio nodes.

AVAudioEngine streamlines real-time audio manipulation on Apple platforms by replacing low-level Core Audio boilerplate with an intuitive node-based architecture. Developers connect specialized nodes (like AVAudioPlayerNode for playback, AVAudioUnitReverb for effects, and AVAudioMixerNode for summing) to build a custom signal processing graph. The engine handles the underlying real-time rendering constraints, hardware integration, and format conversions automatically, allowing apps to easily mix multi-channel audio, apply 3D spatial effects, and capture microphone input with minimal code.

https://developer.apple.com/documentation/avfoundation/avaudioengine

View projects
AVAudioSession

AVAudioSession is Apple's framework-level gateway for configuring how your app's audio interacts with the device's hardware and other active apps.

To build a well-behaved audio app on iOS, iPadOS, or watchOS, you must coordinate with the operating system using AVAudioSession. This singleton object acts as the intermediary between your code and the system's audio engine: it defines your playback behavior through categories like playAndRecord or ambient, manages hardware route changes (such as a user unplugging headphones), and handles critical interruptions like incoming phone calls. By configuring the session's properties, you control whether your app silences background music or mixes with it, ensuring a seamless user experience across the Apple ecosystem.

https://developer.apple.com/documentation/avfaudio/avaudiosession

View projects
realtime speech interpretation APIs

Realtime speech interpretation APIs stream live audio to deliver instant, low-latency speech-to-speech translation and conversational reasoning.

These APIs bypass traditional multi-step pipelines (separating speech-to-text, translation, and text-to-speech) by processing audio natively in a single, low-latency step. Developers use WebSocket or WebRTC connections to stream live voice input and receive translated audio and transcript deltas in real time. Platforms like OpenAI (utilizing gpt-realtime-translate) and Microsoft (with its Live Interpreter API) power seamless cross-lingual communication with sub-second latency, making them ideal for live customer support, multilingual meetings, and interactive voice agents.

https://platform.openai.com/docs/guides/realtime

View projects
local iOS persistence

Local iOS persistence secures, structures, and retains application data directly on a user's device using native frameworks like SwiftData and Core Data.

Modern iOS applications rely on robust local persistence to deliver fast, offline-first experiences. By utilizing Apple's native SwiftData framework (or its mature predecessor, Core Data), developers can model complex object graphs, establish data relationships, and perform lightning-fast queries directly on the device. For lightweight requirements, Swift's Codable protocol paired with the file system or UserDefaults handles basic configuration storage, while SQLite and third-party databases like Realm offer alternative storage engines. Implementing these local storage strategies ensures that user data remains secure, accessible without network connectivity, and instantly available between app launches.

https://developer.apple.com/documentation/SwiftData

View projects