Tim Saucer, PMC member and primary maintainer of Apache DataFusion’s Python bindings, joins PlusOne to talk about how he went from frustrated Spark user to open source maintainer — and how the project is using LLM-powered “skills” to keep its Python API in full parity with the upstream Rust library. He walks through the three developer-facing skills they’ve built (expose new APIs, make them Pythonic, audit the user-facing docs) and shares practical advice for other projects looking to adopt a similar approach: treat your skills like test-driven software, constrain the agent to one shot, and use your existing examples as the ground truth.
The conversation also covers what’s next for DataFusion Python, including lambda functions on array columns, distributed compute via DataFusion Comet and Ballista, and how a single-line code change can move a local workflow to a distributed cluster.
Prefer video? That’s on our YouTube channel.
More about DataFusion at https://datafusion.apache.org/