Ride-sharing is a two-sided marketplace; balancing supply and demand with price in real time is critical to maintaining an efficient system. Dynamic pricing creates fairness for drivers (by raising rates when there is a lot of demand) and maintains good experiences for passengers (by satisfying pick-up time SLAs). This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. In this streaming infrastructure, our system consumes a massive number of events from different sources to make these pricing decisions.
Reacting to these events in a cron scheduler based legacy infrastructure with inherent latency becomes a challenge, especially where timely reactions are required to balance market conditions. By leveraging Apache Beam, Lyft’s Streaming Platform powers pricing by bringing together the best of two worlds: ML models in Python and Apache Flink on JVM as the streaming engine.Topics covered in this talk include:
* A brief discussion of dynamic pricing, including motivation and high-level problem formulation
* Comparison of legacy architecture and new streaming architecture
* Overview of streaming platform architecture and technology stack
* Major gains from streaming architecture
* Lessons learned