In this talk we are going to cover how we have leveraged portability of Beam and make Stream Processing in Python possible on top of Apache Samza. We will first touch points on Apache Samza in general, and how it stands out as the stream processing engine at LinkedIn that scales to over a trillion messages processed per day, with strong state support and flexible deployment models. Next we introduce Samza Runner for Beam, particularly the portable runner. We will cover the efforts we have done to leverage Beam and make stream processing in Python available at LinkedIn, followed by a few use cases. The talk will conclude with our plan for future work.
At LinkedIn, we have a large and ever-expanding corpus of big data processing code written using batch processing frameworks like Pig, Hive, and Spark. These data processing pipelines produce derived data artifacts like metrics, dimensions, and features and are generally run at daily or hourly end-to-end latencies. For a subset of these artifacts, there often arises a need to produce them at a faster cadence, say minutely or even continuously. At that point, the common practice used to be that the developer would re-write the same derivation logic using a stream processing framework (Apache Samza in case of LinkedIn). This causes two problems:
1. The time to go from a pipeline that generates data at hourly grain to a pipeline that generates the same data continuously is governed by developer time to understand the batch code and rewrite it in streaming language; typically weeks to a month to fully operationalize this pipeline.
2. The rewritten pipeline may not be exactly the same as the batch pipeline because of subtle differences in the logic or UDFs across the two code bases. Over time these pipelines can continue to diverge especially if different teams own the batch and streaming pipelines.
In this talk, we present our solution to this batch-stream divide; a system that can auto-generate streaming code from batch logic and employ the Lambda architecture to transparently deliver merged results to data consumers. Technically, our Lambda architecture is very similar to the traditional Lambda architecture; however, our users only need to maintain a single code base in batch logic. This code base serves as a single source of truth to specify what users want to compute. If they need fresher results, they just need to turn on the streaming flag.
We will deep-dive into technical details for this system which is built using open source components: Apache Beam and Apache Calcite. We’ll also share our experience using this system to auto-migrate batch scripts in our unified metrics platform into near realtime pipelines running on Apache Samza and our plans in the future to share this with the community.
Time series processing requires ordered processing of data points, often across windows boundaries. Tricky stuff for distributed processing systems, in this chat we look at how to solve for these problems using a combination of Global Windows, State and the Timer API’ with Apache Beam.
This session will walk through the internals of what the Go SDK does to execute a pipeline. Describing the structure of the SDK, where code is located, and the why of the structure, and how it does it with the particular features of the Go language, which notably has strict types but no generics.
Particular effort will be spent on what the Go SDK does for ease of use, and to reduce DoFn execution overhead.
Using Apache Beam to get data in your data lake? In a agile company you don’t want to re-compile your ingestion pipeline every time a sprint finished. In this talk we go over all mechanisms and building blocks you need to make dynamic pipelines really work.
We’ll see why schemas are so important. How do we get these schemas in our pipelines and discuss methods to protect ourselves from data corruption and incompatible schema evolution.
The new features like schema aware PCollection get a thorough deep dive and finally we go over real world examples and position Apache Beam in the new PLT (Push Load Transform) world.
Access to real-time data is increasingly important for many organizations. At Lyft, we process millions of events per second in real-time to compute prices, balance marketplace dynamics, detect fraud, among many other use cases. To do so, we run dozens of Apache Flink and Apache Beam pipelines. Flink provides a powerful framework that makes it easy for non-experts to write correct, high-scale streaming jobs, while Beam extends that power to our large base of Python programmers.
Historically, we have run Flink clusters on bare, custom-managed EC2 instances. In order to achieve greater elasticity and reliability, we decided to rebuild our streaming platform on top of Kubernetes. In this session, I’ll cover how we designed and built an open-source Kubernetes operator for Flink and Beam, some of the unique challenges of running a complex, stateful application on Kubernetes, and some of the lessons we learned along the way.
Open source draws its strength from the communities that use and build it. It’s their diversity of perspective, skills, and accountability that makes individual projects stronger and builds a richer and solid ecosystem. But while open source is used by the entire world, that broad user community is not yet reflected in the contributor base. In fact, diversity in open source is significantly worse than in proprietary software. While we claim that contribution is open to all, clearly not everybody feels empowered or welcome to contribute to open source projects. And that’s a problem. How can we fix this? How can we enable all developers everywhere to contribute to open source and take advantage of the opportunities it presents? In Q1 2019, Google’s Open Source Strategy team commissioned a user research study to better understand why users do (or do not) contribute back to the project, and how documentation can help remove roadblocks to that project’s adoption and usage – and encourage and enable community contributions. In this talk, we’ll cover the methodology and participant profiles of the study, and describe the four essential user personas we identified – along with their critical user-to-contributor journeys. And we’ll also distill the research to provide concrete recommendations and best practices for creating documentation that helps your community flourish and your project thrive.
When you are starting on your open source adventure, there are lot ofnthings to learn that have very little to do with coding and insteadnrelate to interacting with people. Apache is, at its best, a group ofnpeople who are trying to share their experience and teach new projectsnand contributors how to successfully manage open source projects.nHowever, like the blind people each describing a part of an elephant,neach mentor brings their personal experience to the table, and thusncan give good, yet conflicting advice to new projects. However, thatnaggregate advice has helped many projects to become successful. Based on the author’s experience, this talk will take you through 10ncommon traps in running Apache projects, why they happen, and how tonavoid or mitigate them.
There are lots of companies offering training for, and around, the Apache ecosystem, as well as many other topics that all create their own training material. Keeping this material up to date, especially in the fast-moving open-source world is not an easy task and takes a lot of time and effort. And all the time someone not too far away is probably working on a very similar slide-deck to explain what ZooKeeper does… It is this fundamental issue that the Apache Training project was created to address – centralising training resources and making them easier to access and (re-)use. We believe that training material, while important, is only part of what constitutes good training. The larger part is an experienced trainer who actually understands how the key aspects can be applied in real life and also the consequences of not implementing things correctly.nGreat training tends to come from the experiences and stories that a trainer can share in addition to the actual material. Based on this underlying belief and the very real pain of updating training material the Apache Training project was born. In early 2019, it entered Apache Incubator and is already building a community and gaining various contributions. In this talk, we want to share with you some of the events and the thought process that went into the creation of this project, as well as our main goals and principles. The first part of this talk will focus on how we arrived at the decision that open-source training material is a good thing to have and a little bit of the early history of this project. In the second part, we will discuss what the project is actually about. We will demonstrate the version-control friendly system of designing slides in source control that is at the core of this project. On top of these source files, we aim to create a metadata repository which will enable everybody to efficiently search available content and easily create tailored decks from existing material. And last but very much not least, we will cover how you can contribute to and benefit from this project, as this is front and center an effort from the community for the community!
About 3 years ago, I had an idea of using Open-Source software to create the next generation of industrial it solutions. At ApacheCon 2017 in Miami I introduced this idea to the public with my ‘Building SCADA systems with Apache Software’. 2019 the Apache PLC4X project is an Apache TLP. In this talk I will not talk about technical details of the project itself, but all the steps we took on this journey from a community-building point of view. Starting way before writing the first line of code. After these last two years, I would claim that ‘community-building’ is by far the most challenging task when initiating a new project, but it’s also seems to be one we tend to treat as second-class citizens – even in established projects. Hopefully I will be able to show you how rewarding community-building can be … After all: Apache is all about community over code and it’s that way for a reason
