PlusOne

This presentation will describe the initial experience building and using CassKop, an operator developed for running Cassandra on top of Kubernetes. CassKop works as a usual K8S controller (reconcile the real state with a desired state) and automates the Cassandra operations through JMX. All the operations are launched by calling standard K8S APIs (kubectl apply …) or by using a K8S plugin (kubectl casskop …). CassKop is developed in GO, based on CoreOS operator-sdk framework.nAmong the main features :n- deploying a rack aware cluster (or AZ aware cluster)n- scaling up & down (including cleanups)n- setting and modifying configuration parameters (C* and JVM parameters)n- adding / removing a datacenter in Cassandran- rebuilding nodesn- removing node or replacing node (in case of hardware failure)n- upgrading C* or Java versions (including upgradesstables)n- monitoring (using Prometheus/Grafana)n- … By using local and persistent volumes, it is possible to handle failures or stop/start nodes for maintenance operations with no transfer of data between nodes. Moreover, cassandra-reaper is deployed in K8S and used for scheduling repair sessions (Thanks to Spotify and TheLastPickle teams)nThe Cassandra exporter for Prometheus and backup/restore developed by Instaclustr are also used (Thanks to Instaclustr team) During this session, we will delve into the architecture and implementation of this operator, and share our learnings.

Netflix relies on Apache Cassandra as a critical source of truth database, and while Cassandra is a remarkably resilient database, it does, ever so occasionally, break. This talk explores how complex Cassandra deployments fail in production, but more importantly, the techniques, tools, and approaches our distributed systems engineers use to debug and mitigate these failures. We will first cover software-based failure modes that come either from our software or the software that Cassandra builds upon. For example, retry storms or unbounded queues can effectively overwhelm the database. Cassandra and Linux give you many tools to detect these, and there are numerous strategies you can use to avoid them. Along the way we will also visit common JVM failure modes and how to assess and remediate these. Next, we will cover hardware-based failure modes that come from a typical cloud environment. We will cover the broad classes of drive and network failures that we cope with every day, as well as the metrics and tests we use to detect them. With these understandings, we will learn how to automatically heal these failures and safely meet our latency SLOs. Finally, we will cover systemic failure modes, including complex interactions of multiple components and systems. This will involve a number of concrete failures we observed that involved multiple bugs, system failures, or data modeling issues in combination. At the end of the talk, we hope that the audience leaves understanding how large distributed databases can fail, but also how to use DevOps skills and tools to debug, mitigate, and automate away these failure modes going forward.

We’ve started to provide our stream based data pipeline by using Google Cloud Dataflow and Apache Beam since 2018 fall. It collects event logs from microservices running on GKE, then transforms and forwards the logs to GCS and BigQuery to use for analytics, etc. As you know, implementing and operating streaming jobs are challenging. We’re encountered various issues during that time.
I’d like to share our knowledge on development and operation perspective. There are 3 topics in the development part, 1) Implementing stream jobs with using spotify/scio, 2) How to debug the jobs, especially having DynamicDestination, 3) How to load testing, to ensure our jobs stable. And topics in the next part, 1) How to deploy new jobs safely(with avoiding data loss), 2) How to monitor the jobs and surrounding systems, and misc.

Learning a streaming framework like Apache Beam is exciting but with ‘Hello World!’ examples aren’t fun. There aren’t a lot of free and interesting streaming data sources for beginners to play with, it’s very easy to give up learning if it’s boring. To keep myself learning something new, I find myself need incentives and accomplishments to continue. If you are like me need some motivation to keep learning, this talk will give you some inspiration.
In this talk, I will share my experience of learning Apache Beam by showing demos to you on how I create a streaming data with my custom ‘Marvel Fights Stream Producer’. I will discuss how I went through with the Apache Beam Programming Guide and replaced the official examples with Marvel streaming data I produced. I will also talk about what I learn from creating my custom data stream producer and how that helps me learn Apache Beam better.

Hazelcast Jet is a distributed data processing engine that treats all data as a stream. Jet is built on top of Hazelcast IMDG and thanks to this the Jet cluster can also play the role of the data source and sink. If you use Jet this way, you can achieve perfect data locality and top-of- the-class throughputs.
Jet uses cooperative multithreading (comparable to green threads), its processors correspond to standalone single-threaded tasks that process streams of data. By not depending on OS-level thread scheduling Jet can achieve greater throughput and better saturate the CPU cores. Since the theory underlying Apache Beam has also influenced the development of Jet and there are many conceptual similarities, attempting to write a Runner based on Jet came naturally to us. When we started however the first big obstacle we hit was the relative lack of documentation. Figuring out what exactly a Runner needed to do required reverse engineering existing runners.
A second obstacle turned out to be the generic nature of the user defined functions used by Beam. While Jet’s cooperative execution model allows for great performance, it requires the processing of the data to be non-blocking and this is hard to assure when running completely opaque and generic DoFns.
Yet another difficulty was presented by trying to adapt Jet’s fault tolerance and processing guarantee mechanism for the Runner, a process that, even though difficult, might be more solvable than the above mentioned non-blocking requirement.
Despite these difficulties implementing an initial version of our runner turned out to be quite doable, even though experimental it’s quite capable of running a majority of the existing Beam test suits and we are proud to say that we are now listed on the Capability Matrix. We can also showcase Nexmark test results of our runner and describe possible avenues for future development.

Ride-sharing is a two-sided marketplace; balancing supply and demand with price in real time is critical to maintaining an efficient system. Dynamic pricing creates fairness for drivers (by raising rates when there is a lot of demand) and maintains good experiences for passengers (by satisfying pick-up time SLAs). This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. In this streaming infrastructure, our system consumes a massive number of events from different sources to make these pricing decisions.

Reacting to these events in a cron scheduler based legacy infrastructure with inherent latency becomes a challenge, especially where timely reactions are required to balance market conditions. By leveraging Apache Beam, Lyft’s Streaming Platform powers pricing by bringing together the best of two worlds: ML models in Python and Apache Flink on JVM as the streaming engine.Topics covered in this talk include:
* A brief discussion of dynamic pricing, including motivation and high-level problem formulation
* Comparison of legacy architecture and new streaming architecture
* Overview of streaming platform architecture and technology stack
* Major gains from streaming architecture
* Lessons learned

Part I
Developing ML and deep learning applications to be deployed in production is much more than just training a model. Google has taken years of experience in developing production ML pipelines and offered the open source community TensorFlow Extended (TFX), an open source version of the ML platform that Google uses internally. Pipeline processing is a core requirement of any production ML platform, and the TFX has chosen Apache Beam to implement their pipeline.
Learn from Google’s experience in applying Beam for ML pipelines, including how TFX uses Beam and why Beam was chosen.

Part II
Ludwig is a code-free Deep Learning toolbox based on TensorFlow open-sourced by Uber AI Labs. Ludwig is unique in its ability to help make deep learning easier to understand for non-experts and enable faster model improvement iteration cycles for experienced machine learning developers and researchers alike. By using Ludwig, experts and researchers can simplify the prototyping process and streamline data processing so that they can focus on developing deep learning architectures rather than data wrangling.
Ludwig introduces the notion of data type-specific encoders and decoders, which results in a highly modularized and extensible architecture: each type of data supported (text, images, categories, and so on) has a specific preprocessing function.
In this talk, we’ll be looking at building Beam pipelines to programmatically create Deep Learning models with Ludwig for different input data types for both model training and inference using Beam-Python SDK. We will be showing 2 examples of training deep learning classifiers with text and images on an unbounded source and running inference on that.

Part I:

Apache Arrow and Apache Beam deserve to be together. It is a stated goal of both projects to provide mechanisms for performing data processing tasks across many languages. To accomplish this, both projects define a common binary “model” and provide multiple language implementations of that model.
However, they are approaching the overarching problem of cross-language data processing from two very different angles: Arrow is primarily concerned with moving columnar data across language boundaries with minimal overhead and providing optimized computation primitives to operate on that data, while Beam seeks to enable users to write scalable data pipelines in their language of choice and execute them on any distributed data-processing system. There is significant overlap between these two problem spaces, and we should consider using these models together for the benefit of both projects.
In this talk, I will describe both of these projects at a high-level, and present a vision for how they can be used together. I will also discuss some of the concerns around using a batched columnar data format in the Beam model, along with potential solutions.

Part II:

In this session we will illustrate how Sky moved from an on-premise, batch-focused Hadoop based analytical system, to a streaming-oriented analytics pipeline deployed using Apache Beam and Google Cloud Dataflow. We will go through the architecture of the system, including the design decision made along the way, as well as learnings from the implementation and deployment process. We will end the session with an end-to-end demo and code examples!

We built a scalable and flexible stream data pipeline for our microservices on Google Cloud Platform (GCP), using Cloud Pub/Sub, Google Cloud Storage, BigQuery, and Cloud Dataflow, using Apache Beam. The stream data pipeline is working on the production system for Mercari, one of the biggest C2C e-commerce services in Japan. The pipeline currently accepts logs from 5+ microservices, and the number will increase soon.
Our microservice architecture is based on the following three concepts:
1. Split the log collection and data processing phases to keep the system simple.
2. Use stream processing in order to achieve low latency.
3. Don’t just accumulate raw data—support structured output that is easier to use.
Based on these concepts, we built a data pipeline in GCP.
For each microservice, we will provide a Cloud Pub/Sub “Ramp” to send logs to. Cloud Pub/Sub can have messages that contain an optional byte array in their payloads. The entire message that is ingested into the Ramp uses Cloud Dataflow streaming processing to collect them in the “RawDataHub,” a Cloud Pub/Sub topic for collection. When this happens, the PubsubMessage payload is not changed at all; the metadata necessary for subsequent processing (destination and schema information, data necessary for pipeline metrics, etc.) is provided in the PubsubMessage’s attribute map. This Dataflow job does not do processing for each service or topic—it treats all messages uniformly.
The raw data from RawDataHub is then output to two independent Cloud DataFlow streaming processes: “RawDataLake”(the infrastructure is in Google Cloud Storage, or “GCS”) and “StructuredDataHub”, another Cloud Pub/Sub topic. The StructuredDataHub has structured avro records.
The structured data from StructuredDataHub is then sent to more two independent Cloud DataFlow streaming processes: “StructuredDataLake” on GCS and “Data WareHouse” (Google BigQuery).