PlusOne

At almost two years in the making Apache Cassandra 4.0 is almost here. With a focus on performance and stability, it is full of interesting features. This talk takes you through a tour of the new features and performance improvements. From Zero Copy Faster Streaming support to Virtual Tables and Audit Logging, learn how Apache Cassandra 4.0 has departed from previous major releases to be focused on the features required by the most demanding users. Attendees will not only gain an understanding of new features in Apache Cassandra, but understand what drove their creation and therefore have a better idea of how to best use these features in their own systems.

Apache Beam is a unified data processing framework, allowing you to write batch and streaming pipelines that run anywhere, including Apache Flink, Apache Spark, and Google Cloud Dataflow. With the SQL extension you can now write a pipeline in pure SQL. If you need more, you can write user defined functions in Java or even embed SQL into your existing Java pipeline. This talk will start with a demo pipeline written in pure SQL. We will review how streaming SQL came from collaboration between the Apache Beam, Apache Calcite and Apache Flink communities. Finally, we will deep-dive into the architecture of Beam’s implementation and the work we are doing to make Apache Beam SQL the default choice for writing new streaming pipelines.

One of the key components in any data processing system is IO connectors. These fundamental blocks allow to read and write data, which is stored in different type of sources, in a unified and distributed way. In this sense, Apache Beam is not an exception – it provides a rich API to develop a new connector with your favorite SDK and easily integrate it with Beam runners. In this talk we are going to show you how to write your own IO connector (in Java). We will see what are the differences between bounded and unbounded sources, how to implement efficient sources and sinks and where we need to pay more attention during the development, what kind of API your connector should provide to users and how it can be tested. To achieve this we will rely on examples from existing Beam connectors. We will also give a brief overview of a rather recent feature/pattern in Beam IO – Composable IO connectors and the Splittable DoFn API. We will discuss the advantages of modular IO API design and some new IO design patterns allowed by this style. This talk will be interesting for people, who consider to write their own IO connectors or want to contribute to existing ones, as well as for Beam users, who wish to understand how existing Beam connectors work under the hood.

Apache Beam provides a unified programming model to execute batch and streaming pipelines on all the popular big data engines. The translation layer from Beam to the chosen big data engine is called a runner. The current runner for Apache Spark is based on the RDD/DStream framework. However, there is an ongoing work to move it to Spark next generation framework a.k.a structured streaming. This talk will present why structured streaming is a good fit for Apache Beam, why it is worth the effort, and will give some feedback on how Apache Beam has solved the challenge, what the tough points and the sweet points were.

Apache Beam’s early vision was to create a framework agnostic of execution engines or languages. This vision was not fully realized from the early beginnings. At first, Beam was only agnostic of executions engines but very much tight to one language: Java.
In the recent years much effort has been undergone to implement a language portability layer, to finally realize the vision of portability.
In this talk we want to take a look at the state of portability, how it works, and what new use cases it unlocks.

In this talk we will discuss the infrastructure Apache Beam has built to create a positive contributor experience. Over the last few years we have developed tooling, refactored code, instrumented Jenkins builds, collected metrics, created a metrics dashboard, and involved the community in order to ensure happy contributors. We will discuss the importance of engineering productivity and tooling in developing strong communities, where we have succeeded, and where we still have work to do.

Apache Spark is the most popular open source analytics engine for large-scale data processing. Spark is not only a mature system, but thanks to its support of multiple resource managers like Hadoop, Mesos, and Kubernetes it has become a popular choice for both batch and streaming workloads in the industry. Apache Beam has included a Spark runner since its inception to allow users to execute Beam pipelines on Spark, but until recently the Spark runner could only execute pipelines written in Java. In this talk we will introduce the portability framework and how we adapted it into the existing Spark runner translation to make the Spark runner portable. We will show you how to execute Beam pipelines written in Python and Golang in Spark with Beam and invite you to use the new Spark Portable Runner. We will mention the use case of Tensorflow Extended, the end-to-end platform for data validation and transformation and ML model analysis. Finally we will discuss ongoing work and some future plans for the portable runner.

The data generated from IoT devices, machine logs and similar sources is often semi-structured or unstructured. This poses a challenge for traditional schema-on-write systems that require a fixed schema up-front for querying. Modern analytic applications often have ad hoc usage patterns and require tremendous flexibility over how this data can be consumed. Further, they demand that the data be made available for querying soon after it has landed in their data platform – which may be a distributed file system or NoSQL database or something similar. In this talk I will first describe how Apache Drill’s innovative schema-on-read capability built into a distributed SQL query engine meets the demands of such applications. The raw data in various formats can be queried directly from either the distributed file system or other data sources such as NoSQL DBs without having to define a schema up-front. The Drill query planner (in conjunction with Apache Calcite) supports the ‘ANY’ type for columns which allows type validations. During execution, readers produce ‘RecordBatches’ where all rows in one RecordBatch have the same schema but across batches the schema may change. This provides the core foundation for the schema-on-read capability. Downstream SQL operators in the query pipeline perform run-time Java code generation by generating schema specific code. This code is compiled and executed on the JVM and based on this the operator produces output batches. In cases where Drill is not able to infer the schema correctly or when there are ambiguities, Drill has introduced schema provisioning to complement the schema-on-read. I will describe the functionality to define columns types and nullability, specify columns format and default values and control which columns are projected and their projection order. The schema can be specified declaratively as part of the query or in a separate file. Our expectation is that these complementary strategies will meet the demands of the modern analytic applications that require schema flexibility in addition to performance and scalability.

This talk is about building bridges between two ecosystems, Apache and HDF5.nHDF5 is a widely used storage container for complex data from embedded devicesnto supercomputers, and is developed and maintained as FOSS by The HDF Group.nWhile it is easy to imagine the potential benefits of making HDF5 containersnaccessible from the various Apache frameworks, there are several technicalnchallenges to overcome, and what makes a ‘good’ integration is by no meansnobvious. It requires the input and collaboration of experts from both sides. The core of this presentation will be an overview of a joint project between The HDF Group and Apache Drill contributor Charles Givre. We will show how tonuse Apache Drill to explore collections of HDF5 containers and discuss thenunderlying design decisions and a few technicalities. We will use thisnopportunity to give the Apache community an update on the support forncollections of HDF5 files on HDFS, in cloud storage (such as S3), our Sparkndata source and our new ‘HDF5 in the cloud’ platform, HDF KITA. The intended audience are ‘bridge-builders’ and curious members of the Apachencommunity, and anyone interested in ‘HDF5 demystified.’

Good and predictable performance is a common expectation across customers of Elasticsearch, an unstructured, distributed database. Elasticsearch, however, is not built with performance diagnosis as a design tenet and this makes it hard for customers to troubleshoot performance problems or provision a cluster. Attend this talk to learn about Performance Analyzer, a feature we built on Open Distro for Elasticsearch, to enable real time performance diagnosis on your Elasticsearch clusters. We will cover the abstractions for observability and root cause analysis, and walk through the design choices we made when building Performance Analyzer on Elasticsearch and the JVM. We will also show a live demo of how Performance Analyzer can help you troubleshoot an issue on your Elasticsearch cluster.