PlusOne

The Resource Description Framework (RDF) is a standard model for storing graph data. While the standard was initially created for storing meta-data about the World Wide Web, its flexible format made it a popular choice for storing many different types of information. With the explosive increase in the size of available data, scalable solutions are needed to efficiently store and query very large RDF graphs within big data architectures. Apache Rya (incubating) is a scalable database management system designed for storing and searching very large RDF data. Rya is built on top of Apache Accumulo and also supports a MongoDB back end. nIn this talk, we introduce storage methods, primary and secondary indexing schemes, statistics based query optimization, as well as query evaluation techniques that allow Rya to scale to billions of triples across multiple nodes, while providing fast and easy access to the data through conventional query mechanisms such as SPARQL.

DESCRIPTION: Today’s Nordstrom customers expect order fulfillment options shipping to the most convenient locations in the timeframes that best fit their schedules. This session offers an inside look at how Nordstrom’s Supply Chain Technology team combined an extensive network of brick and mortar facilities with a custom software stack built atop JanusGraph to present fulfillment options tailored to each individual order. ABSTRACT: Over the last decade, the volume of online retail sales as a proportion of overall retail sales has nearly tripled, and there are no signs of this trend slowing down. In a $500 billion online retail market driven by ever more savvy online shoppers, expectations for order fulfillment have grown more demanding. Consumers expect a choice of fulfillment options that offer convenience, flexibility and value. Nordstrom’s Supply Chain Technology team is charged with meeting those customer expectations through efficient distribution of millions of unique items from a cross-country network of brick and mortar facilities. For the engineers on our team, it made perfect sense to use JanusGraph to model a dynamic inventory picture managed by a diverse set of geographically distributed facilities. Despite our enthusiasm, not everyone was eager to adopt an unfamiliar open source graph system. In this presentation, Jeff Callahan will discuss why his team at Nordstrom embraced JanusGraph as the backbone of a new data pipeline built to meet the evolving expectations of customers today (and tomorrow, too). After taking a detailed look at the system architecture, we’ll also discuss some of the internal organizational challenges we faced in adopting JanusGraph, and how we managed to navigate those challenges to ultimately deliver an even stronger solution. NOTE: While this presentation focuses on JanusGraph, there is considerable overlap with Big Data, Machine Learning and Cassandra

Data visualization is a challenging job to do in the Web-based environment due to the limited rendering efficiency of Web browsers. Rendering big data and providing user interaction in real time is even more challenging. nIn this speech, we are going to introduce how we made this possible with ECharts, a popular Web-based data visualization tool. Generally speaking, the solution lies in the following four aspects:n1. Simplifying shapes and rendering faster but with a similar result.n2. Optimizing code to improve rendering efficiency.n3. Applying stream rendering to give viewers a partial response as soon as possible.n4. Split data into several files and render in a pipeline to speed up rendering.

‘At LinkedIn, we generate a staggering 2 trillion events to Kafka each day. nThe ingested events are processed by our streaming platform powered by Apache Samza. nWe currently have over 3000 applications in production that leverage Samza at LinkedIn. nThe use-cases include the LinkedIn feed, anomaly detection, combating fraud, profile notifications, realtime analytics and many more. In this talk, we will describe our journey in building and scaling a distributed stream nprocessing system over these years. Drawing from our experience running 24×7 applicationsnat LinkedIn, we motivate the key-challenges common across them, how we naddressed them and the lessons learnt along the way. Specific challenges include – massive scale, accuracy of results, multi-language support, ndeveloper productivity, high-performance data-access and fast recovery. Lastly, we will nalso share strategic areas for our future work in stream processing.’

Apache Spark has become a popular engine for data analysis in industry, providing a handful SQL interface and processing data from various data sources. While many of our customers are using Apache Spark as an interactive query platform to meet their business requirement, one of the most common complaints we heard is that spark is not quite ‘interactive’ while data goes big. In traditional database systems, we can utilize materialized views to accelerate query processing, through pre-computation and query plan rewriting. We adopt a similar method to Spark, allow user to create flexible caches from a query or cube definition, and rewrite user queries at runtime to utilize pre-computed results. The cached data can persist to external data sources that Spark supports, or stay in memory, and updates automatically when new data ingested. nIn this talk, we’d like to take a deep dive into our design, and show real-world performance gain from our customers. This feature will be contributed to community.

IoT is widely spreading in various industries, and various sensor devices generate large amounts of data in real-time. Such big data is used for visualization, machine learning, and other data analysis. Kudu is one of the data stores for such use cases which require fast inserts and efficient scans. Apache HBase is also often used to handle time series data such as sensor data. We also evaluated HBase for smart meter data a few years ago. HBase is suitable for storing sensor data because it has excellent insert performance, but it is not suitable for big data analysis because scan performance is not so good. On the other hand, Kudu is suitable for both fast insert and big data scan. We evaluated the performance of Kudu for leveraging the power consumption data collected in every second from sensor devices. With this performance test, we confirmed that 4-node Kudu cluster can storing and referring data in real-time generated by 1 million sensor devices. In order to handle a large amount of data in real-time, it is important to increase the memory allocation, to reduce the tablet size, and to define the order of the primary key to reduce the load of compaction. This presentation will introduce the differences between Kudu and other data stores (HBase, HDFS), the use cases suitable for Kudu, and the tuning know-how obtained through performance test.

MaxCompute is a large scale, distributed big data platform of Alibaba, which provides Exabyte storage capacity and massive computing power through tens of thousands of commodity machines. The system supports SQL-like declarative language for advanced query and analysis on web-scale data set. Millions of jobs with hundreds of petabytes of data are processed every day, powering mission critical business within Alibaba, including e-Commerce, mobile payment, logistics, etc. 

Query optimizer plays a key part in determining the optimal execution plan. We first provide a general introduction about the overall architecture of MaxCompute, then we introduce how MaxCompute leverages Apache Calcite to build an efficient and robust query optimizer. We will also discuss the physical operators that MaxCompute creates in order to adapt to Apache Calcite, and the improvements that have been done to Calcite, such as IN list optimization, outer join null skew optimization, and logical relational node preprocessing. We further introduce our data partitioning techniques, including hash and range partitioning, to support advanced query parallelism. We will also talk about HBO (Historical Based Optimization) for regular ETL tasks.

One of the challenges at Meetup is how to build a scalable, reliable and efficient data platform to help our ML team builds models that recommend events fit your interests. With emerging sophisticated batch and streaming frameworks and cloud solutions, our data platform went through massive changes in the past two years. In this talk, I’ll discuss the evolution of how Meetup data platform utilizes Apache-based data systems, including Sqoop, Hive, Flume, Spark, Flink, Beam, Airflow. I’ll talk about architecture changes to our batch and stream pipeline solutions and what pros/cons to move data platform 100% to cloud. I’ll also share some lessons we learned and best practices on building distributed systems for data platform, and how the data platform collaborates with machines learning and data science team.

This talk describes how a food tech company with 50k merchants was at a critical situation by delivering events to devices by using events polling on a Postgres database, and how it could solve this problem by moving the events repository to Apache Ignite using distributed, in-memory SQL. The change also allowed the company to be on the right track to support more than 500k simultaneous merchants.

The AWS S3 protocol is the defacto interface for modern object stores. Ozone supports S3 protocol as the first-class notion in Ozone. For all practical purposes, a user of S3 can start using Ozone without any change to code or tools. S3 protocol support offered by Ozone is strongly consistent, so users don’t need to run sidekick tools like S3Guard when running big data applications like Apache Spark, Apache YARN or Apache Hive. This talk is a deep dive into how Ozone supports S3, and how easy it is to use the cloud tools against Ozone. We will demo S3 over Ozone with different off-the-shelf tools like Goofys, Kubernetes CSI, AWS CLI and also demonstrate how to use applications like Apache Spark, Apache Hive and YARN. We will also discuss some of the challenges faced during building S3 compatible rest server. For example, how we did the mapping of S3 semantics to ozone semantics, how we solved the S3 security issues and mapped them to ozone world. With S3 support Ozone has become a generic object store: it can be used with thousands of s3 compatible tools while it will also provide Hadoop compatible file system access.