PlusOne

Telling the ASF's Stories

Apache Big Data Seville 2016 – Integrators at Work! Real-Life Applications of Apache Big Data Components – Moderated by Phil Archer

December 20, 2016
The Apache Software Foundation

Integrators at Work! Real-Life Applications of Apache Big Data Components – Moderated by Phil Archer

The event will offer both insights and a hands-on opportunity to learn about and try out the Big Data Platform devised by the BigDataEurope (BDE) project. It’s most striking feature is the ease with which many Apache big data components like Apache Spark, Flink, Kafka, Cassandra and Solr can be instantiated through a simple UI thanks to the project’s use of Docker containers and Docker Swarm.

BDE is producing 7 pilot instances aligned with the 7 H2020 Societal Challenges (SC), each of which targets a real-world use-case. Things like handling the 2GB of data produced each day by a single typical wind turbine; data mining academic journals and matching the named entities with further information about them including images; tracking changes in land use and matching them with social and professional media feeds. Many of these use cases depend on another key feature of the BigDataEurope platform – the semantification of big data.

Participants will also have the opportunity to shape the next stage of the BDE platform, based on their unique skills and experiences with the Apache technology.

More information about this talk

Apache Big Data Seville 2016 – Scalable Data Science in R and Apache Spark 2.0 – Felix Cheung

December 20, 2016
The Apache Software Foundation

Scalable Data Science in R and Apache Spark 2.0 – Felix Cheung

R is a very popular platform for Data Science. Apache Spark is a highly scalable data platform. How could we have the best of both worlds? In this talk we will walkthrough many examples how several new features in Apache Spark 2.0.0 will enable this. We will also look at exciting changes coming next in Apache Spark 2.0.1 and 2.1.0.

More information about this talk

Apache Big Data Seville 2016 – Building Apache Spark Application Pipelines for the Kubernetes Ecosystem – Michael McCune

December 20, 2016
The Apache Software Foundation

Building Apache Spark Application Pipelines for the Kubernetes Ecosystem – Michael McCune,

Apache Spark based applications are often comprised of many separate, interconnected components that are a good match for an orchestrated containerized platform like Kubernetes. But with the increased flexibility afforded by these technologies comes a new set challenges for building rich data-centric applications.

In this presentation we will discuss techniques for building multi-component Apache Spark based applications that can be easily deployed and managed on a Kubernetes infrastructure. Building on experiences learned while developing and deploying cloud native applications on an OpenShift platform, we will explore common issues that arise during the engineering process and demonstrate workflows for easing the maintenance factors associated with complex installations.

More information about this talk

Apache Big Data Seville 2016 – Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

December 20, 2016
The Apache Software Foundation

Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

There are many resources available for using Apache Spark to build collaborative filtering models. However, there are relatively few for how to build a large-scale, end-to-end recommender system.

This talk will show how to create such a system, using Apache Kafka, Spark Streaming and Elasticsearch for data ingestion, real-time analytics and data storage, Spark DataFrames and ML pipelines for data aggregation and model building, and Elasticsearch for model management, serving and data visualization. We will also explore techniques for scaling model serving, using Spark Streaming for real-time model updates, and how to incorporate state-of-the-art models into this framework.

The talk will be technical and developer-focused, highlighting experiences from building real-world recommender systems, and providing example code (which will be available as open source).

More information about this talk

Apache Big Data Seville 2016 – Putting A Spark in Web Apps – David Fallside

December 20, 2016
The Apache Software Foundation

Putting A Spark in Web Apps – David Fallside

Web app developers want to take advantage of the sophisticated analytics and big data processing provided by engines such as Apache Spark. Traditionally, web app development would use an enterprise language like Java but with a development emphasis now on agility and simplicity, technologies such as Node.js, Ruby on Rails, and PHP are increasingly being used for such development. Apache Spark has APIs for Scala, Java and Python but no API for Node.js or JavaScript despite their importance for web app development. To fill this gap, the EclairJS open-source project was created to provide an API in Node.js and JavaScript and enable web app developers to incorporate the analytic and other capabilities of Spark. In this presentation, David Fallside will show some web applications that demonstrate Sparkäó»s capabilities and explain how they are implemented using EclairJS.

More information about this talk

Apache Big Data Seville 2016 – Distributed Logistic Model Trees – Mateo Alvarez & Antonio Soriano

December 9, 2016
The Apache Software Foundation

Distributed Logistic Model Trees – Mateo Alvarez & Antonio Soriano

Classification algorithms play an important role in different business areas, such as fraud detection, cross selling or customer behavior. In the business context, interpretability is a very desirable property, sometimes even a hard requirement. However, interpretable algorithms are usually outperformed by other non-interpretable algorithms such as Random Forest. In this talk Antonio Soriano will present a distributed implementation in Spark of the Logistic Model Tree (LMT) algorithm (Landwehr, et al. (2005). Machine Learning, 59(1-2), 161-205.), which consists of a decision tree with logistic classifiers in the leafs. While being highly interpretable, the LMT consistently performs equal or better than other popular algorithms in several performance metrics such as accuracy, precision/recall or area under the ROC curve.

More information about this talk

Apache Big Data Seville 2016 – How Software Engineering Has Changed with Advent of OSS – Nupur Sharma

December 9, 2016
The Apache Software Foundation

How Software Engineering Has Changed with Advent of OSS – Nupur Sharma

The talk shall explore the business of open source and how open source has changed the way software engineering is done and executed. Earlier, every software process was done as a ballpark project and designed with commercial, non-extensible products in mind.With the new open source paradigm, companies are now driving software development with open source products as the core and leveraging the extensibility of the product itself. In the talk, Nupur shall drive through the thought process of product designers through the 1990s, 2000s and now. Nupur shall explain how organisations are adapting Open Source Software and building their entire business models around them. Driving through some use cases, the transition from closed source to open source in many existing and well thought processes shall be discussed and explored. This shall enlighten any org exploring to move to OSS paradigm.

More information about this talk

Apache Big Data Seville 2016 – SQL and Streaming Systems – Atri Sharma

December 9, 2016
The Apache Software Foundation

SQL and Streaming Systems – Atri Sharma

The talk shall focus on to design and build systems for stream based data and exploiting the power of SQL and relational algebra on Streaming data using Apache Apex and Apache Calcite.

More information about this talk

Apache Big Data Seville 2016 – Apache Ignite – JCache and Beyond – Dmitriy Setrakyan

December 9, 2016
The Apache Software Foundation

Apache Ignite – JCache and Beyond – Dmitriy Setrakyan

This presentation will provide a good overview of Apache Ignite project including a detailed look into distributed in-memory Data Grid, Compute Grid, Streaming, in memory SQL, and many other components provided by Apache Ignite. We will also go into detail of how existing in-memory caching products and data grids can be used to share memory across Apache Spark jobs and applications. We will also present a hands on demo demonstrating performance benefits of querying shared memory using SQL.

More information about this talk

Apache Big Data Seville 2016 – Smart Storage Management: Towards Higher HDFS Storage Efficiency – Wei Zhou

December 9, 2016
The Apache Software Foundation

Smart Storage Management: Towards Higher HDFS Storage Efficiency – Wei Zhou

All kinds of data volume increases dramatically in recent years, new storage devices (NVMe SSD, flash SSD, etc.) can be utilized to improve data access performance. HDFS provides methodologies like HDFS Cache, Heterogeneous Storage Management (HSM) and Erasure Coding (EC) to provide such support, but it remains a big challenge to define and adjust different storage strategies for different data in a dynamic environment.

To overcome the challenge and improve the storage efficiency of HDFS, we will introduce a comprehensive solution, aka Smart Storage Management (SSM) in Apache Hadoop. HDFS operation data and system state information are collected from the cluster, based on the metrics collected SSM can extract some äóìdata access patternsäó and based on these patterns SSM will automatically make sophisticated usage of these methodologies to optimize HDFS storage efficiency.

More information about this talk

Powered by WordPress.com.