apachebigdata | PlusOne

Apache Big Data Seville 2016 – Mining and Identifying Security Threat Using Spark SQL, HBase and Solr – Manidipa Mitra

December 5, 2016
The Apache Software Foundation

Mining and Identifying Security Threat Using Spark SQL, HBase and Solr – Manidipa Mitra

This presentation will talk about how to deisgn a highly effective scalable/performant distributed system to find the identity theft and fraud by mining billions of records related to share holding for a leading financial organization. This will also discuss on how Tera bytes of data can be migrated from Oracle to Hadoop, stored in parquet format, processed in a distributed computing framework with Spark DataFrame and pushed to different service layer (HBase, Impala, Solr, HDFS) depends on the query/access pattern. This design will also throw light on how the frequent transactions were handled and data were pre-processed end of the day to meet the seconds response time SLA, creating thousands of report by mining millions of record in minutes time.

More information about this talk

Apache Big Data Seville 2016 – On-Premise, UI-Driven Hadoop/Spark/Flink/Kafka/Zeppelin-as-a-service – Jim Dowling

December 5, 2016
The Apache Software Foundation

On-Premise, UI-Driven Hadoop/Spark/Flink/Kafka/Zeppelin-as-a-service – Jim Dowling

Since April 2016, SICS Swedish ICT has provided Hadoop/Spark/Flink/Kafka/Zeppelin-as-a-service to researchers in Sweden. We have developed a UI-driven multi-tenant platform (Apache v2 licensed) in which researchers securely develop and run their applications. Applications can be either deployed as jobs (batch or streaming) or written and run directly from Notebooks in Apache Zeppelin. All applications are run on YARN within a security framework built on project-based multi-tenancy. A project is simply a grouping of users and datasets. Datasets are first-class entities that can be securely shared between projects. Our platform also introduces a necessary condition for elasticity: pricing. Application execution time in YARN is metered and charged to projects, that also have HDFS quotas for disk usage. We also support project-specific Kafka topics that can also be securely shared.

More information about this talk

Apache Big Data Seville 2016 – SystemML – Declarative Machine Learning – Luciano Resende

December 3, 2016
The Apache Software Foundation

SystemML – Declarative Machine Learning – Luciano Resende

Machine learning in the enterprise is an iterative process. Data scientists will tweak or replace their learning algorithm in a small data sample until they find an approach that works for the business problem and then apply the Analytics to the full data set. Apache SystemML is a new system that accelerates this kind of exploratory algorithm development for large-scale machine learning problems.Think of SystemML as SQL for Machine Learning, it provides a high-level language to quickly implement and run algorithms, and it also enable cost-based optimizer that takes care of low-level decisions about parallelism, allowing users to focus on the algorithm and the real-world problem that the algorithm is trying to solve. This talk will introduce you to SystemML This talk will introduce you to SystemML and get you started building declarative analytics with SystemML using a Zeppelin notebooks.

More information about this talk

Apache Big Data Seville 2016 – Distributed In-Database Machine Learning with Apache MADlib (incubating) – Roman Shaposhnik

December 2, 2016
The Apache Software Foundation

Distributed In-Database Machine Learning with Apache MADlib (incubating) – Roman Shaposhnik

Data science is moving with gusto to the enterprise, where data often resides in relational databases with SQL as the main workload. So how can an enterprise add a data science dimension to their business without a major IT re-architecture?

Apache MADlib (incubating) is an innovative SQL-based open source library for scalable in-database analytics. It provides parallel implementations of mathematical, statistical and machine learning methods. Bringing machine learning computations to the data makes for excellent scale out performance on massively parallel processing (MPP) platforms like Greenplum database and Apache HAWQ (incubating).

In this talk, we will describe the origin of MADlib, review the architecture and common usage patterns, and look ahead to some interesting plans around performance acceleration.

More information about this talk

Apache Big Data Seville, 2016 – Machine Learning in Apache Zeppelin – Alexander Bezzubov

December 2, 2016
The Apache Software Foundation

Machine Learning in Apache Zeppelin – Alexander Bezzubov

There are many Machine Learning projects available: Apache Mahout, Apache SystemML (incubating), Apache Spark MLlib, Tensorflow, Scikit-learn, etc.

In this session we are going to showcase how typical ML predictive analytics workflow can benefit from modern notebooks-style interactive environment like Apache Zeppelin. We going to share examples of successful integration between different project in big-data ecosystem and touch up on various techniques like visual recognition, NLP, and Deep Learning.

More information about this talk

Apache Big Data: Keynote: Training Our Team in the Apache Way – Alan Gates

December 1, 2016
The Apache Software Foundation

Keynote: Training Our Team in the Apache Way – Alan Gates

Hortonworks contributes to a number of Apache projects. When we started we depended on our many experienced Apache community members to train their fellow Hortonworkers in the Apache Way. But we grew quickly, and we found this started to break down. So we have instituted training for our teams in what Apache is, how it works, their responsibilities as part of Apache and how that meshes with their responsibilities as Hortonworkers, and a practical list of dos and don’t. This talk will share some thoughts on the need for this training, give an overview of the content, and review some early results.

More information about this talk

Apache Big Data Seville – Keynote: Stream Processing as a Foundational Paradigm and Apache Flink’s Approach to It – Stephan Ewen

December 1, 2016
The Apache Software Foundation

Keynote: Stream Processing as a Foundational Paradigm and Apache Flink’s Approach to It – Stephan Ewen

Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays.

The fact that stream processing is gaining rapid adoption is also due to more powerful and maturing technology (much of it open source at the ASF) that has solved many of the hard technical challenges.

We discuss Apache Flink’s approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.

More information about this talk

Apache Big Data Seville 2016 – Open Source Operations: Building on Apache Spark with InsightEdge, TensorFlow, Apache Zeppelin, and/or Apache – Samuel Cozannet

November 29, 2016
The Apache Software Foundation

Open Source Operations: Building on Apache Spark with InsightEdge, TensorFlow, Apache Zeppelin, and/or Apache – Samuel Cozannet

As software becomes more free and open it also is becoming more complex and expensive to operate. How can we as an Open Source community distill best practices, and recommended operations to model complex interconnected services so users can focus on their ideas? How can we as developers deliver recommended best practices in our applications and when connected to other applications so users are free to contribute and use the project on their choice of substrate (laptop, cloud, or bare metal [x86, ARM, ppc64el, s390x]).

In this talk we explore how Juju can provide an Open Source method to model a multi-node Apache Spark cluster across a diverse set of substrates, and start adding other services to build additional solutions. This talk will include a demo, and users should be able to take all software shown to try for themselves in a free and Open Source manner.

More information about this talk

Apache Big Data Seville 2016 – Apache S2Graph (incubating) as a User Event Hub – Hyunsung Jo, Daewon Jeong & Hwansung Yu

November 29, 2016
The Apache Software Foundation

Apache S2Graph (incubating) as a User Event Hub – Hyunsung Jo, Daewon Jeong & Hwansung Yu

S2Graph is a graph database designed to handle transactional graph processing at scale.

Its API allows you to store, manage and query relational information using edge and vertex representations in a fully asynchronous and non-blocking manner.

However, at Kakao Corp., where the project was originally started, we believe that it could be so much more.

There have been efforts to utilize S2Graph as the centerpiece of Kakaoäó»s event delivery system taking advantage of its strengths such as

– flexibility of seamless bulk loading, AB testing, and stored procedure features,

– multitenancy that allows interoperability among different services within the company,

– and most of all, the ability to run various operations ranging from basic CRUD to multi-step graph traversal queries in realtime with large volumes.

More information about this talk