PlusOne

Ranking the Web with Spark – Sylvain Zimmer

Common Search is building an open source search engine based on Common Crawl’s monthly dumps of several billion webpages. Ranking every URL on the Web in a transparent and reproducible way is core to the project.

In this presentation, Sylvain Zimmer will explain why Spark is a great match for the job, how the current ranking pipeline works and what challenges it faces to grow in scale and complexity, in order to improve the quality of search results.

Specifically, we will dive in the new Spark 2.0 features that made it practical to compute PageRank from Python on every URL found in Common Crawl, and show how anyone can reproduce and tweak the results on their cloud servers.

More information about this talk

Low Latency Web Crawling on Apache Storm – Julien Nioche

StormCrawler is an open source collection of resources, mostly implemented in Java, for building low-latency, scalable web crawlers on Apache Storm. After a short introduction to Apache Storm and an overview of what StormCrawler provides, we will compare it with similar projects like Apache Nutch and present several real life use cases. In particular we will see how StormCrawler can be used with ElasticSearch and Kibana for crawling and indexing web pages and also monitor the crawl itself.

More information about this talk

Sparkler – Crawler on Apache Spark – Karanjeet Singh & Thamme Gowda

A web crawler is a bot program that fetches resources from the web for the sake of building applications like search engines, knowledge bases, etc. In this presentation, Karanjeet Singh and Thamme Gowda will describe a new crawler called Sparkler (contraction of Spark-Crawler) that makes use of recent advancements in distributed computing and information retrieval domains by conglomerating various Apache projects like Spark, Kafka, Lucene/Solr, Tika, and Felix. Sparkler is extensible, highly scalable, and high-performance web crawler that is an evolution of Apache Nutch and runs on Apache Spark Cluster. GitHub Link – https://github.com/USCDataScience/sparkler

More information about this talk

The Original Vision of Nutch, 14 Years Later: Building an Open Source Search Engine – Sylvain Zimmer

Few people remember that before spinning off Hadoop and focusing on crawling, Nutch was meant to be an alternative to commercial search engines. What if we tried to do it again today?

In this presentation, Sylvain Zimmer will explain how he used projects from the Nutch diaspora like Spark and Elasticsearch to build Common Search, an open source search engine with transparent rankings.

We will go over the architecture of large-scale search engines and how it has evolved since the late 90s. Then we will review the tools from the Apache and open source ecosystems that are best suited to solve the many challenges at hand. Finally, we will discuss what lies ahead for Common Search before it can be useful to the general public.

More information about this talk

Crawling the Web for Common Crawl – Sebastian Nagel

Common Crawl is non-profit organization which regularily crawls a significant sample of the web and makes the data accessible free charge to everyone interested in running machine-scale analysis on web data. The presentation will demonstrate how to use the Common Crawl data covering data formats and tools as well as examples and derived datasets. The monthly crawls are run by Apache Nutch on Apache Hadoop. Sebastian will also share his experience from running a web-scale crawl on a small budget.

More information about this talk

On the Representation and Reuse of Machine Learning Models – Villu Ruusmann

Big Data applications rely on machine learning to derive new value. Model training and deployment are handled by different people in different environments, which makes model transferability a major concern.

This talk inquires into popular R, Scikit-Learn and Apache Spark model types, and connects them at a standardized PMML representation level. PMML adds value to all stages of the workflow, starting from model interpretation, reorganization and persistence, and ending with fully-automated model deployment to schema-full Big Data frameworks.

Attendees will learn that models are not locked-in “black boxes”, but easily accessible and programmable components in the application layer. This realization should translate to improved workflows, and smarter and more performant applications.

More information about this talk

Performance Tuning Tips for Apache Spark Machine Learning Workloads – Shreeharsha GN & Amir Sanjar

Performance Tuning tips for Apache Spark Machine Learning workloads – OpenPOWER 8 architecture is the latest offering of IBM SoftLayer, is the perfect platform for evaluating and optimizing Apache Spark solutions. In under 60 minutes from receiving a Sotlayer welcome package to your new bare-metal Power8 server, you can have Hadoop and Spark, along with many other software applications, installed, configured, optimized, and ready to run Spark ML workload. In this talk we will cover: 1) Apache Spark overview 2) Apache Spark software deployment 3) Spark optimization on highly threaded server 4) Demo

More information about this talk

Machine Learning on Apache Apex with Apache Samoa – Bhupesh Chawda

This talk will be about the integration of Apache Samoa, a distributed streaming machine learning framework (https://samoa.incubator.apache.org) with Apache Apex, a distributed, scalable and fault-tolerant stream processing engine (https://apex.apache.org). Apache Samoa is a kind of WORA (write-once-run-anywhere) framework where algorithms developed on Samoa can be run on other distributed stream processing engines like Storm, Samza and Flink. This talk will introduce the integration story with Apache Apex and outline the process and the challenges therein. In addition, the talk will also dwell upon some comparative analysis on the performance of Samoa algorithms on few popular integrated runners, namely Apache Storm, Apache Flink and Apache Apex.

More information about this talk