apachebigdata | PlusOne

Apache Big Data Seville 2016 – Graph Processing with Apache Tinkerpop on Apache S2Graph – Doyung Yoon

November 29, 2016
The Apache Software Foundation

Graph Processing with Apache Tinkerpop on Apache S2Graph – Doyung Yoon

Since the last conference, Apache S2Graph community has been working on the integration with Apache Tinkerpop. Tinkerpop users are now able to use S2Graph as graph database without changing their Thinkerpop code, and also execute OLAP graph queries over their data in HDFS. We will share our experiences to integrate Thinkerpop as a graph database API, and comment on our current limitations and future plans. We will also present the benchmark results showing the comparison between S2Graph and existing graph databases such as Neo4j, Titan, and OrientDB. We focus our benchmarks on the “neighbors of neighbors” queries and the basic CRUD operations. Similar to Titan, S2Graph supports multiple storage backends, such as HBase, Cassandra, Mysql, Postgresql, and RocksDB, and the S2Graph’s performance for each backend will be presented.

More information about this talk

Apache Big Data Seville 2016 – How Big Data/IoT Leverage the Power of OpenSource to Solve Healthcare Use Cases – Manidipa Mitra

November 29, 2016
The Apache Software Foundation

How Big Data/IoT Leverage the Power of OpenSource to Solve Healthcare Use Cases – Manidipa Mitra

This session will talk about how a Digital Health Care Mgmt platform can be built (using different open source technologies like Kafka,Spark Streaming,HBase,Hive,pySpark,Mirth)to collect patient data,clinical data(HL7 data),claims data,real-time wearables data and create a 360 view/insights for a patient’s health risk and conditions. Also it will talk about how to built a generic platform(by scraping blogs,message board, articles, using an open source called Scrapy.Ingesting fb,twitter data,store,analyse,index,built social-sentiments,create word cloud,segment messages using open source like spark,HBase,Hive,python,Solr)to find out a Key Opinion Leader for a particular disease discussion in social media and how to provide insights/social-sentiments and search capabilities on different medicines used for particular disease/treatment to get feedback on medicines or for research purpose..

More information about this talk

Apache Big Data Seville 2016 – Fighting Identity Theft: Big Data Analytics to the Rescue – Seshika Fernando

November 29, 2016
The Apache Software Foundation

Fighting Identity Theft: Big Data Analytics to the Rescue – Seshika Fernando

Identity Theft is no longer just a consumeräó»s problem. Attackers are now targeting Enterprises for bigger financial gains and greater damage not just to the organizationäó»s infrastructure but more importantly to its corporate image.

While Enterprise Identity Theft Analytics Tools do exist, most organizations find it economically prohibitive to invest in expensive proprietary software. In this session, Seshika will show how a comprehensive Identity Theft Analytics Solution can be built using Open Source Technologies. She will demonstrate how Big Data Analytics can be used to safeguard any Enterprise by covering the 4 Aäó»s of Identity Analytics

More information about this talk

Apache Big Data Seville 2016 – Uber – Your Realtime Data Pipeline is Arriving Now!

November 29, 2016
The Apache Software Foundation

Uber – Your Realtime Data Pipeline is Arriving Now! – Ankur Bansal

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.

Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

More information about this talk

Apache Big Data Seville 2016 – ETL Pipelines with OODT, Solr and Stuff – Tom Barber

November 28, 2016
The Apache Software Foundation

ETL Pipelines with OODT, Solr and Stuff – Tom Barber

Discover a number of Apache projects you may not have heard of and how they can help you process both Clinical and non Clinical data. Apache OODT developed by NASA allows users to ingest and store files and metadata along with process workflows. OODT along with CTakes allows us to extract clinical information from files and then process them and allow end users access to the extracted data.

We can then take these sources and manipulate them further creating a highly flexible ETL pipeline offering reliability and scalability. Backed by Apache SOLR users can then interrogate the data via web interfaces and instigate further post processing and investigation.

Of course you may not have a clinical use case, but the platforms can be repurposed and will allow you to go away and build your own, scalable data pipeline for processing and integstion.

More information about this talk

Apache Big Data Seville 2016 – Large Scale SolrCloud Cluster Management via APIs – Anshum Gupta

November 28, 2016
The Apache Software Foundation

Large Scale SolrCloud Cluster Management via APIs – Anshum Gupta

Apache Solr is widely used by organizations to power their search platforms and often support multiple users. A lot of cluster management APIs were introduced over the last few releases, allowing the users to to manage operations ranging from replica placement to forcing leader elections via API calls. At the end of this talk, intermediate Solr users would understand what’s available, and when can they avoid direct interference with the system, leading to more stable clusters and lower chances of nodes going down. The attendees would also be much better equipped to build their own SolrCloud cluster management tools. I would also talk about when not to use these APIs and what’s planned in the near future to handle specific operational use cases.

More information about this talk

Apache Big Data Seville, 2016 – Fast & Scalable Email System with Apache Solr – Strategies, Tradeoffs and Optimizations – Arnon Yogev

November 28, 2016
The Apache Software Foundation

Fast & Scalable Email System with Apache Solr – Strategies, Tradeoffs and Optimizations – Arnon Yogev

Email interaction has its unique characteristics and is different than traditional web search (for example in that users search their own private mailboxes and are often interested in recent emails rather than the archive).

Taking advantage of these characteristics, we were able to optimize our infrastructure in terms of indexing strategy and query optimization and achieve a significant gain in scalability and performance.

Arnon will present the various tradeoffs that were explored, including multi-tiered indexes, sorted indexes, query optimizations and more.

Arnon will then present the benchmark results that stress the importance of correctly designing a Solr infrastructure and tailoring it to oneäó»s specific use case.

More information about this talk

Apache Big Data Seville 2016 – Managing Deeply Nested Documents in Apache Solr – Anshum Gupta

November 28, 2016
The Apache Software Foundation

Managing Deeply Nested Documents in Apache Solr – Anshum Gupta

Apache Solr in the recent past started supporting deeply-nested documents. Solr can now be used to perform search and faceting on documents such as nested email threads, comments and replies on social media, enriched and annotated documents etc. without having to flatten them before ingestion.

Anshum Gupta would discuss pre-processing of data so that it can be indexed in Solr, making it possible to perform complex search and statistical aggregation on top of it. He would also cover query formation for sample use cases of nested data and multiple options and features that Solr provides for faceting or aggregation of such documents.

By the end of this talk, Solr users would have a better understanding of both, how to work with features that Solr provides to find answers to interesting questions from deeply nested documents as well as work-arounds for the missing pieces.

More information about this talk

Apache Big Data EU 2016: Processing Planetary Sized Datasets – Tim Park

November 23, 2016
The Apache Software Foundation

Processing Planetary Sized Datasets – Tim Park

In my group at Microsoft, we have worked with the United Nations, Guide Dogs for the Blind in the UK, several automotive companies, and Strí_er on a number of projects involving high scale geospatial data.

In this talk, I’ll share some of the best practices and patterns that have come out of those experiences: best practices for storing and indexing geospatial data at scale, incremental ingestion and slice processing of the data, and efficiently building and presenting progressive levels of detail.

The audience will walk away with an understanding of how to efficiently summarize data over a geographic area, general methods for doing ingestion with Apache Kafka (or other event ingestion systems), and incremental updates to large scale datasets with Apache Spark, and best practices around visualizing this data on the frontend.

Speakers
Tim Park

More information about this talk

Apache Big Data EU 2016: Geospatial Track: Crowd Learning for Indoor Navigation – Thomas Burgess

November 23, 2016
The Apache Software Foundation

Geospatial Track: Crowd Learning for Indoor Navigation – Thomas Burgess

indoo.rs enables location based services for indoor applications. With indoo.rs, developers can add new features to their products, including having locations trigger events, track assets, showing closest routes to other places. For this, we use WiFi/beacon radio infrastructure, mobile devices and our cloud which produce lots of geospatial time series data. The real-time indoor navigation fuses independent movement from custom 9D sensor fusion and position estimates obtained by comparing current signal readings to a reference map. This talk will discuss how we create and maintain these maps in our big data machine learning system which leverages crowd data through Kafka and Spark to run SLAM and context aware algorithms to create high quality trajectories. In addition to use in reference maps, these trajectories provide an additional input for our interactive analytics.

Speakers
Thomas Burgess

More information about this talk