87. WrocławJUG

Taking samples from big data: comparing map and reduce operations with Java Stream API and Spark Core API

In this talk we will start from a simple code snippet based on Java Stream API to discover and learn about the challenges of working in a distributed environment. Apache Spark is a framework part of the Hadoop ecosystem that is used to perform distributed computation on data. We’ll see that even though our simple Stream-based code looks very similar to what we could get with Scala collections or Spark Core API, code distribution will put many hurdles and constraints on our way. By discovering those issues one after the other, we’ll learn about Spark Core architecture and how data can be processed in parallel in a highly distributed environment.

Difficulty level: Easy/ Mid

Speaker: Maxime Jeanmart

Maxime already started his career in 1999 as a Java specialist for the telecom equipment companyAlcatel-Lucent, handling the data management part of a call handling platform. In 2008 he became a freelance software architect, still mostly focussed on the java ecosystem. He started to take part in different machine learning and big data projects with the rise of popularity of data science. He finally joined Collibra in 2016 to collaborate in the inception of the Collibra Catalog, sharing his time between good old back-end code and data wrangling in Spark.

We are enterprise software company on a mission: to help people find, understand, and trust their data. Because when that happens, opportunities to uncover hidden insight, gain competitive edge, drive operational efficiency, and transform the business flourish.

The doors will be open from 18:00. Sponsor will provide soft drinks, beers and snacks.