An Introduction to Apache Spark on Kubernetes

Community adoption of Kubernetes (instead of YARN) as a scheduler for Apache Spark has been accelerating since the major improvements from Spark 3.0 release. Companies choose to run Spark on Kubernetes to use a single cloud-agnostic technology across their entire stack, and to benefit from improved isolation and resource sharing for concurrent workloads.

In this talk, Jean-Yves (“JY”), the CEO & co-founder of Data Mechanics, a serverless Spark platform powered by Kubernetes, will show how to easily get started with Spark on Kubernetes.

Presenter Bio:

Jean-Yves Stephan (“JY”) is the CEO and Co-Founder of Data Mechanics, a hassle-free containerized data platform that abstracts away the complexities of Spark and infrastructure management. Prior to that, he was a software engineer and Spark infrastructure team lead at Databricks, growing their cluster-management capabilities from early days to the scale of launching hundreds of thousands of nodes in the cloud every day. JY is passionate about making distributed data technologies 10x more accessible and resource-efficient through automation.