Simplifying management of Spark jobs in the cloud with Cloudera Data Engineering

Spark has become the framework of choice for developing ETL jobs at many enterprises. It’s not uncommon to find 100s of spark jobs across several Cloudera cluster deployments. In talking to customers, specifically Spark developers about their workflow of Spark jobs from development to deployment to debugging we felt that we could greatly simplify the user experience for both Spark users and IT administrators. This is why we released a new experience in CDP Public Cloud called Cloudera Data Engineering which is delivering a new level of Spark jobs development based on pure K8s and packed with the latest Spark & Airflow.

Working on Cloudera in the cloud brings such benefits as better infrastructure efficiency, cost management, and workload performance and especially in a Hybrid infrastructure - much faster onboarding without disrupting existing workloads, and with different needs and requirements.