#43 WDTT - Data Discovery with Select Star and DQO.ai - data observability tool

Dear Members,

We are happy to invite you to the 43rd meetup of our group! We will have guests from DQO.ai and Select Star. The meeting will be held on May 5th, at 18:00.

Agenda:

18:00 - 18:15 Introduction (catering and networking)

18:15-19:00 DQO.ai an open-source data observability tool

Abstract

We are a team of data scientists who are working on Data Quality and Data Observability projects for years. We have delivered thousands of data quality tests that are detecting anomalies and predicting possible issues using AI.

We are building Data Quality scorecards to track the KPIs. We have monitored the data on all mayor cloud providers (AWS, GCP, Azure).

We are monitoring tables up to a petabyte scale on a daily basis. DQO.ai is an open-source data observability tool that handles data quality monitoring for you. It comes with predefined data checks – SQL queries, and rules – python functions, that evaluate the results.

You can modify existing checks and rules, or create the new ones, based on your needs. All you need to do is set a configuration in YAML file.

Speakers

Piotr Czarnas CEO, Founder - Experienced manager and software developer. Has already built and exited one startup in the Data Management industry (Lyftron Inc.). Previously a head of department (headcount: 120) responsible for Quality, Data Warehousing and Microsoft technologies at Roche Pharmaceuticals.

Radosław Nowak, Data engineer - In Dqo.ai for almost a year. He specializes in Big Data cloud tools (GCP environment in particular) and Python programming. Responsible for creating solutions that detect anomalies in data sets

19:00-19:05 Q&A Session

19:05 19:55 Data Discovery with Select Star

Once you have your data warehouse / data lake and ETL setup, the next step is for data scientists to run analyses and reports. However, most data scientists and business analysts end up spending too much time looking for the right data, often having to ask other people to find it.

In this talk, we’ll discuss what data discovery is, why it’s important, and how it can be used. We’ll use Select Star (https://selectstar.com) as an example of data discovery platform to demo what aspects of data discovery are impactful when there are hundreds and thousands of datasets.

Speaker

Shinji Kim - Founder & CEO - Shinji is the Founder & CEO of Select Star, an automated data discovery platform that helps you understand your data. Previously, she was the CEO of Concord Systems (concord.io), a NYC-based data infrastructure startup acquired by Akamai Technologies in 2016. She led building Akamai’s new IoT data platform for real-time messaging, log processing, and edge computing. Prior to Concord, Shinji was the first Product Manager hired at Yieldmo, where she led the Ad Format Lab, A/B testing, and yield optimization. Before Yieldmo, she was analyzing data and building enterprise applications at Deloitte Consulting, Facebook, Sun Microsystems, and Barclays Capital. Shinji studied Software Engineering at University of Waterloo and General Manage

19:55-20:00 Q&A Session