Stenobird

Episode

#236: Building ML Products at Compare the Market, with Conor O'Neill, the Head of Data Science at Compare The Market

Podcast
Data Futurology - Leadership And Strategy in Artificial Intelligence, Machine Learning, Data Science
Published
Jun 13, 2023
Duration seconds
3154
Processing state
processed
Canonical source
https://podcasters.spotify.com/pod/show/datafuturology/episodes/236-Building-ML-Products-at-Compare-the-Market--with-Conor-ONeill--the-Head-of-Data-Science-at-Compare-The-Market-e25mibr
Audio
https://anchor.fm/s/3fab060/podcast/play/72091451/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2023-5-15%2Ff4effd8c-3e93-9db0-cdf7-29ffb06dbc50.mp3
JSON
/v1/public/podcasts/data-futurology-leadership-and-strategy/episodes/236-building-ml-products-at-compare-the-market-with-conor-o-neill-the-head-of-data-science-at-compare-the-market
Markdown
/podcast/data-futurology-leadership-and-strategy/236-building-ml-products-at-compare-the-market-with-conor-o-neill-the-head-of-data-science-at-compare-the-market.md

Actions

  • POST https://stenobird.com/v1/public/podcasts/data-futurology-leadership-and-strategy/episodes/236-building-ml-products-at-compare-the-market-with-conor-o-neill-the-head-of-data-science-at-compare-the-market/transcription-requests
    Idempotently request low-priority transcript generation for this episode.
  • GET https://stenobird.com/podcast/data-futurology-leadership-and-strategy/236-building-ml-products-at-compare-the-market-with-conor-o-neill-the-head-of-data-science-at-compare-the-market.md
    Read the agent-friendly Markdown representation of this episode resource.

Summary

Conor O'Neill explains how to transition from reactive data science to a product-centric machine learning approach. He details the structural changes required to align data engineering, architecture, and science within a large organization.

Topics

  • Machine Learning Product Management
  • Data Transformation
  • Data Science Leadership
  • MLOps
  • Predictive Analytics
  • Data Engineering
  • Generative AI
  • Solution Architecture

Highlights

  • Main idea: Treating machine learning models as internal products via APIs ensures better consumption and scalability across the organization
  • Practical takeaway: Involve data scientists in the early stages of solution design and architecture to prevent downstream delivery bottlenecks
  • Failure mode: Relying on siloed data science work without foundational data engineering and unified documentation leads to unscalable results
  • Strategic insight: Transitioning from a practitioner to a leader requires shifting focus from model accuracy to business value and reusable infrastructure
  • Future trend: Generative AI may shift the data scientist's role toward 'data science as a service,' focusing on stewardship and integration rather than just model building

Chapters

  1. 1:00 Coordinating Data Science and Engineering: The challenges of synchronizing data scientists with API development and data ingestion pipelines.
  2. 5:00 From Astrophysics to Data Science: Conor's career pivot and his experience building the early data science function at Flight Centre.
  3. 8:50 Building Data Capability: The journey of establishing a dedicated data science function within an established organization.
  4. 12:50 Predictive Use Cases: Using machine learning to predict customer purchase behavior and shorten the sales cycle.
  5. 16:50 Dimensionality Reduction in Practice: How complex customer inputs are distilled into actionable scores for business decision-making.
  6. 20:40 Modernizing the Data Stack: Leveraging Databricks to improve modeling, documentation, and cohesive data modeling.
  7. 24:40 Managing Transformation Dependencies: The complexities of coordinating new API development with downstream machine learning needs.