Stenobird

Episode

Data! Data! Data! With Francesco Tisiot

Podcast
Arrested DevOps
Published
Jun 1, 2023
Duration seconds
2793
Processing state
processed
Canonical source
https://www.arresteddevops.com/data-data-data/
Audio
https://media.blubrry.com/arresteddevops/content.blubrry.com/arresteddevops/arrested-devops-podcast-episode186.mp3
JSON
/v1/public/podcasts/arrested-devops/episodes/data-data-data-with-francesco-tisiot
Markdown
/podcast/arrested-devops/data-data-data-with-francesco-tisiot.md

Actions

  • POST https://stenobird.com/v1/public/podcasts/arrested-devops/episodes/data-data-data-with-francesco-tisiot/transcription-requests
    Idempotently request low-priority transcript generation for this episode.
  • GET https://stenobird.com/podcast/arrested-devops/data-data-data-with-francesco-tisiot.md
    Read the agent-friendly Markdown representation of this episode resource.

Summary

Uncontrolled data growth creates significant security risks and operational debt. This episode explores how to manage data lineage, prevent manual 'Excel-based' workarounds, and build resilient pipelines using technologies like Kafka.

Topics

  • Data Engineering
  • Data Lineage
  • Kafka
  • Data Governance
  • Data Pipelines
  • DevOps
  • Data Security
  • Scalability

Highlights

  • Failure mode: Allowing unmanaged data pipelines leads to 'shadow' processes and security vulnerabilities like exposed S3 buckets
  • Practical takeaway: Empowering users with proper tools prevents them from reverting to manual, insecure Excel exports
  • Main idea: Data lineage and visibility are more critical for scaling than simply implementing the latest shiny technology
  • Practical takeaway: Use technologies like Kafka to enable data replayability and schema versioning for more resilient systems
  • Main idea: The real challenge in modern data engineering isn't innovation, but the unglamorous work of data cleansing and management

Chapters

  1. 4:40 The Intersection of People and Data: Francesco discusses his transition from computer science to a role focused on the intersection of technology and human interaction through data.
  2. 8:10 The Data Journey: An exploration of how data points become valuable assets as they move through various stages of a company's lifecycle.
  3. 17:55 The Danger of Undocumented Pipelines: The risks of 'orphaned' data pipelines where the origin, ownership, and cost of data movement become unknown.
  4. 25:25 Preventing the 'Excel Way': How a lack of proper data infrastructure forces engineers into manual, insecure data handling processes.
  5. 29:25 Leveraging Kafka for Connectivity: Using Kafka Connect and JSON payloads to parse and link disparate data sources effectively.
  6. 36:20 Building Trustworthy Systems: Moving beyond speed to focus on error recovery, reliability, and building trust in data outputs.
  7. 40:00 The Power of Data Replay: How Kafka's log retention allows for replaying historical data to support new consumers and schema evolutions.