# Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Page: https://stenobird.com/podcast/data-on-kubernetes-community/unsticking-ourselves-from-glue-migrating-payit-s-data-pipelines-to-argo-workflows-and-hera-dokc-town-hall
Text version: https://stenobird.com/podcast/data-on-kubernetes-community/unsticking-ourselves-from-glue-migrating-payit-s-data-pipelines-to-argo-workflows-and-hera-dokc-town-hall.md
Podcast: [Data on Kubernetes Community](https://stenobird.com/podcast/data-on-kubernetes-community)
Published: 2024-02-06T15:30:00+00:00
Episode link: https://podcasters.spotify.com/pod/show/dokcommunity/episodes/Unsticking-Ourselves-from-Glue-Migrating-PayIts-Data-Pipelines-to-Argo-Workflows-and-Hera--DoKC-Town-Hall-e2ept1e
Audio file: https://anchor.fm/s/2d649bc8/podcast/play/81637870/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2024-0-22%2F364509644-44100-2-0b7f77dfd9ed9.mp3
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/data-on-kubernetes-community/episodes/unsticking-ourselves-from-glue-migrating-payit-s-data-pipelines-to-argo-workflows-and-hera-dokc-town-hall
Duration seconds: 1397

## Resource

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera Presented by Matt Menzenski, Senior Software Engineering Manager, Payitgov At PayIt, we’ve been deploying applications to Kubernetes almost since the beginning of the company. Our data workloads, however, have run instead in AWS Glue. This has worked well enough for the reporting use cases that have been the main focus of this team historically. However, at the beginning of 2022, the PayIt data team began building out a new data platform, and in the process, ran into a number of challenges with Glue. In this talk, I will share the difficulties that we encountered with building, deploying, and orchestrating ETL pipelines in AWS Glue, our decision process for moving those workloads into Kubernetes, and the ELT architecture that we’ve arrived at today. Related Links DoKC Website - https://dok.community/ DoKC Meetups - https://www.meetup.com/data-on-kubernetes-community/ Join Slack - https://join.slack.com/t/dokcommunity/shared_invite/zt-1vgv7ymz7-YtLFvZicrcLP9fS3o_r2_w

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/data-on-kubernetes-community/episodes/unsticking-ourselves-from-glue-migrating-payit-s-data-pipelines-to-argo-workflows-and-hera-dokc-town-hall/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/data-on-kubernetes-community/unsticking-ourselves-from-glue-migrating-payit-s-data-pipelines-to-argo-workflows-and-hera-dokc-town-hall.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.