# Long Context Language Models and their Biological Applications with Eric Nguyen - #690

Page: https://stenobird.com/podcast/twiml-ai-podcast/long-context-language-models-and-their-biological-applications-with-eric-nguyen-690
Text version: https://stenobird.com/podcast/twiml-ai-podcast/long-context-language-models-and-their-biological-applications-with-eric-nguyen-690.md
Podcast: [The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)](https://stenobird.com/podcast/twiml-ai-podcast)
Published: 2024-06-25T18:54:00+00:00
Episode link: https://twimlai.com/podcast/twimlai/long-context-language-models-and-their-biological-applications/
Audio file: https://pscrb.fm/rss/p/traffic.megaphone.fm/MLN4558723289.mp3?updated=1719342347
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/long-context-language-models-and-their-biological-applications-with-eric-nguyen-690
Duration seconds: 2741

## Resource

Today, we're joined by Eric Nguyen, PhD student at Stanford University. In our conversation, we explore his research on long context foundation models and their application to biology particularly Hyena, and its evolution into Hyena DNA and Evo models. We discuss Hyena, a convolutional-based language model developed to tackle the challenges posed by long context lengths in language modeling. We dig into the limitations of transformers in dealing with longer sequences, the motivation for using convolutional models over transformers, its model training and architecture, the role of FFT in computational optimizations, and model explainability in long-sequence convolutions. We also talked about Hyena DNA, a genomic foundation model pre-trained on 1 million tokens, designed to capture long-range dependencies in DNA sequences. Finally, Eric introduces Evo, a 7 billion parameter hybrid model integrating attention layers with Hyena DNA's convolutional framework. We cover generating and designing DNA with language models, hallucinations in DNA models, evaluation benchmarks, the trade-offs between state-of-the-art models, zero-shot versus a few-shot performance, and the exciting potential in areas like CRISPR-Cas gene editing. The complete show notes for this episode can be found at https://twimlai.com/go/690.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/long-context-language-models-and-their-biological-applications-with-eric-nguyen-690/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/twiml-ai-podcast/long-context-language-models-and-their-biological-applications-with-eric-nguyen-690.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.