# Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors

Page: https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors
Text version: https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors.md
Podcast: [CyberCode Academy](https://stenobird.com/podcast/cybercode-academy-7578615)
Published: 2026-07-23T06:00:03+00:00
Episode link: https://www.spreaker.com/episode/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors--72756861
Audio file: https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/72756861/prototyping_data_extraction_in_scrapy_shell.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors
Duration seconds: 1239

## Resource

In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can: Test selectors before writing spiders Inspect HTML responses instantly Experiment with scraping logic 🔹 Key Objects Inside Shell response → HTML content of the page request → HTTP request details spider → scraper context 👉 Key Insight You can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports: 🌐 Live websites 📄 Local HTML files 👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key Insight CSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell: Works on cached HTML Won’t reflect live changes unless restarted 👉 Key Insight Always restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/p Starts from root Very strict 🔹 Relative Path//div/p Searches anywhere More flexible 7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key Insight XPath is powerful for uncertai…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.