Episode
Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors
- Podcast
- CyberCode Academy
- Published
- Jul 23, 2026
- Duration seconds
- 1239
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-13-mastering-scrapy-shell-css-and-xpath-selectors.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can: Test selectors before writing spiders Inspect HTML responses instantly Experiment with scraping logic 🔹 Key Objects Inside Shell response → HTML content of the page request → HTTP request details spider → scraper context 👉 Key Insight You can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports: 🌐 Live websites 📄 Local HTML files 👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key Insight CSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell: Works on cached HTML Won’t reflect live changes unless restarted 👉 Key Insight Always restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/p Starts from root Very strict 🔹 Relative Path//div/p Searches anywhere More flexible 7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key Insight XPath is powerful for uncertai…