Episode
Course 40 - Web Scraping with Python | Episode 14: Building and Automating Custom Spiders with the Scrapy Framework
- Podcast
- CyberCode Academy
- Published
- Jul 24, 2026
- Duration seconds
- 1327
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
In this lesson, you’ll learn about: Scrapy’s full architecture, how to build real spiders from scratch, and how to move from simple extraction to production-ready crawling with structured data pipelines1. Scrapy Architecture (How Everything Works)🔹 Core System FlowScrapy is built around a central engine that coordinates everything.🔹 Main ComponentsComponentRoleEngineControls flowSchedulerQueues URLsDownloaderFetches pagesSpiderExtracts dataPipelineProcesses & stores data👉 Key Insight You don’t control HTTP manually—Scrapy does it for you2. Project Setup & Spider Creation🔹 Initialize a Projectscrapy startproject myproject 🔹 Generate a Spiderscrapy genspider stocks yahoo.com 🔹 Project Structuremyproject/ ├── spiders/ ├── items.py ├── pipelines.py ├── settings.py 👉 Key Insight Each file has a strict responsibility → clean separation of logic3. Extracting Real Data (Yahoo Finance Example)🔹 Target Use CaseWe extract: Company name Stock price Market data 🔹 XPath in Spiderdef parse(self, response): yield { "name": response.xpath("//h1/text()").get(), "price": response.xpath("//fin-streamer[@data-field='regularMarketPrice']/text()").get() } 👉 Key Insight Spiders are just Python classes with extraction rules4. Running the Spider🔹 Execution Commandscrapy crawl stocks 🔹 Output Options Console print JSON export CSV export File writing 🔹 Save to Filescrapy crawl stocks -o data.json 👉 Key Insight Scrapy supports structured output without extra code5. Item Loaders (Cleaner Code)🔹 Why They MatterItem Loaders help: Clean data Normalize values Reduce repeated logic 🔹 Examplefrom scrapy.loader import ItemLoader loader = ItemLoader(item=StockItem(), response=response) loader.add_xpath("price", "//span/text()") return loader.load_item() 👉 Key Insight You separate extraction from tr…