# Course 40 - Web Scraping with Python | Episode 14: Building and Automating Custom Spiders with the Scrapy Framework

Page: https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework
Text version: https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework.md
Podcast: [CyberCode Academy](https://stenobird.com/podcast/cybercode-academy-7578615)
Published: 2026-07-24T06:00:03+00:00
Episode link: https://www.spreaker.com/episode/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework--72756867
Audio file: https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/72756867/automating_web_scraping_with_scrapy_spiders.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework
Duration seconds: 1327

## Resource

In this lesson, you’ll learn about: Scrapy’s full architecture, how to build real spiders from scratch, and how to move from simple extraction to production-ready crawling with structured data pipelines1. Scrapy Architecture (How Everything Works)🔹 Core System FlowScrapy is built around a central engine that coordinates everything.🔹 Main ComponentsComponentRoleEngineControls flowSchedulerQueues URLsDownloaderFetches pagesSpiderExtracts dataPipelineProcesses & stores data👉 Key Insight You don’t control HTTP manually—Scrapy does it for you2. Project Setup & Spider Creation🔹 Initialize a Projectscrapy startproject myproject 🔹 Generate a Spiderscrapy genspider stocks yahoo.com 🔹 Project Structuremyproject/ ├── spiders/ ├── items.py ├── pipelines.py ├── settings.py 👉 Key Insight Each file has a strict responsibility → clean separation of logic3. Extracting Real Data (Yahoo Finance Example)🔹 Target Use CaseWe extract: Company name Stock price Market data 🔹 XPath in Spiderdef parse(self, response): yield { "name": response.xpath("//h1/text()").get(), "price": response.xpath("//fin-streamer[@data-field='regularMarketPrice']/text()").get() } 👉 Key Insight Spiders are just Python classes with extraction rules4. Running the Spider🔹 Execution Commandscrapy crawl stocks 🔹 Output Options Console print JSON export CSV export File writing 🔹 Save to Filescrapy crawl stocks -o data.json 👉 Key Insight Scrapy supports structured output without extra code5. Item Loaders (Cleaner Code)🔹 Why They MatterItem Loaders help: Clean data Normalize values Reduce repeated logic 🔹 Examplefrom scrapy.loader import ItemLoader loader = ItemLoader(item=StockItem(), response=response) loader.add_xpath("price", "//span/text()") return loader.load_item() 👉 Key Insight You separate extraction from tr…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.