{"podcast":{"title":"CyberCode Academy","slug":"cybercode-academy-7578615","podcast_index_feed_id":7578615,"rss_url":"https://www.spreaker.com/show/6790974/episodes/feed","website_url":"https://www.spreaker.com/podcast/cybercode-academy--6790974","image_url":"https://d3wo5wojvuv7l.cloudfront.net/t_rss_itunes_square_1400/images.spreaker.com/original/5f51ccc7b22fdba95149ffa6346f1533.jpg","author":"CyberCode Academy","episode_count":311,"summary":"Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity. 🎧 Each course is divided into a series of short, focused episodes that take you from beginner to advanced level — one lesson at a time. From Python and web development to ethical hacking and digital defense, our content transforms complex concepts into simple, engaging audio learning. Study anywhere, anytime — and level up your skills with CyberCode Academy. 🚀 Learn. Code. Secure. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy","last_synced_at":"2026-07-27T12:18:25.188608+00:00","page_url":"https://stenobird.com/podcast/cybercode-academy-7578615"},"episode":{"title":"Course 40 - Web Scraping with Python | Episode 14: Building and Automating Custom Spiders with the Scrapy Framework","slug":"course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework","published_at":"2026-07-24T06:00:03+00:00","page_url":"https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework","show_page_url":"https://stenobird.com/podcast/cybercode-academy-7578615","url":"https://www.spreaker.com/episode/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework--72756867","audio_url":"https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/72756867/automating_web_scraping_with_scrapy_spiders.mp3","summary":"In this lesson, you’ll learn about: Scrapy’s full architecture, how to build real spiders from scratch, and how to move from simple extraction to production-ready crawling with structured data pipelines1. Scrapy Architecture (How Everything Works)🔹 Core System FlowScrapy is built around a central engine that coordinates everything.🔹 Main ComponentsComponentRoleEngineControls flowSchedulerQueues URLsDownloaderFetches pagesSpiderExtracts dataPipelineProcesses & stores data👉 Key Insight You don’t control HTTP manually—Scrapy does it for you2. Project Setup & Spider Creation🔹 Initialize a Projectscrapy startproject myproject 🔹 Generate a Spiderscrapy genspider stocks yahoo.com 🔹 Project Structuremyproject/ ├── spiders/ ├── items.py ├── pipelines.py ├── settings.py 👉 Key Insight Each file has a strict responsibility → clean separation of logic3. Extracting Real Data (Yahoo Finance Example)🔹 Target Use CaseWe extract: Company name Stock price Market data 🔹 XPath in Spiderdef parse(self, response): yield { \"name\": response.xpath(\"//h1/text()\").get(), \"price\": response.xpath(\"//fin-streamer[@data-field='regularMarketPrice']/text()\").get() } 👉 Key Insight Spiders are just Python classes with extraction rules4. Running the Spider🔹 Execution Commandscrapy crawl stocks 🔹 Output Options Console print JSON export CSV export File writing 🔹 Save to Filescrapy crawl stocks -o data.json 👉 Key Insight Scrapy supports structured output without extra code5. Item Loaders (Cleaner Code)🔹 Why They MatterItem Loaders help: Clean data Normalize values Reduce repeated logic 🔹 Examplefrom scrapy.loader import ItemLoader loader = ItemLoader(item=StockItem(), response=response) loader.add_xpath(\"price\", \"//span/text()\") return loader.load_item() 👉 Key Insight You separate extraction from tr…","meta_description":"In this lesson, you’ll learn about: Scrapy’s full architecture, how to build real spiders from scratch, and how to move from simple extraction to producti…","key_points":[],"chapters":[],"topics":[],"duration_seconds":1327,"processing_state":"not_requested","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-14-building-and-automating-custom-spiders-with-the-scrapy-framework.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}