{"podcast":{"title":"CyberCode Academy","slug":"cybercode-academy-7578615","podcast_index_feed_id":7578615,"rss_url":"https://www.spreaker.com/show/6790974/episodes/feed","website_url":"https://www.spreaker.com/podcast/cybercode-academy--6790974","image_url":"https://d3wo5wojvuv7l.cloudfront.net/t_rss_itunes_square_1400/images.spreaker.com/original/5f51ccc7b22fdba95149ffa6346f1533.jpg","author":"CyberCode Academy","episode_count":311,"summary":"Welcome to CyberCode Academy — your audio classroom for Programming and Cybersecurity. 🎧 Each course is divided into a series of short, focused episodes that take you from beginner to advanced level — one lesson at a time. From Python and web development to ethical hacking and digital defense, our content transforms complex concepts into simple, engaging audio learning. Study anywhere, anytime — and level up your skills with CyberCode Academy. 🚀 Learn. Code. Secure. You can listen and download our episodes for free on more than 10 different platforms: https://linktr.ee/cybercode_academy","last_synced_at":"2026-07-27T12:18:25.188608+00:00","page_url":"https://stenobird.com/podcast/cybercode-academy-7578615"},"episode":{"title":"Course 40 - Web Scraping with Python | Episode 6: From Scrapy Framework Foundations to Professional Spiders","slug":"course-40-web-scraping-with-python-episode-6-from-scrapy-framework-foundations-to-professional-spiders","published_at":"2026-07-16T06:00:02+00:00","page_url":"https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-6-from-scrapy-framework-foundations-to-professional-spiders","show_page_url":"https://stenobird.com/podcast/cybercode-academy-7578615","url":"https://www.spreaker.com/episode/course-40-web-scraping-with-python-episode-6-from-scrapy-framework-foundations-to-professional-spiders--72756762","audio_url":"https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/72756762/scale_web_scraping_with_python_scrapy.mp3","summary":"In this lesson, you’ll learn about: building scalable scraping systems with Scrapy, mastering selectors in real time, and designing efficient, production-ready spiders1. What is Scrapy (and Why It Matters)?🔹 The Framework ApproachUse Scrapy Not just a library → a full scraping engine Handles: Requests scheduling Data pipelines Middleware Concurrency 👉 Key Insight Scrapy follows the Hollywood Principle:“Don’t call us, we’ll call you” You define rules → Scrapy controls execution2. Project Setup with Scrapy CLI🔹 Initialize a Projectscrapy startproject myproject cd myproject scrapy genspider example example.com 🔹 Project Structure Overview spiders/ → your scraping logic items.py → data models pipelines.py → cleaning & storage settings.py → configuration 👉 Clean structure = scalable scraping system3. Mastering the Scrapy Shell🔹 Interactive Testing Toolscrapy shell \"https://example.com\" 🔹 Why It’s Powerful Test CSS selectors instantly Test XPath queries in real time Debug without running full spiders 🔹 Handling 403 Forbidden ErrorsWebsites may block bots → fix using User-Agentscrapy shell -s USER_AGENT=\"Mozilla/5.0\" \"https://example.com\" 👉 Key Insight Many blocks are superficial → mimic real browser behavior4. Building a Professional Spider🔹 Basic Spider Structureimport scrapy class ExampleSpider(scrapy.Spider): name = \"example\" def start_requests(self): urls = [\"https://example.com\"] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): for item in response.css(\"div.item\"): yield { \"title\": item.css(\"h2::text\").get(), \"link\": item.css(\"a::attr(href)\").get() } 🔹 Key Concepts1. Inheritance Spider inherits from scrapy.Spider Gains built-in crawling behavior 2. start_requests Entry point of the spider Sends initial HTTP requests 3. p…","meta_description":"In this lesson, you’ll learn about: building scalable scraping systems with Scrapy, mastering selectors in real time, and designing efficient, production-…","key_points":[],"chapters":[],"topics":[],"duration_seconds":1431,"processing_state":"not_requested","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/cybercode-academy-7578615/episodes/course-40-web-scraping-with-python-episode-6-from-scrapy-framework-foundations-to-professional-spiders/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/cybercode-academy-7578615/course-40-web-scraping-with-python-episode-6-from-scrapy-framework-foundations-to-professional-spiders.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}