{"podcast":{"title":"Unsupervised Learning","slug":"unsupervised-learning","podcast_index_feed_id":779530,"rss_url":"https://www.omnycontent.com/d/playlist/070af456-729b-4a0f-9c09-a6c100397b59/3b159371-276d-429e-ae86-a6c1003b01c4/7b61d4e1-bd3d-4d3f-97c2-a6c1003b01c9/podcast.rss","website_url":"https://danielmiessler.com/podcast/","image_url":"https://www.omnycontent.com/d/playlist/070af456-729b-4a0f-9c09-a6c100397b59/3b159371-276d-429e-ae86-a6c1003b01c4/7b61d4e1-bd3d-4d3f-97c2-a6c1003b01c9/image.jpg?t=1622036215&size=Large","author":"Daniel Miessler","episode_count":541,"summary":"Unsupervised Learning is about ideas and trends in Cybersecurity, National Security, AI, Technology, and Culture—and how best to upgrade ourselves to be ready for what's coming.","last_synced_at":null,"page_url":"https://stenobird.com/podcast/unsupervised-learning"},"episode":{"title":"Using the Smartest AI to Rate Other AI","slug":"using-the-smartest-ai-to-rate-other-ai","published_at":"2025-04-19T06:00:00+00:00","page_url":"https://stenobird.com/podcast/unsupervised-learning/using-the-smartest-ai-to-rate-other-ai","show_page_url":"https://stenobird.com/podcast/unsupervised-learning","url":"https://omny.fm/shows/unsupervised-learning/using-the-smartest-ai-to-rate-other-ai","audio_url":"https://mgln.ai/e/p5837/pscrb.fm/rss/p/traffic.omny.fm/d/clips/070af456-729b-4a0f-9c09-a6c100397b59/3b159371-276d-429e-ae86-a6c1003b01c4/8a5bf7e7-84d2-4b9b-9080-b2c1013e0fd5/audio.mp3?utm_source=Podcast&in_playlist=7b61d4e1-bd3d-4d3f-97c2-a6c1003b01c9","summary":"In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels. I talk about: 1. Using One AI to Evaluate Another The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review. 2. A Human-Centric Grading System Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected. 3. Custom Prompts That Push for Deeper Evaluation The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance. Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler See you in the next one! Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.","meta_description":"In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smarte…","key_points":[],"chapters":[],"topics":[],"duration_seconds":575,"processing_state":"failed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/unsupervised-learning/episodes/using-the-smartest-ai-to-rate-other-ai/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/unsupervised-learning/using-the-smartest-ai-to-rate-other-ai.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}