In this episode of Search Off the Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.
Resources:
Episode transcript → https://goo.gle/sotr070-transcript
Managing your crawl budget → https://goo.gle/3IzRZxl
Dave Smart on LinkedIn → https://goo.gle/3wPSuRA
Tame the Bots → https://goo.gle/4cfCQ1P
Search Central Help Forum → https://goo.gle/sc-forum
Indexing API docs → https://goo.gle/3v8yVU0
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast
Speaker: Gary Illyes, Lizzi Sassman
Products Mentioned: Search Off The Record
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations.
The Nikki Glaser Podcast
Every week comedian and infamous roaster Nikki Glaser provides a fun, fast-paced, and brutally honest look into current pop-culture and her own personal life.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.