Then, maybe the "crawler" should be an actual PC navigating to the browser, taki... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		XCSme on July 25, 2023 \| parent \| context \| favorite \| on: The Fall of Stack Overflow Then, maybe the "crawler" should be an actual PC navigating to the browser, taking a screenshot (or live feed) of the page and processing it with AI.

pjc50 on July 25, 2023 [–]

Eh, Google choose to be identifiable as googlebot and to obey robots.txt for other reasons of "good citizenship", because not everybody wants to be crawled.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact