Any scraper is also a “user agent doing work for users”. Which ones should respe... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		what 10 months ago \| parent \| context \| favorite \| on: Show HN: Browser MCP – Automate your browser using... Any scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?

randunel 10 months ago [–]

Does the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact