ML//agent//web agent

A web agent is an agent whose tools include search, navigation, fetching or browser automation: a researcher who can use the Internet.

A web agent is an agent whose tools include search, navigation, fetching or browser automation: a researcher who can use the Internet.

The difference from a crawler is the goal. A crawler wants to traverse, and it follows routes by rules; an agent wants to obtain something, and of forty links on a page it can pick the only one worth opening. The agent loop applies unchanged:

1Set the goal2Observe3Reason4Choose an action5Observe the result

Its tools are the stages of crawling used on demand, with the opposite selection policy (same fetch, same parse, but the crawler follows every road because it has no opinion about destinations): a search engine for candidates, fetching for a page, browser automation when the content or the action needs a real browser, an API when one exists.

Crawler plus agent is a real combination: crawler behavior for coverage, agentic reasoning for selection. A site can also address it directly with an llms.txt, the clean version for readers that are models. Text hidden to steer it is spam.