ML//agent//web agent

An agent whose tools include search, navigation, fetching or browser automation. A researcher who can use the Internet.

An agent whose tools include search, navigation, fetching or browser automation. A researcher who can use the Internet.

The difference from a crawler is the goal. A crawler wants to traverse; it follows routes by rules. An agent wants to obtain something; of forty links on a page it can say "this is the only one worth opening". The agent loop applies unchanged: goal, observe, reason, choose an action, observe the result, repeat (Agent Loop).

Its tools are the stages of crawling used on demand: a search engine for candidates, fetching for a page, browser automation when the content or the action needs a real browser, an API when one exists.

Crawler plus agent is a real combination: crawler behavior for coverage, agentic reasoning for selection. And a site can address it directly with an llms.txt, the clean version for readers that are models. Text hidden to steer it is spam.