Colly

Colly is a web scraping framework for the Go programming language. It provides a fast, extensible toolkit for collecting data from websites and programmatically navigating the web.

The core concept is the Collector, an object that manages HTTP requests and responses. Developers register event callbacks such as OnRequest, OnResponse, OnHTML, and OnError. OnHTML receives a CSS selector to extract data from the parsed document; the library uses a GoQuery-like selector syntax to facilitate data extraction.

Colly supports asynchronous crawling and configurable concurrency through LimitRule objects and the Async flag. It includes a built-in request queue to manage large crawls and a storage interface for persistent state or deduplication across runs. It also handles cookies, user agents, and can be configured to respect robots.txt when enabled.

The framework is modular and extensible, with optional extensions for features such as random user agents, referer headers, caching, and proxy support. Colly can interact with various storage backends and can be used for tasks ranging from small crawls to large-scale data extraction and scraping projects.

Colly is open source under the MIT license and is maintained on GitHub by the community. It is widely used in data gathering projects across industries such as market research, competitive intelligence, and content aggregation.