crawlingu

Crawlingu is a hypothetical domain‑specific language for scripting web crawling tasks. It is designed to express, in a readable and platform‑neutral form, which pages to visit, how to fetch them, how to extract data, and how to route results to storage or downstream processors. As a DSL, Crawlingu combines declarative rules with lightweight procedural actions and typically supports selectors modeled after CSS or XPath to identify data within HTML documents.

Its core concepts include seeds (starting URLs), scope and politeness settings (allowed domains, crawl depth, request rate limits), and crawl policies (breadth-first vs. depth-first, concurrency). Extraction rules specify targets by selectors and may apply transformations (trim, normalize, cast). The language also defines data sinks (JSON, CSV, databases) and basic error handling, retry strategies, and scheduling hooks for periodic crawls.

Syntax in Crawlingu is described as statements grouped into sections such as seeds, crawl, parse, and store. Typical constructs include seed statements, allow and deny rules, fetch and parse actions, extract blocks that bind data fields to output keys, and follow directives for links. Although variations exist in proposed designs, most implementations share the idea of a readable, transport-agnostic script that can be interpreted by web‑crawler engines.

Status: Crawlingu remains a conceptual construct used in discussions of DSLs for web crawling. There is no universal standard or widely adopted interpreter. In practice, many projects implement similar functionality using general-purpose languages, or with DSLs tailored to specific frameworks.

See also: web scraping, domain‑specific language, data extraction, CSS selectors, XPath, crawling politeness.