crawlmatic
Single library for static or dynamic website crawling needs.
A standard wrapper for HCCrawler & node-crawler, based on bluebird promises.
Install using npm:
npm i crawlmatic --save
Dynamic crawling example:
const DynamicCrawler } = ; // Initialize with HCCrawler optionsconst crawler = //dynamically evaluate page title evaluatePage: title: text ;//Setup - resolved when Chromium instance is upcrawler;
Static crawling example:
const StaticCrawler } = ;//Initialize with node-crawler optionsconst staticCrawler = maxConnections: 10 retries: 3; //setup internal node-crawler instance and resolves promisestaticCrawler;