UtilitesΒΆ

These documents provide information about the agnostic utility classes used within Scrapy Cluster that may be useful in other applications.

Argparse Helper
Simple module to assist in argument parsing with subparsers.
Log Factory
Module for logging multithreaded or concurrent processes to files, stdout, and/or json.
Method Timer
A method decorator to timeout function calls.
Redis Queue
A module for creating easy redis based FIFO, Stack, and Priority Queues.
Redis Throttled Queue
A wrapper around the Redis Queue module to enable distributed throttled pops from the queue.
Settings Wrapper
Easy to use module to load both default and local settings for your python application and provides a dictionary object in return.
Stats Collector
Module for statistics based collection in Redis, including counters, rolling time windows, and hyperloglog counters.
Zookeeper Watcher
Module for watching a zookeeper file and handles zookeeper session connection troubles and re-establishment of watches.