Scrapy extenstion to control spiders using JSON-RPC
Project description
scrapy-jsonrpc is an extension to control a running Scrapy web crawler via JSON-RPC. The service provides access to the main Crawlera object via the JSON-RPC 2.0 protocol.
It’s enabled by the JSONRPC_ENABLED setting. The web server will listen in the port specified in JSONRPC_PORT, and will log to the file specified in JSONRPC_LOGFILE.
The endpoint for accessing the crawler object is:
http://localhost:6080/crawler
Example client
There is a command line tool provided for illustration purposes on how to build a client. You can find it in example-client.py. It supports a few basic commands such as listing the running spiders, etc.
Settings
These are the settings that control the web service behaviour:
JSONRPC_ENABLED
Default: True
A boolean which specifies if the web service will be enabled (provided its extension is also enabled).
JSONRPC_LOGFILE
Default: None
A file to use for logging HTTP requests made to the web service. If unset web the log is sent to standard scrapy log.
JSONRPC_PORT
Default: [6080, 7030]
The port range to use for the web service. If set to None or 0, a dynamically assigned port is used.
JSONRPC_HOST
Default: '127.0.0.1'
The interface the web service should listen on.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for scrapy_jsonrpc-0.2.0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3610403c4e7954a410c35afb9742aec3e2ffc5730e91fbea7621bcda5b25c355 |
|
MD5 | fcf3fde0b3c17b94bbba5487171b8b34 |
|
BLAKE2b-256 | 7f1d21b2e86dc423510441bc490e7bbb7092fe21e73de18e9f14aea86056a89d |