Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
Project description
# spidey.py
> Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
## Installation
> Pypi Location: https://pypi.python.org/pypi/spidey.py
- Using Pypi - `pip install spidey`
## Usage
> Run `spidey` for Detailed help.
- `spidey --dir NEW_DIR --filter DOMAIN --url URL [--base BASE_URL]`
- `spidey --dir NEW_DIR --filter DOMAIN --url URL --max MAX_DOWNLOADS`
- Example - `spidey --dir test --filter 'www.google.com' --url 'https://www.google.com/' --max 20`
### More Examples
```
spidey \
-d test \
-f 'www.google.com' \
-u 'https://www.google.com/' \
-b 'https://www.google.com/' \
-hh '{"Accept" : "application/json"}' \
-n 2 \
-m 10 \
-s 5
```
```
spidey \
--dir test \
--filter 'www.google.com' \
--url 'https://www.google.com/'' \ \
--base 'https://www.google.com/
--headers '{"Accept" : "application/json"}' \
--depth 2 \
--max 10 \
--sleep 5
```
> Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
## Installation
> Pypi Location: https://pypi.python.org/pypi/spidey.py
- Using Pypi - `pip install spidey`
## Usage
> Run `spidey` for Detailed help.
- `spidey --dir NEW_DIR --filter DOMAIN --url URL [--base BASE_URL]`
- `spidey --dir NEW_DIR --filter DOMAIN --url URL --max MAX_DOWNLOADS`
- Example - `spidey --dir test --filter 'www.google.com' --url 'https://www.google.com/' --max 20`
### More Examples
```
spidey \
-d test \
-f 'www.google.com' \
-u 'https://www.google.com/' \
-b 'https://www.google.com/' \
-hh '{"Accept" : "application/json"}' \
-n 2 \
-m 10 \
-s 5
```
```
spidey \
--dir test \
--filter 'www.google.com' \
--url 'https://www.google.com/'' \ \
--base 'https://www.google.com/
--headers '{"Accept" : "application/json"}' \
--depth 2 \
--max 10 \
--sleep 5
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spidey.py-0.4.5.tar.gz
(4.0 kB
view hashes)
Built Distribution
Close
Hashes for spidey.py-0.4.5-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e958927065cf9a26dfd3dfcf0007103db01ead2b3d6657ce93db2b5bd25cc02 |
|
MD5 | d0f22da6ea8637d19fd21fd0f0deb5db |
|
BLAKE2b-256 | 63aba640c7c0e233ec3a7908e97a76d12dd7bb55e5026547744095e2e55a13bd |