Tools for exporting IoTeX blockchain data to JSON
Project description
IoTeX ETL CLI
IoTeX ETL CLI lets you convert IoTeX data into JSON newline-delimited format.
Full documentation available here.
Quickstart
Install IoTeX ETL CLI:
pip3 install iotex-etl
Export blocks, actions and logs (Schema, Reference):
> iotexetl export_blocks --start-block 1 --end-block 500000 \
--output-dir output --provider-uri grpcs://api.mainnet.iotex.one:443
Stream blocks, actions and logs to console (Reference):
> pip3 install iotex-etl[streaming]
> iotexetl stream --start-block 500000 -e block,action,log --log-file log.txt \
--provider-uri grpcs://api.mainnet.iotex.one:443
Find other commands here.
For the latest version, check out the repo and call
> pip3 install -e .
> python3 iotexetl.py
Useful Links
Running Tests
> pip3 install -e .[dev,streaming]
> export IOTEXETL_PROVIDER_URI=grpcs://api.mainnet.iotex.one:443
> pytest -vv
Running Tox Tests
> pip3 install tox
> tox
Running in Docker
-
Install Docker https://docs.docker.com/install/
-
Build a docker image
> docker build -t iotex-etl:latest . > docker image ls
-
Run a container out of the image
> docker run -v $HOME/output:/iotex-etl/output iotex-etl:latest export_blocks -s 1 -e 5499999 -b 1000 -o out
-
Run streaming to console or Pub/Sub
> docker build -t iotex-etl:latest -f Dockerfile . > echo "Stream to console" > docker run iotex-etl:latest stream --start-block 500000 --log-file log.txt > echo "Stream to Pub/Sub" > docker run -v /path_to_credentials_file/:/iotex-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/iotex-etl/credentials_file.json iotex-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/mainnet
Generating iotex types and api code
-
Clone iotex-proto git repository
https://github.com/iotexproject/iotex-proto
-
Install grpcio-tools package
pip install grpcio-tools
-
Run following commands to generate code to python directory
mkdir python
python -m grpc_tools.protoc -I. -I ./proto/types --python_out=./python --grpc_python_out=./python ./proto/api/api.proto
python -m grpc_tools.protoc -I. -I ./proto/types --python_out=./python --grpc_python_out=./python ./proto/types/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.