scdlbot 0.9.1

=============================
Music Downloader Telegram Bot
=============================


.. image:: https://img.shields.io/badge/Telegram-@scdlbot-blue.svg
:target: https://t.me/scdlbot
:alt: Telegram Bot

.. image:: https://readthedocs.org/projects/scdlbot/badge/?version=latest
:target: https://scdlbot.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status

.. image:: https://img.shields.io/github/license/gpchelkin/scdlbot.svg
:target: https://raw.githubusercontent.com/gpchelkin/scdlbot/master/LICENSE.txt
:alt: GitHub License

.. image:: https://img.shields.io/pypi/v/scdlbot.svg
:target: https://pypi.org/project/scdlbot
:alt: PyPI Version

.. image:: https://pyup.io/repos/github/gpchelkin/scdlbot/shield.svg?token=376ffde2-5188-4912-bf3c-5f316e52d43f
:target: https://pyup.io/repos/github/gpchelkin/scdlbot/
:alt: PyUp Updates

.. image:: https://img.shields.io/travis/gpchelkin/scdlbot.svg
:target: https://travis-ci.org/gpchelkin/scdlbot
:alt: Travis CI Build Status

.. image:: https://codeclimate.com/github/gpchelkin/scdlbot/badges/gpa.svg
:target: https://codeclimate.com/github/gpchelkin/scdlbot
:alt: Code Climate GPA

.. image:: https://codeclimate.com/github/gpchelkin/scdlbot/badges/issue_count.svg
:target: https://codeclimate.com/github/gpchelkin/scdlbot
:alt: Code Climate Issue Count

.. image:: https://codeclimate.com/github/gpchelkin/scdlbot/badges/coverage.svg
:target: https://codeclimate.com/github/gpchelkin/scdlbot/coverage
:alt: Code Climate Test Coverage

.. image:: https://api.codacy.com/project/badge/Grade/7dfb6d8e7a094987b303e9283fc7368c
:target: https://www.codacy.com/app/gpchelkin/scdlbot
:alt: Codacy Build Status

.. image:: https://scrutinizer-ci.com/g/gpchelkin/scdlbot/badges/quality-score.png?b=master
:target: https://scrutinizer-ci.com/g/gpchelkin/scdlbot/?branch=master
:alt: Scrutinizer Code Quality

.. image:: https://bettercodehub.com/edge/badge/gpchelkin/scdlbot?branch=master
:target: https://bettercodehub.com
:alt: Better Code Hub Compliance

.. image:: https://codebeat.co/badges/102be98c-56c1-46af-895d-f1f15b2f2520
:target: https://codebeat.co/projects/github-com-gpchelkin-scdlbot-master
:alt: Codebeat Quality


Telegram Bot for downloading MP3 rips of tracks/sets from SoundCloud, Bandcamp, YouTube with tags and artwork.


* Free software: MIT license
* Documentation: https://scdlbot.readthedocs.io


.. contents:: :depth: 2


Bot Usage
---------

Send ``/start`` or ``/help`` command to bot or refer directly to the `help message <scdlbot/messages/help.tg.md>`__.

Please report all bugs and issues and suggest your improvements to `issues <https://github.com/gpchelkin/scdlbot/issues>`__.

Supported sites and used packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- `Telegram Bot API <https://core.telegram.org/bots/api>`__:
`python-telegram-bot <https://github.com/python-telegram-bot/python-telegram-bot>`__
- `SoundCloud <https://soundcloud.com>`__:
`scdl <https://github.com/flyingrub/scdl>`__
- `Bandcamp <https://bandcamp.com>`__:
`bandcamp-dl <https://github.com/iheanyi/bandcamp-dl>`__
- `YouTube <https://www.youtube.com/>`__,
`Mixcloud <https://www.mixcloud.com/>`__, etc.:
`youtube-dl <https://rg3.github.io/youtube-dl>`__

Development
-----------

Installation
~~~~~~~~~~~~

Requirements
^^^^^^^^^^^^

Those should be available in your ``PATH``:

- `Python 3.4+ <https://www.python.org/>`__
(`pyenv <https://github.com/pyenv/pyenv>`__ recommended)
- `FFmpeg <https://ffmpeg.org/download.html>`__ for running locally
(fresh builds for `Windows <https://ffmpeg.zeranoe.com/builds/>`__
and `Linux <https://johnvansickle.com/ffmpeg/>`__ recommended)
- `Heroku CLI <https://cli.heroku.com/>`__ is recommended

Install / Update from `PyPI <https://pypi.python.org/pypi/scdlbot>`__ (recommended)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

pip3 install scdlbot

Install / Update from `Git source <https://github.com/gpchelkin/scdlbot>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

git clone https://github.com/gpchelkin/scdlbot.git
cd scdlbot
pip3 install --requirement requirements.txt

# Update:
git pull
pip3 install --requirement requirements.txt

# System-wide install link to current sources, recommended:
python3 setup.py develop

# System-wide install copy of current sources, not recommended:
python3 setup.py install

Configuration
~~~~~~~~~~~~~

Download or copy config file sample and set up config environment variables in it:

::

# For PyPI installs: download sample config:
curl -o .env https://raw.githubusercontent.com/gpchelkin/scdlbot/master/.env.sample

# For Git source installs: copy sample config:
cp .env.sample .env

# Use your favourite editor:
nano .env

Required
^^^^^^^^

- ``TG_BOT_TOKEN``: Telegram Bot API Token, `obtain
here <https://t.me/BotFather>`__
- ``STORE_CHAT_ID``: Chat ID for storing audios for inline mode
- ``SC_AUTH_TOKEN``: SoundCloud Auth Token, `obtain
here <https://flyingrub.github.io/scdl/>`__

Optional
^^^^^^^^

- ``USE_WEBHOOK``: use webhook for bot updates: ``1``, use polling
(default): ``0``, `more
info <https://core.telegram.org/bots/api#getting-updates>`__
- ``APP_URL``: app URL like
``https://<appname>.herokuapp.com/``, required for webhook
- ``PORT``: port for webhook to listen to; Heroku sets this automatically
for web dynos
- ``BOTAN_TOKEN``: `Botan.io <http://botan.io/>`__
`token <http://appmetrica.yandex.com/>`__
- ``NO_CLUTTER_CHAT_IDS``: Comma-separated chat IDs with no replying
and caption hashtags
- ``BIN_PATH``: Custom directory where ``scdl`` and ``bandcamp-dl``
binaries are available, e.g. ``~/.pyenv/shims/`` if you use pyenv,
default: empty
- ``DL_DIR``: Parent directory for MP3 download directory, default: ~
(user's home directory)
- ``SYSLOG_ADDRESS``: Syslog server, for example ``logsX.papertrailapp.com:ABCDE``
- ``SYSLOG_DEBUG``: Enable verbose debug logging: 1
- ``HOSTNAME``: Hostname to show up in Syslog messages
- ``GOOGL_API_KEY``: `Goo.gl URL shortener <https://goo.gl>`__
`API key <https://developers.google.com/url-shortener/v1/getting_started#APIKey>`__

Telegram Settings
^^^^^^^^^^^^^^^^^

Send the commands from filenames in ``telegram_settings`` dir to `@BotFather <https://t.me/BotFather>`__, choose your bot and copy corresponding values in order to use the bot conveniently. Also disable privacy mode if you want to.


Running Locally
~~~~~~~~~~~~~~~

Using `Heroku Local <https://devcenter.heroku.com/articles/heroku-local#run-your-app-locally-using-the-heroku-local-command-line-tool>`__ (preferred)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You will need `Heroku CLI <https://cli.heroku.com/>`__ installed.

::

# For PyPI installs: first download Procfile:
curl -O https://raw.githubusercontent.com/gpchelkin/scdlbot/master/Procfile

# For long polling:
heroku local worker
# For webhooks (you will also need to set up some NGINX with SSL):
heroku local web

Using just Python
^^^^^^^^^^^^^^^^^

::

# For PyPI or Git source system-wide installs:
export $(cat .env | xargs)
scdlbot
# or just:
env $(cat .env | xargs) scdlbot

# For not-installed Git source repository directory:
export $(cat .env | xargs)
python -m scdlbot
# or just:
env $(cat .env | xargs) python -m scdlbot


Deploying to `Heroku <https://heroku.com/>`__
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

|Deploy|

Register on Heroku, press the button above and configure variables for deploying.
When app is deployed you **must** set only one dyno working on
"Resources" tab in your app settings depending on `which way of getting
updates <https://core.telegram.org/bots/api#getting-updates>`__ you have
chosen and set in config variables: ``worker`` for polling or ``web``
for webhook.

Manually
^^^^^^^^

You can do the same as the button above but using `Heroku
CLI <https://cli.heroku.com/>`__, not much of a fun. Assuming you are in
``scdbot`` repository directory:

::

heroku login
# Create app with Python3 buildpack and set it for upcoming builds:
heroku create --buildpack heroku/python
heroku buildpacks:set heroku/python
# Add FFmpeg buildpack needed for youtube-dl:
heroku buildpacks:add --index 1 https://github.com/laddhadhiraj/heroku-buildpack-ffmpeg.git --app scdlbot
# Deploy app to Heroku:
git push heroku master
# Set config vars automatically from your .env file
heroku plugins:install heroku-config
heroku config:push
# Or set them one by one:
heroku config:set TG_BOT_TOKEN="<TG_BOT_TOKEN>" STORE_CHAT_ID="<STORE_CHAT_ID>" ...

If you use webhook, start web dyno and stop worker dyno:

::

heroku ps:scale web=1 worker=0
heroku ps:stop worker

If you use polling, start worker dyno and stop web dyno:

::

heroku ps:scale worker=1 web=0
heroku ps:stop web

Some useful commands:

::

# Attach to logs:
heroku logs -t
# Test run ffprobe
heroku run "ffprobe -version"

Deploying to `Dokku <https://github.com/dokku/dokku>`__
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use Dokku (your own Heroku) installed on your own server. App is tested and fully
ready for deployment with polling (no webhook yet).
https://github.com/python-telegram-bot/python-telegram-bot/wiki/Webhooks#creating-a-self-signed-certificate-using-openssl

::

export DOKKU=<your_dokku_server>
scp .env $DOKKU:~
ssh $DOKKU
export DOKKU=<your_dokku_server>
dokku apps:create scdlbot
dokku certs:generate scdlbot scdlbot.$DOKKU
dokku config:set scdlbot $(cat .env | xargs)
logout
git remote add dokku dokku@$DOKKU:scdlbot
git push dokku master
ssh $DOKKU
dokku ps:scale scdlbot worker=1 web=0
dokku ps:restart scdlbot

.. |Deploy| image:: https://www.herokucdn.com/deploy/button.svg
:target: https://heroku.com/deploy


=======
History
=======

0.9.1 (2017-11-20)
------------------
* more error checking and responding
* limit youtube-dl time to download
* avoid live downloads
* logging refactor and refinement
* help updates

0.9.0 (2017-11-20)
------------------
* return of inline mode as fast download (link is sent to telegram servers for download)
* refactor and refinement
* help updates
* add some spam captions :)

0.8.3 (2017-11-19)
------------------
* gc.collect() according to https://github.com/jiaaro/pydub/issues/89#issuecomment-75245610

0.8.2 (2017-11-19)
------------------
* cool refinements in logging
* store urls, so button response is faster now

0.8.1 (2017-11-19)
------------------
* some logging fixes

0.8.0 (2017-11-19)
------------------
* many fixes and workarounds
* alerting & logging

0.7.10 (2017-11-05)
------------------
* botanio fix - send user id, not chat id

0.7.9 (2017-11-05)
------------------
* botanio fix
* tmpreaper config sample
* clutter fix

0.7.8 (2017-11-04)
------------------
* botanio
* maintenance

0.7.7 (2017-09-11)
------------------
* maintenance

0.7.6 (2017-09-11)
------------------
* SYSLOG_DEBUG env var to disable logging of full messages
* maintenance
* Logentries support

0.7.5.1 (2017-09-03)
------------------
* YouTube number remove

0.7.5 (2017-09-03)
------------------
* maintenance

0.7.4 (2017-08-03)
------------------
* msg_store fixes

0.7.3 (2017-07-20)
------------------
* orig_msg_id hotfix and don't send chat action on every link

0.7.2 (2017-07-19)
------------------
* Updated requirements

0.7.1 (2017-07-05)
------------------
* Hotfix

0.7.0 (2017-07-05)
------------------
* Travis CI, tests and docs from cookiecutter

0.6.3 (2017-07-04)
------------------

* Back to bandcamp-dl and scdl and download timeouts

0.6.2 (2017-07-04)
------------------

* Help message in groups now redirects to PM

0.6.1 (2017-07-03)
------------------

* Async run of download/send command
* Link command

0.6.0 (2017-07-02)
------------------

* Added text files to sdist
* Bandcamp and SoundCloud-widgets is now downloaded with youtube-dl
* Supported parsing widgets from pages
* Refactor

0.5.1 (2017-07-02)
------------------

* New clutter command
* Help refinements
* Some fixes

0.5.0 (2017-06-28)
------------------

* Big refactor to class-based
* Syslog support
* Some fixes

0.4.0 (2017-06-15)
------------------

* Console script!
* Setup script version improvements
* Ask in groups only, download immediately in private
* Bandcamp: Download links without 'bandcamp' for /dl
* Move TODOs to issues
* Button to destroy music from the Internet

0.3.1 (2017-06-12)
------------------

* Markdown to reStructuredText
* Copy tags to parts

0.3.0 (2017-06-10)
------------------

* YouTube playlists support
* Split audio by 50 MB size for sending
* Disable privacy mode and ask for download

0.2.0 (2017-06-06)
------------------

* Webhooks and async

0.1.0 (2017-06-04)
------------------

* First usable and stable version.