s3tail 0.2.1

===============================
s3tail
===============================

.. image:: https://img.shields.io/pypi/v/s3tail.svg
:target: https://pypi.python.org/pypi/s3tail

.. image:: https://img.shields.io/travis/bradrf/s3tail.svg
:target: https://travis-ci.org/bradrf/s3tail

.. image:: https://readthedocs.org/projects/s3tail/badge/?version=latest
:target: https://s3tail.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status

.. image:: https://pyup.io/repos/github/bradrf/s3tail/shield.svg
:target: https://pyup.io/repos/github/bradrf/s3tail/
:alt: Updates


S3tail is a simple tool to help access log files stored in an S3 bucket in the same way one might
use the \*nix ``tail`` command (with far fewer options, most notably the lack of ``follow``).

* Free software: MIT license
* Documentation: https://s3tail.readthedocs.io.

Simplest install method is via ``pip install s3tail`` (see installation_ for other methods).


Features
--------

S3tail downloads and displays the content of files stored in S3, optionally starting at a specific
prefix. For example, the following will start dumping all the log file contents found for August the
fourth in the order S3 provides from that prefix onward:

.. code-block:: console

$s3tails3://my-logs/production-s3-access-2016-08-04Whens3tailisstoppedorinterrupted,i{t}^{\prime }llprintabookmarktobeusedtopickupattheexactspotfollowingthelastlogprintedinapreviousrun.Somethinglikethefollowingmightbeusedtoleveragethisabilitytocontinuetailingfromapreviousstoppingpoint:..code-block::console$ s3tail s3://my-logs/production-s3-access-2016-08-04
...
...a-bunch-of-file-output...
...
Bookmark: production-s3-access-2016-08-04-00-20-31-61059F36E0DBF36E:706

This can then be used to pick up at line ``707`` later on, like this:

.. code-block:: console

$s3tails3://my-logs/production-s3-access-2016-08-04--bookmarkproduction-s3-access-2016-08-04-00-20-31-61059F36E0DBF36E:706Additionally,i{t}^{\prime }softenusefultolets3tailtrackwherethingswereleftoffandpickupatthatspotwithoutneedingtocopyandpastethepreviousbookmark.Thisiswhere"namedbookmarks"comeinhandy.Theexamplesabovecouldhavebeenreducedtotheseoperations:..code-block::console$ s3tail --bookmark my-special-spot s3://my-logs/production-s3-access-2016-08-04
...
^C
$s3tail--bookmarkmy-special-spots3://my-logs/production-s3-accessStartingproduction-s3-access-2016-08-04-02-22-32-415AE699C8233AC3Foundproduction-s3-access-2016-08-04-02-22-32-415AE699C8233AC3incachePickedupatline707...I{t}^{\prime }ssafetoreruns3tailsessionswhenworkingwithpipedcommandssearchingfordatainthestream(e.g.‘‘grep‘‘).S3tailkeepsfilesinalocalfilesystemcache(for24hoursbydefault)andwillalwaysreadanddisplayfromthecachebeforedownloadingfromS3.Thisisdoneinabest-effortbackgroundthreadtoavoidimpactingperformance.Thefilecacheisstoredintheuse{r}^{\prime }s‘‘HOME‘‘directory,inan‘‘.s3tailcache‘‘subdirectory,wherethefilenamesaretheS3keyshashedwithSHA-256.Thesecanbelistedthroughtheuseofthe‘‘--cache-lookup‘‘option:..code-block::console$ s3tail --cache-lookup s3://my-logs/production-s3-access-2016-08-04

my-logs/production-s3-access-2016-08-04-23-20-40-9935D31F89E5E38B
=> NOT IN CACHE
my-logs/production-s3-access-2016-08-04-23-20-45-D76C63A0478F829B
=> NOT IN CACHE
my-logs/production-s3-access-2016-08-04-23-20-51-C14A8D0980A9F562
=> NOT IN CACHE
...
my-logs/production-s3-access-2016-08-04-23-24-02-C9DF441E6B14EFBB
=> /Users/brad/.s3tailcache/05/0536db5ed3938c0b7fb8d2809bf8b4eb1a686ba14c9dc9b09aafc20780ef0528
my-logs/production-s3-access-2016-08-04-23-24-10-E9E55E9019AA46D0
=> /Users/brad/.s3tailcache/d1/d1c8b060d7c9a59c6387fc93b7a3d42db09ce90df2ed4eb71449e88e010ab4a8
my-logs/production-s3-access-2016-08-04-23-24-58-28FE2F9927BCBEA3
=> /Users/brad/.s3tailcache/46/46de81db7cd618074a8ff24cef938dca0d8353da3af8ccc67f517ba8600c3963

Check out usage_ for more details and examples (like how to leverage GoAccess to
generate beautiful traffic reports!).


Credits
-------

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project
template.

.. _installation: http://s3tail.readthedocs.io/en/latest/installation.html#installation
.. _usage: http://s3tail.readthedocs.io/en/latest/usage.html#usage
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage

.. raw:: html

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-84482808-1', 'auto');
ga('send', 'pageview');
</script>


=======
History
=======

0.2.1 (2016-12-27)
------------------

* Documentation.


0.2.0 (2016-12-27)
------------------

* Add gunzip for \*.gz files found (based only on extension name for now).
* Save configuration using ConfigStruct w/ overridable values.


0.1.7 (2016-09-18)
------------------

* Fix incorrect final bookmark when no more logs to read from key.


0.1.6 (2016-09-12)
------------------

* Documentation.


0.1.5 (2016-09-12)
------------------

* Documentation.


0.1.4 (2016-09-11)
------------------

* Fix bug in prefix matching when using named bookmarks.
* Added timestamps to logs.


0.1.3 (2016-09-11)
------------------

* Added "named" bookmarks to pick up automatically from last position when possible.
* Added option to disable cache entirely.


0.1.2 (2016-09-07)
------------------

* Better perf when reading from cache.
* Improved docs.


0.1.1 (2016-08-29)
------------------

* Refactor into classes and provide some minimal docs.


0.1.0 (2016-08-25)
------------------

* First release on PyPI.