buku 2.7

<h1 align="center">Buku</h1>

<p align="center">
<a href="https://github.com/jarun/Buku/releases/latest"><img src="https://img.shields.io/github/release/jarun/buku.svg" alt="Latest release" /></a>
<a href="https://aur.archlinux.org/packages/buku"><img src="https://img.shields.io/aur/version/buku.svg" alt="AUR" /></a>
<a href="http://braumeister.org/formula/buku"><img src="https://img.shields.io/homebrew/v/buku.svg" alt="Homebrew" /></a>
<a href="https://github.com/jarun/buku/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-GPLv3-yellow.svg?maxAge=2592000" alt="License" /></a>
<a href="https://travis-ci.org/jarun/Buku"><img src="https://travis-ci.org/jarun/Buku.svg?branch=master" alt="Build Status" /></a>
</p>

<p align="center">
<a href="https://asciinema.org/a/9l6s2ppivpo661nu5slwk2t6y"><img src="https://asciinema.org/a/9l6s2ppivpo661nu5slwk2t6y.png" alt="Asciicast" width="734"/></a>
</p>

`buku` is a powerful bookmark management utility written in Python3 and SQLite3. When I started writing it, I couldn't find a flexible cmdline solution with a private, portable, merge-able database along with browser integration. Hence, `buku` (after my son's nickname).

With tagging and multiple options to search bookmarks, including regex and a deep scan mode (particularly for URLs), finding a bookmark is very easy. Multiple search results can be opened in the browser at once.

Though a terminal utility, it's possible to add bookmarks to `buku` without touching the terminal! Refer to the section on [GUI integration](#gui-integration). If you prefer the terminal, thanks to the shell completion scripts, you don't need to memorize any of the options. There's an Easter egg to revisit random forgotten bookmarks too.

*Buku* is too busy to track you - no history, obsolete records, usage analytics or homing.

<br>
<p align="center">
<a href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=RMLTQ76JSXJ4Q"><img src="https://img.shields.io/badge/paypal-donate-orange.svg?maxAge=2592000" alt="Donate" /></a>
&nbsp;
<a href="https://gitter.im/jarun/Buku"><img src="https://img.shields.io/gitter/room/jarun/buku.svg?maxAge=2592000" alt="gitter chat" /></a>
</p>

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Dependencies](#dependencies)
- [Installing from this repository](#installing-from-this-repository)
- [Running as a standalone utility](#running-as-a-standalone-utility)
- [Debian package](#debian-package)
- [Installing with a package manager](#installing-with-a-package-manager)
- [Shell completion](#shell-completion)
- [Usage](#usage)
- [Cmdline options](#cmdline-options)
- [Operational notes](#operational-notes)
- [GUI integration](#gui-integration)
- [Add bookmarks from anywhere](#add-bookmarks-from-anywhere)
- [Import bookmarks to browser](#import-bookmarks-to-browser)
- [Sync database across systems](#sync-database-across-systems)
- [As a library](#as-a-library)
- [Related projects](#related-projects)
- [Mentions](#mentions)
- [Examples](#examples)
- [Contributions](#contributions)
- [Copyright](#copyright)

## Features

- Add, open, tag, comment on, update, remove, shorten URLs
- Fetch page title from web, refresh all titles in a go
- Multiple search options, continuous search at prompt
- Portable, merge-able database to sync between systems
- Import/export in Markdown or HTML (FF, Chrome compatible)
- Open (multiple) search results directly in default browser
- Manual password protection using AES256 encryption
- Completion scripts (Bash, Fish, Zsh), man page with examples
- Several options for power users (see help or man page)
- Fast and clean interface, distinct symbols for record fields
- Minimal dependencies

## Installation

### Dependencies

`buku` requires Python 3.3 or later.

To install package dependencies, run:

$ sudo pip3 install urllib3 cryptography beautifulsoup4 requests
or on Ubuntu:

$ sudo apt-get install python3-urllib3 python3-cryptography python3-bs4 python3-requests

### Installing from this repository

If you have git installed, run:

$ git clone https://github.com/jarun/Buku/
or download the latest [stable release](https://github.com/jarun/Buku/releases/latest) or [development version](https://github.com/jarun/Buku/archive/master.zip).

Install to default location (`/usr/local`):

$ sudo make install

To remove, run:

$ sudo make uninstall
`PREFIX` is supported. You may need to use `sudo` with `PREFIX` depending on your permissions on destination directory.

#### Running as a standalone utility

`buku` is a standalone utility. From the containing directory, run:

$ chmod +x buku.py
$ ./buku.py

#### Debian package

If you are on a Debian (including Ubuntu) based system visit [the latest stable release](https://github.com/jarun/Buku/releases/latest) and download the `.deb` package. To install, run:

$ sudo dpkg -i buku-$version-all.deb

Please substitute `$version` with the appropriate package version.

### Installing with a package manager

`buku` is also available on
- [AUR](https://aur.archlinux.org/packages/buku/) for Arch Linux
- [Ubuntu](https://launchpad.net/ubuntu/+source/buku)
- [Homebrew](http://braumeister.org/formula/buku) for OS X
- [Debian Sid](https://packages.debian.org/sid/buku)
- [Ubuntu PPA](https://launchpad.net/~twodopeshaggy/+archive/ubuntu/jarun/)
- Void Linux repos ( `$ sudo xbps-install -S buku` )

## Shell completion

Shell completion scripts for Bash, Fish and Zsh can be found in respective subdirectories of [auto-completion/](https://github.com/jarun/Buku/blob/master/auto-completion). Please refer to your shell's manual for installation instructions.

`buku` has a [rofi frontend](https://github.com/carnager/buku_run) written by Rasmus Steinke.

## Usage

### Cmdline options

usage: buku [OPTIONS] [KEYWORD [KEYWORD ...]]

Powerful command-line bookmark manager. Your mini web!

general options:
-a, --add URL [tag, ...]
bookmark URL with comma-separated tags
-u, --update [...] update fields of bookmark at DB indices
accepts indices and ranges
refresh all titles, if no arguments
refresh titles of bookmarks at indices,
if no edit options are specified
-d, --delete [...] delete bookmarks. Valid inputs: either
a hyphenated single range (100-200),
OR space-separated indices (100 15 200)
delete results with search options
delete all bookmarks, if no arguments
-h, --help show this information and exit

edit options:
--url keyword specify url, works only with -u option
--tag [+|-] [...] set comma-separated tags with -a and -u
clear tags, if no arguments
append to tags, if preceded by '+'
remove from tags, if preceded by '-'
-t, --title [...] manually set title, works with -a, -u
if no arguments:
-a: do not set title, -u: clear title
-c, --comment [...] description of the bookmark, works with
-a, -u; clears comment, if no arguments
--immutable N disable title fetch from web on update
works with -a, -u
N=0: mutable (default), N=1: immutable

search options:
-s, --sany keyword [...]
find records with ANY search keyword
-S, --sall keyword [...]
find records with ALL search keywords
special keywords -
"blank": entries with empty title/tag
"immutable": entries with locked title
--deep match substrings ('pen' matches 'opens')
--sreg expression run a regex search
--stag [...] search bookmarks by a tag
list all tags, if no arguments

encryption options:
-l, --lock [N] encrypt DB file with N (> 0, default 8)
hash iterations to generate key
-k, --unlock [N] decrypt DB file with N (> 0, default 8)
hash iterations to generate key

power toys:
-e, --export file export bookmarks to Firefox format html
use --tag to export only specific tags
-i, --import file import bookmarks from html file
FF and Google Chrome formats supported
--markdown use markdown with -e and -i
format: [title](url), 1 per line
-m, --merge file add bookmarks from another buku DB file
-p, --print [...] show details of bookmark by DB index
accepts indices and ranges
show all bookmarks, if no arguments
-f, --format N limit fields in -p or Json search output
1: URL, 2: URL and tag, 3: title
-r, --replace oldtag [newtag ...]
replace oldtag with newtag everywhere
delete oldtag, if no newtag
-j, --json Json formatted output for -p and search
--noprompt do not show the prompt, run and exit
-o, --open [N] open bookmark at DB index N in browser
open a random index if N is omitted
--shorten N/URL fetch shortened url from tny.im service
accepts either a DB index or a URL
--tacit reduce verbosity
--threads N max network connections in full refresh
default 4, min 1, max 10
--upstream check latest upstream version available
-z, --debug show debug information and verbose logs

symbols:
> title
+ comment
# tags

### Operational notes

- The database file is stored in:
- **$XDG_DATA_HOME/buku/bookmarks.db**, if XDG_DATA_HOME is defined (first preference) or
- **$HOME/.local/share/buku/bookmarks.db**, if HOME is defined (second preference) or
- the **current directory**.
- If the URL contains characters like `;`, `&` or brackets they may be interpreted specially by the shell. To avoid it, add the URL within single or double quotes (`'`/`"`).
- URLs are unique in DB. The same URL cannot be added twice.
- Bookmarks with immutable titles are listed with bold `(L)` after the URL.
- **Tags**:
- Comma (`,`) is the tag delimiter in DB. A tag cannot have comma(s) in it. Tags are filtered (for unique tags) and sorted. Tags are stored in lower case and can be replaced, appended or deleted.
- **Update** operation:
- If --title, --tag or --comment is passed without argument, clear the corresponding field from DB.
- If --url is passed (and --title is omitted), update the title from web using the URL.
- If indices are passed without any other options (--url, --title, --tag, --comment and --immutable), read the URLs from DB and update titles from web. Bookmarks marked immutable are skipped.
- **Delete** operation:
- When a record is deleted, the last record is moved to the index.
- Delete doesn't work with range and indices provided together as arguments. It's an intentional decision to avoid extra sorting, in-range checks and to keep the auto-DB compaction functionality intact. On the same lines, indices are deleted in descending order.
- Can delete bookmarks matching a search, when combined with any of the search options.
- **Search** works in mysterious ways:
- Case-insensitive.
- Matches words in URL, title and tags.
- --sany : match any of the keywords in URL, title or tags.
- --sall : match all the keywords in URL, title or tags.
- --deep : match **substrings** (`match` matches `rematched`) in URL, title and tags.
- --sreg : match a regular expression (ignores --deep).
- --stag : search bookmarks by a tag, or list all tags alphabetically with usage count (if no arguments).
- Search results are indexed serially. This index is different from actual database index of a bookmark record which is shown in bold within `[]` after the URL.
- **Encryption** is optional and manual. AES256 algorithm is used. To use encryption, the database file should be unlocked (-k) before using buku and locked (-l) afterwards. Between these 2 operations, the database file lies unencrypted on the disk, and NOT in memory. Also, note that the database file is *unencrypted on creation*.
- **Proxy** support: environment variable *https_proxy*, if defined, is used to tunnel data for both http and https connections. The supported format is:

http[s]://[username:password@]proxyhost:proxyport/

## GUI integration


`buku` can be integrated in a GUI environment with simple tweaks.

### Add bookmarks from anywhere

With support for piped input, it's possible to add bookmarks to `buku` using keyboard shortcuts on Linux and OS X. CLIPBOARD (plus PRIMARY on Linux) text selections can be added directly this way. The additional utility required is `xsel` (on Linux) or `pbpaste` (on OS X).

The following steps explore the procedure on Linux with Ubuntu as the reference platform.

1. To install `xsel` on Ubuntu, run:

$ sudo apt install xsel
2. Create a new script `bukuadd` with the following content:

#!/bin/bash

xsel | buku -a

`-a` is the option to add a bookmark.
3. Make the script executable:

$ chmod +x bukuadd
4. Copy it somewhere in your `PATH`.
5. Add a new keyboard shortcut to run the script. I use `<Alt-b>`.

#### Test drive

Select a URL anywhere or copy a link and press the keyboard shortcut to add it to the `buku` database. The addition might take a few seconds to reflect depending on your internet speed and the time `buku` needs to fetch the title from the URL. To avoid title fetch from the web, add the `-t` option to the script.

To verify that the bookmark has indeed been added, run:

$ buku -p | tail -3
and check the entry.

#### Tips

- To add the last visited URL in Firefox to `buku`, use the following script:

#!/bin/bash

sqlite3 $HOME/.mozilla/firefox/*.default/places.sqlite "select url from moz_places where last_visit_date=(select max(last_visit_date) from moz_places)" | buku -a
- If you want to tag these bookmarks, look them up later using:

$ buku -S blank
Use option `-u` to tag these bookmarks.

### Import bookmarks to browser

`buku` can export (or import) bookmarks in HTML format recognized by Firefox, Google Chrome and Internet Explorer.

To export all bookmarks, run:

$ buku --export path_to_bookmarks.html
To export specific tags, run:

$ buku --export path_to_bookmarks.html --tag tag 1, tag 2
Once exported, import the html file in your browser.

## Sync database across systems

`buku` has the capability to import records from another `buku` database file. However, users with a cloud service client installed on multiple systems can keep the database synced across these systems automatically. To achieve this store the actual database file in a synced directory and create a symbolic link to it in the location where the database file would exist otherwise. For example, `$HOME/.local/share/buku/bookmarks.db` can be a symbolic link to `~/synced_dir/bookmarks.db`.

## As a library

`buku` can be used as a powerful bookmark management library. All functionality are available through carefully designed APIs. `main()` is a good usage example. It's also possible to use a custom database file in multi-user scenarios. Check out the documentation for the following APIs which accept an optional argument as database file:

BukuDb.initdb(dbfile=None)
BukuCrypt.encrypt_file(iterations, dbfile=None)
BukuCrypt.decrypt_file(iterations, dbfile=None)
NOTE: This flexibility is not exposed in the program.

## Related projects

- [buku_run](https://github.com/carnager/buku_run), a rofi frontend
- [oil](https://github.com/AndreiUlmeyda/oil), a search-as-you-type cli frontend

## Mentions

- [One Thing Well](http://onethingwell.org/post/144952807044/buku)
- [It's F.O.S.S.](https://itsfoss.com/buku-command-line-bookmark-manager-linux/)
- [Make Tech Easier](https://www.maketecheasier.com/manage-browser-bookmarks-ubuntu-command-line/)

## Examples

1. **Add** a bookmark with **tags** `linux news` and `open source`, **comment** `Informative website on Linux and open source`, **fetch page title** from the web:

$ buku -a https://tuxdiary.com linux news, open source -c Informative website on Linux and open source
Title: [TuxDiary – Linux, open source, command-line, leisure.]
Added at index 336

336. https://tuxdiary.com
> TuxDiary – Linux, open source, command-line, leisure.
+ Informative website on Linux and open source
# linux news,open source
where, >: title, +: comment, #: tags
2. **Add** a bookmark with tags `linux news` and `open source` & **immutable custom title** `Linux magazine`:

$ buku -a http://tuxdiary.com linux news, open source -t 'Linux magazine' --immutable 1
336. http://tuxdiary.com (L)
> Linux magazine
# linux news,open source
Note that URL must precede tags.
3. **Add** a bookmark **without a title** (works for update too):

$ buku -a http://tuxdiary.com linux news, open source -t
4. **Update** existing bookmark at index 15012014 with new URL, tags and comments, fetch title from the web:

$ buku -u 15012014 --url http://tuxdiary.com/ --tag linux news, open source, magazine -c site for Linux utilities
5. **Fetch and update only title** for bookmark at 15012014:

$ buku -u 15012014
6. **Update only comment** for bookmark at 15012014:

$ buku -u 15012014 -c this is a new comment
Applies to --url, --title and --tag too.
7. **Export** bookmarks tagged `tag 1` or `tag 2` to HTML and markdown:

$ buku -e bookmarks.html --tag tag 1, tag 2
$ buku -e bookmarks.md --markdown --tag tag 1, tag 2
All bookmarks are exported if --tag is not specified.
8. **Import** bookmarks from HTML and markdown:

$ buku -i bookmarks.html
$ buku -i bookmarks.md --markdown
9. **Delete only comment** for bookmark at 15012014:

$ buku -u 15012014 -c
Applies to --title and --tag too. URL cannot be deleted without deleting the bookmark.
10. **Update** or refresh **full DB** with page titles from the web:

$ buku -u
$ buku -u --tacit (show only failures and exceptions)
This operation does not modify the indexes, URLs, tags or comments. Only title is refreshed if fetched title is non-empty.
11. **Delete** bookmark at index 15012014:

$ buku -d 15012014
Index 15012020 moved to 15012014
The last index is moved to the deleted index to keep the DB compact.
12. **Delete all** bookmarks:

$ buku -d
13. **Delete** a **range or list** of bookmarks:

$ buku -d 100-200
$ buku -d 100 15 200
14. **Search** bookmarks for **ANY** of the keywords `kernel` and `debugging` in URL, title or tags:

$ buku -s kernel debugging
15. **Search** bookmarks with **ALL** the keywords `kernel` and `debugging` in URL, title or tags:

$ buku -S kernel debugging
16. **Search** bookmarks **tagged** `general kernel concepts`:

$ buku --stag general kernel concepts
17. List **all unique tags** alphabetically:

$ buku --stag
18. Run a **search and delete** the results:

$ buku -s kernel debugging -d
19. **Encrypt or decrypt** DB with **custom number of iterations** (15) to generate key:

$ buku -l 15
$ buku -k 15
The same number of iterations must be specified for one lock & unlock instance. Default is 8, if omitted.
20. **Show details** of bookmarks at index 15012014 and ranges 20-30, 40-50:

$ buku -p 20-30 15012014 40-50
21. **Show all** bookmarks with real index from database:

$ buku -p
$ buku -p | more
22. **Replace tag** 'old tag' with 'new tag':

$ buku -r 'old tag' new tag
23. **Delete tag** 'old tag' from DB:

$ buku -r 'old tag'
24. **Append (or delete) tags** 'tag 1', 'tag 2' to (or from) existing tags of bookmark at index 15012014:

$ buku -u 15012014 --tag + tag 1, tag 2
$ buku -u 15012014 --tag - tag 1, tag 2
25. **Open URL** at index 15012014 in browser:

$ buku -o 15012014
26. List bookmarks with **no title or tags** for bookkeeping:

$ buku -S blank
27. List bookmarks with **immutable title**:

$ buku -S immutable
28. **Shorten URL** www.google.com and the URL at index 20:

$ buku --shorten www.google.com
$ buku --shorten 20
29. More **help**:

$ buku -h
$ man buku

## Contributions

Pull requests are welcome. Please visit [#103](https://github.com/jarun/Buku/issues/103) for a list of TODOs.

## Copyright

Copyright © 2015-2016 [Arun Prakash Jana](mailto:engineerarun@gmail.com)