Use environment vars; fix modularising a bit

Add a Codacy badge to README.rst

.gitignore

@@ -87,3 +87,13 @@ ENV/
 
 # Rope project settings
 .ropeproject
+
+# vim
+*.swp
+
+# digimarks
+static/favicons
+tags
+*.db_*
+*.db
+settings.py

CHANGELOG.md

@@ -1,14 +1,136 @@
+# Changelog
+All notable changes to this project will be documented in this file.
 
-## v0.2.0 (unreleased)
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+## TODO
 
 - Sorting of bookmarks
   - Sort by title
   - Sort by date
+- Logging of actions
+- Add new way of authentication and editing bookmark collections:
+  https://github.com/aquatix/digimarks/issues/8 and https://github.com/aquatix/digimarks/issues/9
+- Change adding tags to use the MaterializeCSS tags: https://materializecss.com/chips.html
+- Do calls to the API endpoint of an existing bookmark when editing properties
+  (for example to update tags, title and such, also to already suggest title)
+- Look into compatibility with del.icio.us, so we can make use of existing browser integration
+- Add unit tests
 
 
-## v0.1.0
+## [Unreleased]
 
-2016-07-26
+### Added
+- 'lightblue' theme
+- 'black amoled' theme
+- Python 3 compatibility (tested with Python 3.5 and 3.6)
+- Accept 'HTTP 202' responses as 'OK'
+- API: Added endpoint for 'bookmarks', returning JSON
+- Top navigation items now have icons too, like the sidebar in mobile view
+- Download favicons from RealFaviconGenerator: https://realfavicongenerator.net/api/download_website_favicon
+- Added `/<systemkey>/findmissingfavicons` endpoint to fill in the blanks in the favicon collection
+- Added fallback favicon image (semitransparent digimarks 'M' logo) for bookmarks without a favicon. No more broken images.
+- Added theme support for buttons.
+- Autocompletion in bookmark search field
+- API: search endpoint
+- Redirect endpoint for a bookmark, de-referring to its url (`/r/<userkey>/<urlhash>`)
+
+### Changed
+- Fixed theming of browser chrome in mobile browsers
+- Changed link colour of 'dark' theme from blue to orange
+- Modified card padding so it fits more content
+- Fixed ability to select a checkbox in the add/edit bookmark form
+- Made the 404 page theme aware, falls back to default (green) theme
+- Fixed admin pages not working anymore due to `settings` object name clash
+- On Add/Edit bookmark and encountering a 301, show a better message about automatically changing the URL with the provided button
+- Switched to 1.0 (alpha 4) version of MaterializeCSS
+- jQuery-b-gone: changed all jQuery code to regular JavaScript code/MaterializeCSS framework
+- Fixed colour of filter text in search field for dark themes
+- Unified rendering of 'private' and 'public' views of bookmark cards
+- Code cleanups, readability fixes
+- digimarks User Agent string to correctly identify ourselves, also preventing servers blocking 'bots'
+- Text search now also finds matches in the 'note' and 'url' of a bookmark, aside from its title
+- Main navigation items ('tags' and 'add bookmark') are now buttons, better visible as action items.
+- Removed item limit for feeds
+- Form fields are now themed
+- Disabled browser autocomplete for forms, which generally interfered with editing bookmarks (e.g., tag field) and the search field,
+  which has its own autocomplete now
+- Changed default theme to the 'freshgreen' variant
+- Links are now themed in the proper colours everywhere
+
+### Removed
+- Removed dependency on jQuery
+
+
+## [1.1.0] - 2017-07-22
+
+### Added
+- Show 404 page if bookmark is not found when editing
+- Cache buster to force loading of the latest styling
+- Theming support, default is 'green'
+- Themes need an extra `theme` field in the User table
+- Added 'freshgreen' and 'dark' themes
+
+### Changed
+- Make running in a virtualenv optional
+- Fix for misalignment and size of hamburger icon
+- Updated Python (pip) dependencies
+- Updated MaterializeCSS and jQuery
+
+### Removed
+- Removed dependency on more_itertools
+- Removed dependency on utilkit
+
+
+## [1.0.0] - 2016-12-29
+
+- json view of public tag pages, returns all items
+- feed (rss/atom) view of public tag pages, returns latest 15
+- feed link on public tag page
+- Support for bookmarklets
+- UI tweaks
+- Redesigned cards with bigger favicon. Looks cleaner
+- Different favicon service with 60x60px icons
+- Prevent duplicate form submission on add/edit bookmark
+- Delete bookmark from bookmark card
+- Undo delete link in "Bookmark has been deleted" message
+- Delete public tag page
+- On tags overview page:
+  - Show which tags have public pages, with link
+  - How many bookmarks each tag has
+  - Statistics on:
+    - total tags
+    - number of public tag pages
+    - total number of bookmarks
+    - number of starred bookmarks
+    - number of bookmarks with a non-OK http status
+    - number of deleted bookmarks
+- Filter on 'star' status, 'broken' status (non-http-200-OK)
+- Bookmark can have a note now
+  - Note icon on card with text in title (desktop browser)
+  - Filter on bookmarks with a note
+- Show url domain name along with 'no title' for items without title
+- Catch connection timeouts and such
+- Open in new tab/window, prevent
+  http://davidebove.com/blog/2016/05/05/target_blank-the-vulnerability-in-your-browser/
+- Put the tag selection in a collapsible element to prevent clutter in edit window
+- Updated MaterializeCSS and jQuery
+
+
+## [0.2.0] - 2016-08-02
+
+- Favicon courtesy Freepik on flaticon.com
+- Tag tags for easy adding of tags
+- Updates to MaterializeCSS and jQuery
+- Several bug- and code style fixes
+- Styling tweaks
+- Added 'Add bookmark' FAB to the bookmarks overview
+- Option to strip parameters from url (like '?utm_source=social')
+
+
+## [0.1.0] - 2016-07-26
 
 - Initial release
 - Flask application with functionality to add users, add and edit bookmarks,

README.rst

@@ -1,7 +1,7 @@
 digimarks
 =========
 
-|PyPI version| |PyPI downloads| |PyPI license| |Code health|
+|PyPI version| |PyPI license| |Code health| |Codacy|
 
 Simple bookmarking service, using a SQLite database to store bookmarks, supporting tags and automatic title fetching.
 
@@ -45,6 +45,16 @@ url's when wanted.
 Url's are of the form https://marks.example.com/<userkey>/<action>
 
 
+Bookmarklet
+~~~~~~~~~~~
+
+To easily save a link from your browser, open its bookmark manager and create a new bookmark with as url:
+
+.. code-block:: javascript
+
+    javascript:location.href='http://marks.example.com/1234567890abcdef/add?url='+encodeURIComponent(location.href);
+
+
 Creating a new user
 -------------------
 
@@ -72,18 +82,26 @@ What's new?
 See the `Changelog`_.
 
 
+Attributions
+------------
+
+'M' favicon by `Freepik`_.
+
+
 .. _digimarks: https://github.com/aquatix/digimarks
 .. _webhook: https://en.wikipedia.org/wiki/Webhook
 .. |PyPI version| image:: https://img.shields.io/pypi/v/digimarks.svg
    :target: https://pypi.python.org/pypi/digimarks/
-.. |PyPI downloads| image:: https://img.shields.io/pypi/dm/digimarks.svg
-   :target: https://pypi.python.org/pypi/digimarks/
 .. |PyPI license| image:: https://img.shields.io/github/license/aquatix/digimarks.svg
    :target: https://pypi.python.org/pypi/digimarks/
 .. |Code health| image:: https://landscape.io/github/aquatix/digimarks/master/landscape.svg?style=flat
    :target: https://landscape.io/github/aquatix/digimarks/master
    :alt: Code Health
+.. |Codacy| image:: https://api.codacy.com/project/badge/Grade/9a34319d917b43219a29e59e9ac75e3b
+   :alt: Codacy Badge
+   :target: https://app.codacy.com/app/aquatix/digimarks?utm_source=github.com&utm_medium=referral&utm_content=aquatix/digimarks&utm_campaign=badger
 .. _hook settings: https://github.com/aquatix/digimarks/blob/master/example_config/examples.yaml
 .. _vhost for Apache2.4: https://github.com/aquatix/digimarks/blob/master/example_config/apache_vhost.conf
 .. _uwsgi.ini: https://github.com/aquatix/digimarks/blob/master/example_config/uwsgi.ini
 .. _Changelog: https://github.com/aquatix/digimarks/blob/master/CHANGELOG.md
+.. _Freepik: http://www.flaticon.com/free-icon/letter-m_2041

digimarks.py

@@ -1,430 +0,0 @@
-import datetime
-import hashlib
-import os
-import sys
-import requests
-import shutil
-import bs4
-from more_itertools import unique_everseen
-from urlparse import urlparse
-
-from utilkit import datetimeutil
-
-from flask import Flask, abort, redirect, render_template, request, url_for
-from flask_peewee.db import Database
-#from flask_peewee.utils import get_object_or_404
-from peewee import *
-
-try:
-    import settings
-except ImportError:
-    print('Copy settings_example.py to settings.py and set the configuration to your own preferences')
-    sys.exit(1)
-
-# app configuration
-APP_ROOT = os.path.dirname(os.path.realpath(__file__))
-MEDIA_ROOT = os.path.join(APP_ROOT, 'static')
-MEDIA_URL = '/static/'
-DATABASE = {
-    'name': os.path.join(APP_ROOT, 'bookmarks.db'),
-    'engine': 'peewee.SqliteDatabase',
-}
-#PHANTOM = '/usr/local/bin/phantomjs'
-#SCRIPT = os.path.join(APP_ROOT, 'screenshot.js')
-
-# create our flask app and a database wrapper
-app = Flask(__name__)
-app.config.from_object(__name__)
-db = Database(app)
-
-# set custom url for the app, for example '/bookmarks'
-try:
-    app.config['APPLICATION_ROOT'] = settings.APPLICATION_ROOT
-except AttributeError:
-    pass
-
-# Cache the tags
-all_tags = {}
-
-
-def clean_tags(tags_list):
-    tags_res = [x.strip() for x in tags_list]
-    tags_res = list(unique_everseen(tags_res))
-    tags_res.sort()
-    if tags_res and tags_res[0] == '':
-        del tags_res[0]
-    return tags_res
-
-
-class User(db.Model):
-    """ User account """
-    username = CharField()
-    key = CharField()
-    created_date = DateTimeField(default=datetime.datetime.now)
-
-    def generate_key(self):
-        """ Generate userkey """
-        self.key = os.urandom(24).encode('hex')
-        return self.key
-
-
-class Bookmark(db.Model):
-    """ Bookmark instance, connected to User """
-    # Foreign key to User
-    userkey = CharField()
-
-    title = CharField(default='')
-    url = CharField()
-    #image = CharField(default='')
-    url_hash = CharField(default='')
-    tags = CharField(default='')
-    starred = BooleanField(default=False)
-
-    # Website (domain) favicon
-    favicon = CharField(null=True)
-
-    # Status code: 200 is OK, 404 is not found, for example (showing an error)
-    http_status = IntegerField(default=200)
-
-    created_date = DateTimeField(default=datetime.datetime.now)
-    modified_date = DateTimeField(null=True)
-    deleted_date = DateTimeField(null=True)
-
-    # Bookmark status; deleting doesn't remove from DB
-    VISIBLE = 0
-    DELETED = 1
-    status = IntegerField(default=VISIBLE)
-
-
-    class Meta:
-        ordering = (('created_date', 'desc'),)
-
-    #def fetch_image(self):
-    #    url_hash = hashlib.md5(self.url).hexdigest()
-    #    filename = 'bookmark-%s.png' % url_hash
-
-    #    outfile = os.path.join(MEDIA_ROOT, filename)
-    #    params = [PHANTOM, SCRIPT, self.url, outfile]
-
-    #    exitcode = subprocess.call(params)
-    #    if exitcode == 0:
-    #        self.image = os.path.join(MEDIA_URL, filename)
-
-    def set_hash(self):
-        """ Generate hash """
-        self.url_hash = hashlib.md5(self.url).hexdigest()
-
-    def set_title_from_source(self):
-        """ Request the title by requesting the source url """
-        try:
-            result = requests.get(self.url)
-            self.http_status = result.status_code
-        except:
-            # For example 'MissingSchema: Invalid URL 'abc': No schema supplied. Perhaps you meant http://abc?'
-            self.http_status = 404
-        if self.http_status == 200:
-            html = bs4.BeautifulSoup(result.text, 'html.parser')
-            try:
-                self.title = html.title.text.strip()
-            except AttributeError:
-                self.title = ''
-        return self.title
-
-    def set_status_code(self):
-        """ Check the HTTP status of the url, as it might not exist for example """
-        result = requests.head(self.url)
-        self.http_status = result.status_code
-        return self.http_status
-
-    def set_favicon(self):
-        """ Fetch favicon for the domain """
-        # http://codingclues.eu/2009/retrieve-the-favicon-for-any-url-thanks-to-google/
-        u = urlparse(self.url)
-        domain = u.netloc
-        filename = os.path.join(MEDIA_ROOT, 'favicons/' + domain + '.png')
-        # if file exists, don't re-download it
-        response = requests.get('http://www.google.com/s2/favicons?domain=' + domain, stream=True)
-        with open(filename, 'wb') as out_file:
-            shutil.copyfileobj(response.raw, out_file)
-        del response
-        self.favicon = domain + '.png'
-
-    def set_tags(self, tags):
-        """ Set tags from `tags`, strip and sort them """
-        tags_split = tags.split(',')
-        tags_clean = clean_tags(tags_split)
-        self.tags = ','.join(tags_clean)
-
-    @property
-    def tags_list(self):
-        """ Get the tags as a list, iterable in template """
-        if self.tags:
-            return self.tags.split(',')
-        else:
-            return []
-
-
-    def to_dict(self):
-        result = {
-                'title': self.title,
-                'url': self.url,
-                'created':  datetimeutil.datetime_to_string(self.created_date),
-                'url_hash': self.url_hash,
-                'tags': self.tags,
-                }
-        return result
-
-
-class PublicTag(db.Model):
-    """ Publicly shared tag """
-    tagkey = CharField()
-    userkey = CharField()
-    tag = CharField()
-    created_date = DateTimeField(default=datetime.datetime.now)
-
-    def generate_key(self):
-        """ Generate hash-based key for publicly shared tag """
-        self.tagkey = os.urandom(16).encode('hex')
-
-
-def get_tags_for_user(userkey):
-    """ Extract all tags from the bookmarks """
-    bookmarks = Bookmark.select().filter(Bookmark.userkey == userkey, Bookmark.status == Bookmark.VISIBLE)
-    tags = []
-    for bookmark in bookmarks:
-        tags += bookmark.tags_list
-    return clean_tags(tags)
-
-
-@app.errorhandler(404)
-def page_not_found(e):
-    return render_template('404.html'), 404
-
-
-@app.route('/')
-def index():
-    """ Homepage, point visitors to project page """
-    return render_template('index.html')
-
-
-@app.route('/<userkey>', methods=['GET', 'POST'])
-@app.route('/<userkey>/sort/<sortmethod>', methods=['GET', 'POST'])
-def bookmarks(userkey, sortmethod = None):
-    """ User homepage, list their bookmarks, optionally filtered and/or sorted """
-    #return object_list('bookmarks.html', Bookmark.select())
-    #user = User.select(key=userkey)
-    #if user:
-    #    bookmarks = Bookmark.select(User=user)
-    #    return render_template('bookmarks.html', bookmarks)
-    #else:
-    #    abort(404)
-    message = request.args.get('message')
-    if request.method == 'POST':
-        filter_on = request.form['filter']
-        bookmarks = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.title.contains(filter_on),
-                                            Bookmark.status == Bookmark.VISIBLE).order_by(Bookmark.created_date.desc())
-        return render_template('bookmarks.html', bookmarks=bookmarks, userkey=userkey, tags=all_tags[userkey], filter=filter_on, message=message)
-    else:
-        bookmarks = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.status == Bookmark.VISIBLE).order_by(Bookmark.created_date.desc())
-        return render_template('bookmarks.html', bookmarks=bookmarks, userkey=userkey, tags=all_tags[userkey], message=message)
-
-
-
-#@app.route('/<userkey>/<urlhash>')
-#def viewbookmark(userkey, urlhash):
-#    """ Bookmark detail view """
-#    bookmark = Bookmark.select(Bookmark.url_hash == urlhash, Bookmark.userkey == userkey)
-#    return render_template('viewbookmark.html', userkey=userkey, bookmark=bookmark)
-
-
-@app.route('/<userkey>/<urlhash>/json')
-def viewbookmarkjson(userkey, urlhash):
-    """ Serialise bookmark to json """
-    bookmark = Bookmark.select(Bookmark.url_hash == urlhash, Bookmark.userkey == userkey, Bookmark.status == Bookmark.VISIBLE)
-    return bookmark.to_dict()
-
-
-@app.route('/<userkey>/<urlhash>')
-@app.route('/<userkey>/<urlhash>/edit')
-def editbookmark(userkey, urlhash):
-    """ Bookmark edit form """
-    # bookmark = getbyurlhash()
-    bookmark = Bookmark.get(Bookmark.url_hash == urlhash, Bookmark.userkey == userkey)
-    message = request.args.get('message')
-    return render_template('edit.html', action='Edit bookmark', userkey=userkey, bookmark=bookmark, message=message, formaction='edit')
-
-
-@app.route('/<userkey>/add')
-def addbookmark(userkey):
-    """ Bookmark add form """
-    bookmark = Bookmark(title='', url='', tags='')
-    return render_template('edit.html', action='Add bookmark', userkey=userkey, bookmark=bookmark)
-
-
-def updatebookmark(userkey, request, urlhash = None):
-    """ Add (no urlhash) or edit (urlhash is set) a bookmark """
-    title = request.form.get('title')
-    url = request.form.get('url')
-    tags = request.form.get('tags')
-    starred = False
-    if request.form.get('starred'):
-        starred = True
-
-    if url and not urlhash:
-        # New bookmark
-        bookmark, created = Bookmark.get_or_create(url=url, userkey=userkey)
-        if not created:
-            message = 'Existing bookmark, did not overwrite with new values'
-            return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash, message=message))
-    elif url:
-        # Existing bookmark, get from DB
-        bookmark = Bookmark.get(userkey == userkey, Bookmark.url_hash == urlhash)
-        # Editing this bookmark, set modified_date to now
-        bookmark.modified_date = datetime.datetime.now()
-
-    bookmark.title = title
-    bookmark.url = url
-    bookmark.starred = starred
-    bookmark.set_tags(tags)
-    bookmark.set_hash()
-    #bookmark.fetch_image()
-    if not title:
-        # Title was empty, automatically fetch it from the url, will also update the status code
-        bookmark.set_title_from_source()
-    else:
-        bookmark.set_status_code()
-
-    if bookmark.http_status == 200:
-        bookmark.set_favicon()
-
-    bookmark.save()
-    return bookmark
-
-
-@app.route('/<userkey>/adding', methods=['GET', 'POST'])
-#@app.route('/<userkey>/adding')
-def addingbookmark(userkey):
-    """ Add the bookmark from form submit by /add """
-
-    if request.method == 'POST':
-        bookmark = updatebookmark(userkey, request)
-        if type(bookmark).__name__ == 'Response':
-            return bookmark
-        all_tags[userkey] = get_tags_for_user(userkey)
-        return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash))
-    return redirect(url_for('add'))
-
-
-@app.route('/<userkey>/<urlhash>/editing', methods=['GET', 'POST'])
-def editingbookmark(userkey, urlhash):
-    """ Edit the bookmark from form submit """
-
-    if request.method == 'POST':
-        bookmark = updatebookmark(userkey, request, urlhash=urlhash)
-        all_tags[userkey] = get_tags_for_user(userkey)
-        return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash))
-    return redirect(url_for('add'))
-
-
-@app.route('/<userkey>/<urlhash>/delete', methods=['GET', 'POST'])
-def deletingbookmark(userkey, urlhash):
-    """ Delete the bookmark from form submit by <urlhash>/delete """
-    query = Bookmark.update(status=Bookmark.DELETED).where(Bookmark.userkey==userkey, Bookmark.url_hash==urlhash)
-    query.execute()
-    query = Bookmark.update(deleted_date = datetime.datetime.now()).where(Bookmark.userkey==userkey, Bookmark.url_hash==urlhash)
-    query.execute()
-    message = 'Bookmark deleted'
-    all_tags[userkey] = get_tags_for_user(userkey)
-    return redirect(url_for('bookmarks', userkey=userkey, message=message))
-
-
-@app.route('/<userkey>/tags')
-def tags(userkey):
-    """ Overview of all tags used by user """
-    tags = get_tags_for_user(userkey)
-    print tags
-    return render_template('tags.html', tags=tags, userkey=userkey)
-
-
-@app.route('/<userkey>/tag/<tag>')
-def tag(userkey, tag):
-    """ Overview of all bookmarks with a certain tag """
-    bookmarks = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.tags.contains(tag), Bookmark.status == Bookmark.VISIBLE)
-    tags = get_tags_for_user(userkey)
-    pageheader = 'tag: ' + tag
-    message = request.args.get('message')
-
-    try:
-        publictag = PublicTag.get(PublicTag.userkey == userkey, PublicTag.tag == tag)
-    except PublicTag.DoesNotExist:
-        publictag = None
-
-    return render_template('bookmarks.html', bookmarks=bookmarks, userkey=userkey, tags=tags, tag=tag, publictag=publictag, action=pageheader, message=message)
-
-
-@app.route('/pub/<tagkey>')
-def publictag(tagkey):
-    """ Read-only overview of the bookmarks in the userkey/tag of this PublicTag """
-    #this_tag = get_object_or_404(PublicTag.select().where(PublicTag.tagkey == tagkey))
-    try:
-        this_tag = PublicTag.get(PublicTag.tagkey == tagkey)
-        bookmarks = Bookmark.select().where(Bookmark.userkey == this_tag.userkey, Bookmark.tags.contains(this_tag.tag), Bookmark.status == Bookmark.VISIBLE)
-        return render_template('publicbookmarks.html', bookmarks=bookmarks, tag=tag, action=this_tag.tag)
-    except PublicTag.DoesNotExist:
-        abort(404)
-
-
-@app.route('/<userkey>/<tag>/makepublic', methods=['GET', 'POST'])
-def addpublictag(userkey, tag):
-    #user = get_object_or_404(User.get(User.key == userkey))
-    try:
-        User.get(User.key == userkey)
-    except User.DoesNotExist:
-        abort(404)
-    try:
-        publictag = PublicTag.get(PublicTag.userkey == userkey, PublicTag.tag == tag)
-    except PublicTag.DoesNotExist:
-        publictag = None
-    if not publictag:
-        newpublictag = PublicTag()
-        newpublictag.generate_key()
-        newpublictag.userkey = userkey
-        newpublictag.tag = tag
-        newpublictag.save()
-
-        message = 'Public link to this tag created'
-        return redirect(url_for('tag', userkey=userkey, tag=tag, message=message))
-    else:
-        message = 'Public link already existed'
-        return redirect(url_for('tag', userkey=userkey, tag=tag, message=message))
-
-
-@app.route('/<systemkey>/adduser')
-def adduser(systemkey):
-    """ Add user endpoint, convenience """
-    if systemkey == settings.SYSTEMKEY:
-        newuser = User()
-        newuser.generate_key()
-        newuser.username = 'Nomen Nescio'
-        newuser.save()
-        all_tags[newuser.key] = []
-        return redirect('/' + newuser.key, code=302)
-    else:
-        abort(404)
-
-
-# Initialise
-# create the bookmark, user and public tag tables if they do not exist
-Bookmark.create_table(True)
-User.create_table(True)
-PublicTag.create_table(True)
-
-users = User.select()
-print 'Current user keys:'
-for user in users:
-    all_tags[user.key] = get_tags_for_user(user.key)
-    print user.key
-
-if __name__ == '__main__':
-    # run the application
-    app.run(port=9999, debug=True)

digimarks/__init__.py

@@ -0,0 +1,29 @@
+import os
+
+from flask import Flask
+
+
+def create_app(test_config=None):
+    # create our flask app and a database wrapper
+    app = Flask(__name__)
+    app.config.from_object(__name__)
+
+    if test_config is None:
+        # load the instance config, if it exists, when not testing
+        app.config.from_pyfile('settings.py', silent=True)
+    else:
+        # load the test config if passed in
+        app.config.from_mapping(test_config)
+
+    # Strip unnecessary whitespace due to jinja2 codeblocks
+    app.jinja_env.trim_blocks = True
+    app.jinja_env.lstrip_blocks = True
+
+    # set custom url for the app, for example '/bookmarks'
+    try:
+        # TODO: get settings from ENV vars
+        app.config['APPLICATION_ROOT'] = os.environ['APPLICATION_ROOT']
+    except AttributeError:
+        pass
+
+    return app

digimarks/marks.py

@@ -0,0 +1,649 @@
+from __future__ import print_function
+
+import datetime
+import os
+import sys
+from urllib.parse import urljoin
+
+from flask import abort, jsonify, redirect, render_template, request, url_for
+from werkzeug.contrib.atom import AtomFeed
+
+from . import themes
+from .models import Bookmark, PublicTag, User, get_tags_for_user
+
+DIGIMARKS_USER_AGENT = 'digimarks/2.0.0-dev'
+
+try:
+    import settings
+except ImportError:
+    print('Copy settings_example.py to settings.py and set the configuration to your own preferences')
+    sys.exit(1)
+
+# app configuration
+APP_ROOT = os.path.dirname(os.path.realpath(__file__))
+MEDIA_ROOT = os.path.join(APP_ROOT, 'static')
+MEDIA_URL = '/static/'
+DATABASE = {
+    'name': os.path.join(APP_ROOT, 'bookmarks.db'),
+    'engine': 'peewee.SqliteDatabase',
+}
+#PHANTOM = '/usr/local/bin/phantomjs'
+#SCRIPT = os.path.join(APP_ROOT, 'screenshot.js')
+SYSTEMKEY = None
+try:
+    SYSTEMKEY = os.environ['SYSTEMKEY']
+except KeyError:
+    print('No ENV var found for SYSTEMKEY')
+
+MASHAPE_API_KEY = None
+try:
+    MASHAPE_API_KEY = os.environ['MASHAPE_API_KEY']
+except KeyError:
+    print('No ENV var found for MASHAPE_API_KEY')
+
+# Cache the tags
+all_tags = {}
+usersettings = {}
+
+
+def get_cached_tags(userkey):
+    """ Fail-safe way to get the cached tags for `userkey` """
+    try:
+        return all_tags[userkey]
+    except KeyError:
+        return []
+
+
+def get_theme(userkey):
+    try:
+        usertheme = usersettings[userkey]['theme']
+        return themes.themes[usertheme]
+    except KeyError:
+        return themes.themes[themes.DEFAULT_THEME]  # default
+
+
+def make_external(url):
+    return urljoin(request.url_root, url)
+
+
+def _find_bookmarks(userkey, filter_text):
+    return Bookmark.select().where(
+        Bookmark.userkey == userkey,
+        (
+            Bookmark.title.contains(filter_text) |
+            Bookmark.url.contains(filter_text) |
+            Bookmark.note.contains(filter_text)
+        ),
+        Bookmark.status == Bookmark.VISIBLE
+    ).order_by(Bookmark.created_date.desc())
+
+
+@app.errorhandler(404)
+def page_not_found(e):
+    theme = themes.themes[themes.DEFAULT_THEME]
+    return render_template('404.html', error=e, theme=theme), 404
+
+
+@app.route('/')
+def index():
+    """ Homepage, point visitors to project page """
+    theme = themes.themes[themes.DEFAULT_THEME]
+    return render_template('index.html', theme=theme)
+
+
+def get_bookmarks(userkey, filtermethod=None, sortmethod=None):
+    """ User homepage, list their bookmarks, optionally filtered and/or sorted """
+    #return object_list('bookmarks.html', Bookmark.select())
+    #user = User.select(key=userkey)
+    #if user:
+    #    bookmarks = Bookmark.select(User=user)
+    #    return render_template('bookmarks.html', bookmarks)
+    #else:
+    #    abort(404)
+    message = request.args.get('message')
+    bookmarktags = get_cached_tags(userkey)
+
+    filter_text = ''
+    if request.form:
+        filter_text = request.form['filter_text']
+
+    filter_starred = False
+    if filtermethod and filtermethod.lower() == 'starred':
+        filter_starred = True
+
+    filter_broken = False
+    if filtermethod and filtermethod.lower() == 'broken':
+        filter_broken = True
+
+    filter_note = False
+    if filtermethod and filtermethod.lower() == 'note':
+        filter_note = True
+
+    if filter_text:
+        bookmarks = _find_bookmarks(userkey, filter_text)
+    elif filter_starred:
+        bookmarks = Bookmark.select().where(Bookmark.userkey == userkey,
+                                            Bookmark.starred).order_by(Bookmark.created_date.desc())
+    elif filter_broken:
+        bookmarks = Bookmark.select().where(Bookmark.userkey == userkey,
+                                            Bookmark.http_status != 200).order_by(Bookmark.created_date.desc())
+    elif filter_note:
+        bookmarks = Bookmark.select().where(Bookmark.userkey == userkey,
+                                            Bookmark.note != '').order_by(Bookmark.created_date.desc())
+    else:
+        bookmarks = Bookmark.select().where(
+            Bookmark.userkey == userkey,
+            Bookmark.status == Bookmark.VISIBLE
+        ).order_by(Bookmark.created_date.desc())
+
+    return bookmarks, bookmarktags, filter_text, message
+
+
+@app.route('/<userkey>', methods=['GET', 'POST'])
+@app.route('/<userkey>/filter/<filtermethod>', methods=['GET', 'POST'])
+@app.route('/<userkey>/sort/<sortmethod>', methods=['GET', 'POST'])
+@app.route('/<userkey>/<show_as>', methods=['GET', 'POST'])
+@app.route('/<userkey>/<show_as>/filter/<filtermethod>', methods=['GET', 'POST'])
+@app.route('/<userkey>/<show_as>/sort/<sortmethod>', methods=['GET', 'POST'])
+def bookmarks_page(userkey, filtermethod=None, sortmethod=None, show_as='cards'):
+    bookmarks, bookmarktags, filter_text, message = get_bookmarks(userkey, filtermethod, sortmethod)
+    theme = get_theme(userkey)
+    return render_template(
+        'bookmarks.html',
+        bookmarks=bookmarks,
+        userkey=userkey,
+        tags=bookmarktags,
+        filter_text=filter_text,
+        message=message,
+        theme=theme,
+        editable=True,  # bookmarks can be edited
+        showtags=True,  # tags should be shown with the bookmarks
+        filtermethod=filtermethod,
+        sortmethod=sortmethod,
+        show_as=show_as,  # show list of bookmarks instead of cards
+    )
+
+
+@app.route('/<userkey>/js')
+def bookmarks_js(userkey):
+    """ Return list of bookmarks with their favicons, to be used for autocompletion """
+    bookmarks = Bookmark.select().where(
+        Bookmark.userkey == userkey,
+        Bookmark.status == Bookmark.VISIBLE
+    ).order_by(Bookmark.created_date.desc())
+    return render_template(
+        'bookmarks.js',
+        bookmarks=bookmarks
+    )
+
+
+@app.route('/r/<userkey>/<urlhash>')
+def bookmark_redirect(userkey, urlhash):
+    """ Securely redirect a bookmark to its url, stripping referrer (if browser plays nice) """
+    # @TODO: add counter to this bookmark
+    try:
+        bookmark = Bookmark.get(
+            Bookmark.url_hash == urlhash,
+            Bookmark.userkey == userkey,
+            Bookmark.status == Bookmark.VISIBLE
+        )
+    except Bookmark.DoesNotExist:
+        abort(404)
+    return render_template('redirect.html', url=bookmark.url)
+
+
+@app.route('/api/v1/<userkey>', methods=['GET', 'POST'])
+@app.route('/api/v1/<userkey>/filter/<filtermethod>', methods=['GET', 'POST'])
+@app.route('/api/v1/<userkey>/sort/<sortmethod>', methods=['GET', 'POST'])
+def bookmarks_json(userkey, filtermethod=None, sortmethod=None):
+    bookmarks, bookmarktags, filter_text, message = get_bookmarks(userkey, filtermethod, sortmethod)
+
+    bookmarkslist = [i.serialize for i in bookmarks]
+
+    the_data = {
+        'bookmarks': bookmarkslist,
+        'tags': bookmarktags,
+        'filter_text': filter_text,
+        'message': message,
+        'userkey': userkey,
+    }
+    return jsonify(the_data)
+
+
+@app.route('/api/v1/<userkey>/<urlhash>')
+def bookmark_json(userkey, urlhash):
+    """ Serialise bookmark to json """
+    try:
+        bookmark = Bookmark.get(
+            Bookmark.url_hash == urlhash,
+            Bookmark.userkey == userkey,
+            Bookmark.status == Bookmark.VISIBLE
+        )
+        return jsonify(bookmark.to_dict())
+    except Bookmark.DoesNotExist:
+        return jsonify({'message': 'Bookmark not found', 'status': 'error 404'})
+
+
+@app.route('/api/v1/<userkey>/search/<filter_text>')
+def search_bookmark_titles_json(userkey, filter_text):
+    """ Serialise bookmark to json """
+    bookmarks = _find_bookmarks(userkey, filter_text)
+    result = []
+    for bookmark in bookmarks:
+        result.append(bookmark.to_dict())
+    return jsonify(result)
+
+
+@app.route('/<userkey>/<urlhash>')
+@app.route('/<userkey>/<urlhash>/edit')
+def editbookmark(userkey, urlhash):
+    """ Bookmark edit form """
+    # bookmark = getbyurlhash()
+    try:
+        bookmark = Bookmark.get(Bookmark.url_hash == urlhash, Bookmark.userkey == userkey)
+    except Bookmark.DoesNotExist:
+        abort(404)
+    message = request.args.get('message')
+    tags = get_cached_tags(userkey)
+    if not bookmark.note:
+        # Workaround for when an existing bookmark has a null note
+        bookmark.note = ''
+    theme = get_theme(userkey)
+    return render_template(
+        'edit.html',
+        action='Edit bookmark',
+        userkey=userkey,
+        bookmark=bookmark,
+        message=message,
+        formaction='edit',
+        tags=tags,
+        theme=theme
+    )
+
+
+@app.route('/<userkey>/add')
+def addbookmark(userkey):
+    """ Bookmark add form """
+    url = request.args.get('url')
+    if not url:
+        url = ''
+    if request.args.get('referrer'):
+        url = request.referrer
+    bookmark = Bookmark(title='', url=url, tags='')
+    message = request.args.get('message')
+    tags = get_cached_tags(userkey)
+    theme = get_theme(userkey)
+    return render_template(
+        'edit.html',
+        action='Add bookmark',
+        userkey=userkey,
+        bookmark=bookmark,
+        tags=tags,
+        message=message,
+        theme=theme
+    )
+
+
+def updatebookmark(userkey, urlhash=None):
+    """ Add (no urlhash) or edit (urlhash is set) a bookmark """
+    title = request.form.get('title')
+    url = request.form.get('url')
+    tags = request.form.get('tags')
+    note = request.form.get('note')
+    starred = False
+    if request.form.get('starred'):
+        starred = True
+    strip_params = False
+    if request.form.get('strip'):
+        strip_params = True
+
+    if url and not urlhash:
+        # New bookmark
+        bookmark, created = Bookmark.get_or_create(url=url, userkey=userkey)
+        if not created:
+            message = 'Existing bookmark, did not overwrite with new values'
+            return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash, message=message))
+    elif url:
+        # Existing bookmark, get from DB
+        bookmark = Bookmark.get(Bookmark.userkey == userkey, Bookmark.url_hash == urlhash)
+        # Editing this bookmark, set modified_date to now
+        bookmark.modified_date = datetime.datetime.now()
+    else:
+        # No url was supplied, abort. @TODO: raise exception?
+        return None
+
+    bookmark.title = title
+    if strip_params:
+        url = Bookmark.strip_url_params(url)
+    bookmark.url = url
+    bookmark.starred = starred
+    bookmark.set_tags(tags)
+    bookmark.note = note
+    bookmark.set_hash()
+    #bookmark.fetch_image()
+    if not title:
+        # Title was empty, automatically fetch it from the url, will also update the status code
+        bookmark.set_title_from_source()
+    else:
+        bookmark.set_status_code()
+
+    if bookmark.http_status == 200 or bookmark.http_status == 202:
+        try:
+            bookmark.set_favicon()
+        except IOError:
+            # Icon file could not be saved possibly, don't bail completely
+            pass
+
+    bookmark.save()
+    return bookmark
+
+
+@app.route('/<userkey>/adding', methods=['GET', 'POST'])
+#@app.route('/<userkey>/adding')
+def addingbookmark(userkey):
+    """ Add the bookmark from form submit by /add """
+    tags = get_cached_tags(userkey)
+
+    if request.method == 'POST':
+        bookmark = updatebookmark(userkey)
+        if not bookmark:
+            return redirect(url_for('addbookmark', userkey=userkey, message='No url provided', tags=tags))
+        if type(bookmark).__name__ == 'Response':
+            return bookmark
+        all_tags[userkey] = get_tags_for_user(userkey)
+        return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash))
+    return redirect(url_for('addbookmark', userkey=userkey, tags=tags))
+
+
+@app.route('/<userkey>/<urlhash>/editing', methods=['GET', 'POST'])
+def editingbookmark(userkey, urlhash):
+    """ Edit the bookmark from form submit """
+
+    if request.method == 'POST':
+        bookmark = updatebookmark(userkey, urlhash=urlhash)
+        all_tags[userkey] = get_tags_for_user(userkey)
+        return redirect(url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash))
+    return redirect(url_for('editbookmark', userkey=userkey, urlhash=urlhash))
+
+
+@app.route('/<userkey>/<urlhash>/delete', methods=['GET', 'POST'])
+def deletingbookmark(userkey, urlhash):
+    """ Delete the bookmark from form submit by <urlhash>/delete """
+    query = Bookmark.update(status=Bookmark.DELETED).where(Bookmark.userkey == userkey, Bookmark.url_hash == urlhash)
+    query.execute()
+    query = Bookmark.update(deleted_date=datetime.datetime.now()).where(
+        Bookmark.userkey == userkey,
+        Bookmark.url_hash == urlhash
+    )
+    query.execute()
+    message = 'Bookmark deleted. <a href="{}">Undo deletion</a>'.format(url_for(
+        'undeletebookmark',
+        userkey=userkey,
+        urlhash=urlhash
+    ))
+    all_tags[userkey] = get_tags_for_user(userkey)
+    return redirect(url_for('bookmarks_page', userkey=userkey, message=message))
+
+
+@app.route('/<userkey>/<urlhash>/undelete')
+def undeletebookmark(userkey, urlhash):
+    """ Undo deletion of the bookmark identified by urlhash """
+    query = Bookmark.update(status=Bookmark.VISIBLE).where(Bookmark.userkey == userkey, Bookmark.url_hash == urlhash)
+    query.execute()
+    message = 'Bookmark restored'
+    all_tags[userkey] = get_tags_for_user(userkey)
+    return redirect(url_for('bookmarks_page', userkey=userkey, message=message))
+
+
+@app.route('/<userkey>/tags')
+def tags_page(userkey):
+    """ Overview of all tags used by user """
+    tags = get_cached_tags(userkey)
+    #publictags = PublicTag.select().where(Bookmark.userkey == userkey)
+    alltags = []
+    for tag in tags:
+        try:
+            publictag = PublicTag.get(PublicTag.userkey == userkey, PublicTag.tag == tag)
+        except PublicTag.DoesNotExist:
+            publictag = None
+
+        total = Bookmark.select().where(
+            Bookmark.userkey == userkey,
+            Bookmark.tags.contains(tag),
+            Bookmark.status == Bookmark.VISIBLE
+        ).count()
+        alltags.append({'tag': tag, 'publictag': publictag, 'total': total})
+    totaltags = len(alltags)
+    totalbookmarks = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.status == Bookmark.VISIBLE).count()
+    totalpublic = PublicTag.select().where(PublicTag.userkey == userkey).count()
+    totalstarred = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.starred).count()
+    totaldeleted = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.status == Bookmark.DELETED).count()
+    totalnotes = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.note != '').count()
+    totalhttperrorstatus = Bookmark.select().where(Bookmark.userkey == userkey, Bookmark.http_status != 200).count()
+    theme = get_theme(userkey)
+    return render_template(
+        'tags.html',
+        tags=alltags,
+        totaltags=totaltags,
+        totalpublic=totalpublic,
+        totalbookmarks=totalbookmarks,
+        totaldeleted=totaldeleted,
+        totalstarred=totalstarred,
+        totalhttperrorstatus=totalhttperrorstatus,
+        totalnotes=totalnotes,
+        userkey=userkey,
+        theme=theme
+    )
+
+
+@app.route('/<userkey>/tag/<tag>')
+def tag_page(userkey, tag):
+    """ Overview of all bookmarks with a certain tag """
+    bookmarks = Bookmark.select().where(
+        Bookmark.userkey == userkey,
+        Bookmark.tags.contains(tag),
+        Bookmark.status == Bookmark.VISIBLE
+    ).order_by(Bookmark.created_date.desc())
+    tags = get_cached_tags(userkey)
+    pageheader = 'tag: ' + tag
+    message = request.args.get('message')
+
+    try:
+        publictag = PublicTag.get(PublicTag.userkey == userkey, PublicTag.tag == tag)
+    except PublicTag.DoesNotExist:
+        publictag = None
+
+    theme = get_theme(userkey)
+    return render_template(
+        'bookmarks.html',
+        bookmarks=bookmarks,
+        userkey=userkey,
+        tags=tags,
+        tag=tag,
+        publictag=publictag,
+        action=pageheader,
+        message=message,
+        theme=theme,
+        editable=True,
+        showtags=True,
+    )
+
+
+def get_publictag(tagkey):
+    """ Return tag and bookmarks in this public tag collection """
+    this_tag = PublicTag.get(PublicTag.tagkey == tagkey)
+    bookmarks = Bookmark.select().where(
+        Bookmark.userkey == this_tag.userkey,
+        Bookmark.tags.contains(this_tag.tag),
+        Bookmark.status == Bookmark.VISIBLE
+    ).order_by(Bookmark.created_date.desc())
+    return this_tag, bookmarks
+
+
+@app.route('/pub/<tagkey>')
+def publictag_page(tagkey):
+    """ Read-only overview of the bookmarks in the userkey/tag of this PublicTag """
+    #this_tag = get_object_or_404(PublicTag.select().where(PublicTag.tagkey == tagkey))
+    try:
+        this_tag, bookmarks = get_publictag(tagkey)
+        theme = themes.themes[themes.DEFAULT_THEME]
+        return render_template(
+            'publicbookmarks.html',
+            bookmarks=bookmarks,
+            tag=this_tag.tag,
+            action=this_tag.tag,
+            tagkey=tagkey,
+            theme=theme
+        )
+    except PublicTag.DoesNotExist:
+        abort(404)
+
+
+@app.route('/api/v1/pub/<tagkey>')
+def publictag_json(tagkey):
+    """ json representation of the Read-only overview of the bookmarks in the userkey/tag of this PublicTag """
+    try:
+        this_tag, bookmarks = get_publictag(tagkey)
+        result = {
+            #'tag': this_tag,
+            'tagkey': tagkey,
+            'count': len(bookmarks),
+            'items': [],
+        }
+        for bookmark in bookmarks:
+            result['items'].append(bookmark.to_dict())
+        return jsonify(result)
+    except PublicTag.DoesNotExist:
+        abort(404)
+
+
+@app.route('/pub/<tagkey>/feed')
+def publictag_feed(tagkey):
+    """ rss/atom representation of the Read-only overview of the bookmarks in the userkey/tag of this PublicTag """
+    try:
+        this_tag = PublicTag.get(PublicTag.tagkey == tagkey)
+        bookmarks = Bookmark.select().where(
+            Bookmark.userkey == this_tag.userkey,
+            Bookmark.tags.contains(this_tag.tag),
+            Bookmark.status == Bookmark.VISIBLE
+        )
+        feed = AtomFeed(this_tag.tag, feed_url=request.url, url=make_external(url_for('publictag_page', tagkey=tagkey)))
+        for bookmark in bookmarks:
+            updated_date = bookmark.modified_date
+            if not bookmark.modified_date:
+                updated_date = bookmark.created_date
+            bookmarktitle = '{} (no title)'.format(bookmark.url)
+            if bookmark.title:
+                bookmarktitle = bookmark.title
+            feed.add(
+                bookmarktitle,
+                content_type='html',
+                author='digimarks',
+                url=bookmark.url,
+                updated=updated_date,
+                published=bookmark.created_date
+            )
+        return feed.get_response()
+    except PublicTag.DoesNotExist:
+        abort(404)
+
+
+@app.route('/<userkey>/<tag>/makepublic', methods=['GET', 'POST'])
+def addpublictag(userkey, tag):
+    #user = get_object_or_404(User.get(User.key == userkey))
+    try:
+        User.get(User.key == userkey)
+    except User.DoesNotExist:
+        abort(404)
+    try:
+        publictag = PublicTag.get(PublicTag.userkey == userkey, PublicTag.tag == tag)
+    except PublicTag.DoesNotExist:
+        publictag = None
+    if not publictag:
+        newpublictag = PublicTag()
+        newpublictag.generate_key()
+        newpublictag.userkey = userkey
+        newpublictag.tag = tag
+        newpublictag.save()
+
+        message = 'Public link to this tag created'
+        return redirect(url_for('tag_page', userkey=userkey, tag=tag, message=message))
+
+    message = 'Public link already existed'
+    return redirect(url_for('tag_page', userkey=userkey, tag=tag, message=message))
+
+
+@app.route('/<userkey>/<tag>/removepublic/<tagkey>', methods=['GET', 'POST'])
+def removepublictag(userkey, tag, tagkey):
+    q = PublicTag.delete().where(PublicTag.userkey == userkey, PublicTag.tag == tag, PublicTag.tagkey == tagkey)
+    q.execute()
+    message = 'Public link deleted'
+    return redirect(url_for('tag_page', userkey=userkey, tag=tag, message=message))
+
+
+@app.route('/<systemkey>/adduser')
+def adduser(systemkey):
+    """ Add user endpoint, convenience """
+    if systemkey == settings.SYSTEMKEY:
+        newuser = User()
+        newuser.generate_key()
+        newuser.username = 'Nomen Nescio'
+        newuser.save()
+        all_tags[newuser.key] = []
+        return redirect('/{}'.format(newuser.key.decode("utf-8")), code=302)
+    else:
+        abort(404)
+
+
+@app.route('/<systemkey>/refreshfavicons')
+def refreshfavicons(systemkey):
+    """ Add user endpoint, convenience """
+    if systemkey == settings.SYSTEMKEY:
+        bookmarks = Bookmark.select()
+        for bookmark in bookmarks:
+            if bookmark.favicon:
+                try:
+                    filename = os.path.join(MEDIA_ROOT, 'favicons/' + bookmark.favicon)
+                    os.remove(filename)
+                except OSError as e:
+                    print(e)
+            bookmark.set_favicon()
+        return redirect('/')
+    else:
+        abort(404)
+
+
+@app.route('/<systemkey>/findmissingfavicons')
+def findmissingfavicons(systemkey):
+    """ Add user endpoint, convenience """
+    if systemkey == settings.SYSTEMKEY:
+        bookmarks = Bookmark.select()
+        for bookmark in bookmarks:
+            try:
+                if not bookmark.favicon or not os.path.isfile(os.path.join(MEDIA_ROOT, 'favicons/' + bookmark.favicon)):
+                    # This favicon is missing
+                    # Clear favicon, so fallback can be used instead of showing a broken image
+                    bookmark.favicon = None
+                    bookmark.save()
+                    # Try to fetch and save new favicon
+                    bookmark.set_favicon()
+                    bookmark.save()
+            except OSError as e:
+                print(e)
+        return redirect('/')
+    else:
+        abort(404)
+
+
+# Initialisation == create the bookmark, user and public tag tables if they do not exist
+# TODO: move to __init__.py
+Bookmark.create_table(True)
+User.create_table(True)
+PublicTag.create_table(True)
+
+users = User.select()
+print('Current user keys:')
+for user in users:
+    all_tags[user.key] = get_tags_for_user(user.key)
+    usersettings[user.key] = {'theme': user.theme}
+    print(user.key)

digimarks/models.py

@@ -0,0 +1,312 @@
+"""digimarks data models and accompanying convenience functions"""
+import binascii
+import datetime
+import gzip
+import hashlib
+import os
+import shutil
+from urllib.parse import urlparse, urlunparse
+
+import bs4
+import requests
+from peewee import *  # noqa
+
+from . import themes
+
+DATABASE_PATH = os.path.dirname(os.path.realpath(__file__))
+if 'DIGIMARKS_DB_PATH' in os.environ:
+    DATABASE_PATH = os.environ['DIGIMARKS_DB_PATH']
+database = SqliteDatabase(os.path.join(DATABASE_PATH, 'bookmarks.db'))
+
+
+def ifilterfalse(predicate, iterable):
+    # ifilterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8
+    if predicate is None:
+        predicate = bool
+    for x in iterable:
+        if not predicate(x):
+            yield x
+
+
+def unique_everseen(iterable, key=None):
+    "List unique elements, preserving order. Remember all elements ever seen."
+    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
+    # unique_everseen('ABBCcAD', str.lower) --> A B C D
+    seen = set()
+    seen_add = seen.add
+    if key is None:
+        for element in ifilterfalse(seen.__contains__, iterable):
+            seen_add(element)
+            yield element
+    else:
+        for element in iterable:
+            k = key(element)
+            if k not in seen:
+                seen_add(k)
+                yield element
+
+def clean_tags(tags_list):
+    tags_res = [x.strip() for x in tags_list]
+    tags_res = list(unique_everseen(tags_res))
+    tags_res.sort()
+    if tags_res and tags_res[0] == '':
+        del tags_res[0]
+    return tags_res
+
+
+magic_dict = {
+    b"\x1f\x8b\x08": "gz",
+    b"\x42\x5a\x68": "bz2",
+    b"\x50\x4b\x03\x04": "zip"
+    }
+
+max_len = max(len(x) for x in magic_dict)
+
+def file_type(filename):
+    with open(filename, "rb") as f:
+        file_start = f.read(max_len)
+    for magic, filetype in magic_dict.items():
+        if file_start.startswith(magic):
+            return filetype
+    return "no match"
+
+
+class BaseModel(Model):
+    class Meta:
+        database = database
+
+
+class User(BaseModel):
+    """ User account """
+    username = CharField()
+    key = CharField()
+    theme = CharField(default=themes.DEFAULT_THEME)
+    created_date = DateTimeField(default=datetime.datetime.now)
+
+    def generate_key(self):
+        """ Generate userkey """
+        self.key = binascii.hexlify(os.urandom(24))
+        return self.key
+
+
+class Bookmark(BaseModel):
+    """ Bookmark instance, connected to User """
+    # Foreign key to User
+    userkey = CharField()
+
+    title = CharField(default='')
+    url = CharField()
+    note = TextField(default='')
+    #image = CharField(default='')
+    url_hash = CharField(default='')
+    tags = CharField(default='')
+    starred = BooleanField(default=False)
+
+    # Website (domain) favicon
+    favicon = CharField(null=True)
+
+    # Status code: 200 is OK, 404 is not found, for example (showing an error)
+    HTTP_CONNECTIONERROR = 0
+    HTTP_OK = 200
+    HTTP_ACCEPTED = 202
+    HTTP_MOVEDTEMPORARILY = 304
+    HTTP_NOTFOUND = 404
+
+    http_status = IntegerField(default=200)
+    redirect_uri = None
+
+    created_date = DateTimeField(default=datetime.datetime.now)
+    modified_date = DateTimeField(null=True)
+    deleted_date = DateTimeField(null=True)
+
+    # Bookmark status; deleting doesn't remove from DB
+    VISIBLE = 0
+    DELETED = 1
+    status = IntegerField(default=VISIBLE)
+
+
+    class Meta:
+        ordering = (('created_date', 'desc'),)
+
+    def set_hash(self):
+        """ Generate hash """
+        self.url_hash = hashlib.md5(self.url.encode('utf-8')).hexdigest()
+
+    def set_title_from_source(self):
+        """ Request the title by requesting the source url """
+        try:
+            result = requests.get(self.url, headers={'User-Agent': DIGIMARKS_USER_AGENT})
+            self.http_status = result.status_code
+        except:
+            # For example 'MissingSchema: Invalid URL 'abc': No schema supplied. Perhaps you meant http://abc?'
+            self.http_status = 404
+        if self.http_status == 200 or self.http_status == 202:
+            html = bs4.BeautifulSoup(result.text, 'html.parser')
+            try:
+                self.title = html.title.text.strip()
+            except AttributeError:
+                self.title = ''
+        return self.title
+
+    def set_status_code(self):
+        """ Check the HTTP status of the url, as it might not exist for example """
+        try:
+            result = requests.head(self.url, headers={'User-Agent': DIGIMARKS_USER_AGENT})
+            self.http_status = result.status_code
+        except requests.ConnectionError:
+            self.http_status = self.HTTP_CONNECTIONERROR
+        return self.http_status
+
+    def _set_favicon_with_iconsbetterideaorg(self, domain):
+        """ Fetch favicon for the domain """
+        fileextension = '.png'
+        meta = requests.head(
+            'http://icons.better-idea.org/icon?size=60&url=' + domain,
+            allow_redirects=True,
+            headers={'User-Agent': DIGIMARKS_USER_AGENT}
+        )
+        if meta.url[-3:].lower() == 'ico':
+            fileextension = '.ico'
+        response = requests.get(
+            'http://icons.better-idea.org/icon?size=60&url=' + domain,
+            stream=True,
+            headers={'User-Agent': DIGIMARKS_USER_AGENT}
+        )
+        filename = os.path.join(MEDIA_ROOT, 'favicons/' + domain + fileextension)
+        with open(filename, 'wb') as out_file:
+            shutil.copyfileobj(response.raw, out_file)
+        del response
+        filetype = file_type(filename)
+        if filetype == 'gz':
+            # decompress
+            orig = gzip.GzipFile(filename, 'rb')
+            origcontent = orig.read()
+            orig.close()
+            os.remove(filename)
+            with open(filename, 'wb') as new:
+                new.write(origcontent)
+        self.favicon = domain + fileextension
+
+    def _set_favicon_with_realfavicongenerator(self, domain):
+        """ Fetch favicon for the domain """
+        response = requests.get(
+            'https://realfavicongenerator.p.rapidapi.com/favicon/icon?platform=android_chrome&site=' + domain,
+            stream=True,
+            headers={'User-Agent': DIGIMARKS_USER_AGENT, 'X-Mashape-Key': MASHAPE_API_KEY}
+        )
+        if response.status_code == 404:
+            # Fall back to desktop favicon
+            response = requests.get(
+                'https://realfavicongenerator.p.rapidapi.com/favicon/icon?platform=desktop&site=' + domain,
+                stream=True,
+                headers={'User-Agent': DIGIMARKS_USER_AGENT, 'X-Mashape-Key': settings.MASHAPE_API_KEY}
+            )
+        # Debug for the moment
+        print(domain)
+        print(response.headers)
+        if 'Content-Length' in response.headers and response.headers['Content-Length'] == '0':
+            # No favicon found, likely
+            print('Skipping this favicon, needs fallback')
+            return
+        # Default to 'image/png'
+        fileextension = '.png'
+        if response.headers['content-type'] == 'image/jpeg':
+            fileextension = '.jpg'
+        if response.headers['content-type'] == 'image/x-icon':
+            fileextension = '.ico'
+        filename = os.path.join(MEDIA_ROOT, 'favicons/' + domain + fileextension)
+        with open(filename, 'wb') as out_file:
+            shutil.copyfileobj(response.raw, out_file)
+        del response
+        filetype = file_type(filename)
+        if filetype == 'gz':
+            # decompress
+            orig = gzip.GzipFile(filename, 'rb')
+            origcontent = orig.read()
+            orig.close()
+            os.remove(filename)
+            with open(filename, 'wb') as new:
+                new.write(origcontent)
+        self.favicon = domain + fileextension
+
+    def set_favicon(self):
+        """ Fetch favicon for the domain """
+        u = urlparse(self.url)
+        domain = u.netloc
+        if os.path.isfile(os.path.join(MEDIA_ROOT, 'favicons/' + domain + '.png')):
+            # If file exists, don't re-download it
+            self.favicon = domain + '.png'
+            return
+        if os.path.isfile(os.path.join(MEDIA_ROOT, 'favicons/' + domain + '.ico')):
+            # If file exists, don't re-download it
+            self.favicon = domain + '.ico'
+            return
+        #self._set_favicon_with_iconsbetterideaorg(domain)
+        self._set_favicon_with_realfavicongenerator(domain)
+
+    def set_tags(self, newtags):
+        """ Set tags from `tags`, strip and sort them """
+        tags_split = newtags.split(',')
+        tags_clean = clean_tags(tags_split)
+        self.tags = ','.join(tags_clean)
+
+    def get_redirect_uri(self):
+        if self.redirect_uri:
+            return self.redirect_uri
+        if self.http_status == 301 or self.http_status == 302:
+            result = requests.head(self.url, allow_redirects=True, headers={'User-Agent': DIGIMARKS_USER_AGENT})
+            self.http_status = result.status_code
+            self.redirect_uri = result.url
+            return result.url
+        return None
+
+    def get_uri_domain(self):
+        parsed = urlparse(self.url)
+        return parsed.hostname
+
+    @classmethod
+    def strip_url_params(cls, url):
+        parsed = urlparse(url)
+        return urlunparse((parsed.scheme, parsed.netloc, parsed.path, parsed.params, '', parsed.fragment))
+
+    @property
+    def tags_list(self):
+        """ Get the tags as a list, iterable in template """
+        if self.tags:
+            return self.tags.split(',')
+        return []
+
+    def to_dict(self):
+        result = {
+            'title': self.title,
+            'url': self.url,
+            'created':  self.created_date.strftime('%Y-%m-%d %H:%M:%S'),
+            'url_hash': self.url_hash,
+            'tags': self.tags,
+        }
+        return result
+
+    @property
+    def serialize(self):
+        return self.to_dict()
+
+
+class PublicTag(BaseModel):
+    """ Publicly shared tag """
+    tagkey = CharField()
+    userkey = CharField()
+    tag = CharField()
+    created_date = DateTimeField(default=datetime.datetime.now)
+
+    def generate_key(self):
+        """ Generate hash-based key for publicly shared tag """
+        self.tagkey = binascii.hexlify(os.urandom(16))
+
+
+def get_tags_for_user(userkey):
+    """ Extract all tags from the bookmarks """
+    bookmarks = Bookmark.select().filter(Bookmark.userkey == userkey, Bookmark.status == Bookmark.VISIBLE)
+    tags = []
+    for bookmark in bookmarks:
+        tags += bookmark.tags_list
+    return clean_tags(tags)

digimarks/static/css/digimarks.css

@@ -0,0 +1,67 @@
+/**
+ * digimarks styling
+ */
+
+/** Navigation **/
+
+nav .sidenav-trigger
+{
+    /* Fix for misalignment of hamburger icon */
+    margin: 0;
+}
+
+nav .sidenav-trigger i
+{
+    /* Make the hamburger icon great again */
+    font-size: 2.7rem;
+}
+
+/** Cards and tags **/
+
+.card .card-content,
+.card .card-reveal
+{
+    padding: 12px;
+}
+
+.card.tiny
+{
+    height: 140px;
+    overflow: hidden;
+}
+
+.card.tiny .card-title
+{
+    font-size: 18px;
+}
+
+.card .card-reveal .digimark-card-header,
+.card .digimark-card-header.activator,
+.chip.clickable
+{
+    cursor: pointer;
+    /*display: block;*/
+}
+
+.card .digimark-card-header-tags
+{
+    padding-top: 10px;
+}
+
+.card-image
+{
+    min-width: 60px;
+}
+
+.card-image i,
+.list-image i
+{
+    padding: 5px 0 0 15px;
+}
+
+.card.horizontal .card-image img.favicon,
+.list-image img.favicon
+{
+    height: 60px;
+    width: 60px;
+}

digimarks/static/js/init.js

@@ -0,0 +1,11 @@
+/* global M */
+
+var options = {};
+var elem = document.querySelector(".sidenav");
+var instance = M.Sidenav.init(elem, options);
+
+elem = document.querySelector(".collapsible");
+instance = M.Collapsible.init(elem, {
+    // inDuration: 1000,
+    // outDuration: 1000
+});

digimarks/templates/base.html

@@ -0,0 +1,126 @@
+<!doctype html>
+<html>
+    <head>
+        <title>{% block title %}{% endblock %} - digimarks</title>
+
+        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
+        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0"/>
+        <link rel="shortcut icon" href="{{ url_for('static', filename='favicon.ico') }}"/>
+
+        <!-- Chrome, Firefox OS and Opera -->
+        <meta name="theme-color" content="{{ theme.BROWSERCHROME }}" />
+        <!-- Windows Phone -->
+        <meta name="msapplication-navbutton-color" content="{{ theme.BROWSERCHROME }}">
+        <!-- iOS Safari -->
+        <meta name="apple-mobile-web-app-capable" content="yes">
+        <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
+
+        <meta name="referrer" content="never">
+        <meta name="referrer" content="no-referrer">
+
+        <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
+        <link href='https://fonts.googleapis.com/css?family=Roboto+Mono&subset=latin,latin-ext' rel='stylesheet' type='text/css'>
+        <link href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css" type="text/css" rel="stylesheet" media="screen,projection"/>
+        <style>
+            a
+            {
+                color: {{ theme.LINK_TEXT }};
+            }
+            .card-content a
+            {
+                color: {{ theme.CARD_LINK }};
+            }
+            .chip a
+            {
+                color: {{ theme.CHIP_TEXT }};
+            }
+            /* label color */
+            .input-field .prefix ~ input, .input-field .prefix ~ textarea, .input-field .prefix ~ label, .input-field .prefix ~ .validate ~ label, .input-field .prefix ~ .autocomplete-content, .input-field input[type=text]
+            {
+                color: {{ theme.TEXTHEX }};
+            }
+            /* label focus color */
+            .input-field input[type=text]:focus + label,
+            .input-field .prefix ~ input[type=text]:focus + label
+            {
+                color: {{ theme.BUTTON }};
+            }
+            /* label underline focus color */
+            .input-field input[type=text]:focus,
+            .input-field .prefix ~ input[type=text]:focus,
+            .input-field input[type=text].autocomplete:focus
+            {
+                border-bottom: 1px solid {{ theme.BUTTON }};
+                box-shadow: 0 1px 0 0 {{ theme.BUTTON }};
+            }
+            /* icon prefix focus color */
+            .input-field .prefix.active
+            {
+                color: {{ theme.BUTTON }};
+            }
+            .btn, .btn:visited
+            {
+                background-color: {{ theme.BUTTON }};
+            }
+            .btn:hover, .btn:active
+            {
+                background-color: {{ theme.BUTTON_ACTIVE }};
+            }
+            .deletebtn
+            {
+                background-color: red;
+            }
+            .deletebtn:hover
+            {
+                background-color: #ef5350; /* red lighten-1 */
+            }
+        </style>
+        <link href="{{ url_for('static', filename='css/digimarks.css') }}?20180330" type="text/css" rel="stylesheet" media="screen,projection"/>
+{% if not sortmethod %}
+    {% set sortmethod = None %}
+{% endif %}
+{% if not show_as %}
+    {% set show_as = None %}
+{% endif %}
+    </head>
+    <body class="{{ theme.BODY }} {{ theme.TEXT }}">
+        <nav class="{{ theme.NAV }}" role="navigation">
+            <div class="nav-wrapper container"><a id="logo-container" href="{% if userkey %}{{ url_for('bookmarks_page', userkey=userkey, sortmethod=sortmethod, show_as=show_as) }}{% else %}{{ url_for('index') }}{% endif %}" class="brand-logo">digimarks</a>
+                <ul class="right hide-on-med-and-down">
+                    {% if userkey %}
+                    <li><a href="{{ url_for('tags_page', userkey=userkey) }}" class="waves-effect waves-light btn"><i class="material-icons left">label</i>Tags</a></li>
+                    <li><a href="{{ url_for('addbookmark', userkey=userkey) }}" class="waves-effect waves-light btn"><i class="material-icons left">add</i>Add bookmark</a></li>
+                    {% endif %}
+                </ul>
+
+                {% if userkey %}
+                <ul id="nav-mobile" class="sidenav">
+                    <li><a class="waves-effect" href="{{ url_for('bookmarks_page', userkey=userkey) }}"><i class="material-icons left">turned_in</i>Home</a></li>
+                    <li><a class="waves-effect" href="{{ url_for('tags_page', userkey=userkey) }}"><i class="material-icons left">label</i>Tags</a></li>
+                    <li><a class="waves-effect" href="{{ url_for('addbookmark', userkey=userkey) }}"><i class="material-icons left">add</i>Add bookmark</a></li>
+                </ul>
+                <a href="#" data-target="nav-mobile" class="sidenav-trigger"><i class="material-icons">menu</i></a>
+                {% endif %}
+            </div>
+        </nav>
+        <div class="section no-pad-bot" id="index-banner">
+            <div class="container">
+                <div class="header {{ theme.PAGEHEADER }}">
+                    <h1>{% block pageheader %}Bookmarks{% endblock %}</h1>
+                </div>
+            </div>
+        </div>
+        <div class="container">
+            <div class="section">
+                {% block pagecontent %}
+                {% endblock %}
+            </div>
+        </div>
+
+        <!-- Scripts -->
+        <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
+        <script src="{{ url_for('static', filename='js/init.js') }}?20180309"></script>
+
+        {% block extrajs %}{% endblock %}
+    </body>
+</html>

digimarks/templates/bookmarks.html

@@ -0,0 +1,113 @@
+{% extends "base.html" %}
+{% if not action %}
+{% set action = 'Bookmarks' %}
+{% endif %}
+{% block title %}{{ action }}{% endblock %}
+{% block pageheader %}{{ action }}{% endblock %}
+{% block pagecontent %}
+
+{% if tag and not publictag %}
+<div class="row">
+    <div class="col s12">
+        <a href="{{ url_for('addpublictag', userkey=userkey, tag=tag) }}">Create public page <i class="material-icons right">tag</i></a>
+    </div>
+</div>
+{% endif %}
+
+{% if tag and publictag %}
+<div class="row">
+    <div class="col s12"><a href="{{ url_for('publictag_page', tagkey=publictag.tagkey) }}">Public link</a></div>
+</div>
+{% endif %}
+
+{% if message %}
+<div class="row">
+    <div class="col s12">
+        <div class="card-panel {{ theme.MESSAGE_BACKGROUND }}">
+            <span class="{{ theme.MESSAGE_TEXT }}">
+                {{ message|safe }}
+            </span>
+        </div>
+    </div>
+</div>
+{% endif %}
+
+<div class="row">
+    <form action="{{ url_for('bookmarks_page', userkey=userkey) }}" name="filterForm" method="POST" autocomplete="off">
+        <div class="input-field col l9 m9 s8">
+            <input placeholder="search text" type="text" name="filter_text" id="filter_text" class="autocomplete" value="{{ filter_text }}" autocomplete="false" />
+        </div>
+
+        <div class="input-field col l3 m3 s4">
+            <p class="right-align"><button class="btn waves-effect waves-light" type="submit" name="submitBtn" title="Find"><i class="material-icons">search</i></button>
+            {% if show_as and show_as == 'list' %}
+                <a href="{{ url_for('bookmarks_page', userkey=userkey, filtermethod=filtermethod, sortmethod=sortmethod, show_as=None) }}" class="waves-effect waves-light btn" title="Show as cards"><i class="material-icons">apps</i></a>
+            {% else %}
+                <a href="{{ url_for('bookmarks_page', userkey=userkey, filtermethod=filtermethod, sortmethod=sortmethod, show_as='list') }}" class="waves-effect waves-light btn" title="Show as list"><i class="material-icons">reorder</i></a>
+            {% endif %}
+            </p>
+        </div>
+    </form>
+</div>
+
+{% if tags %}
+<div class="row">
+    <div class="col s12">
+        <ul class="collapsible" data-collapsible="expandable">
+            <li>
+              <div class="collapsible-header"><i class="material-icons">label</i>Filter on star/problem/comment/tag</div>
+              <div class="collapsible-body" style="padding: 10px;">
+                <div class="chip">
+                    <a href="{{ url_for('bookmarks_page', userkey=userkey, filtermethod='starred') }}"><i class="tiny material-icons {{ theme.STAR }}">star</i></a>
+                </div>
+                <div class="chip">
+                    <a href="{{ url_for('bookmarks_page', userkey=userkey, filtermethod='broken') }}"><i class="tiny material-icons {{ theme.PROBLEM }}">report_problem</i></a>
+                </div>
+                <div class="chip">
+                    <a href="{{ url_for('bookmarks_page', userkey=userkey, filtermethod='note') }}"><i class="tiny material-icons {{ theme.COMMENT }}">comment</i></a>
+                </div>
+                {% for tag in tags %}
+                <div class="chip">
+                    <a href="{{ url_for('tag_page', userkey=userkey, tag=tag) }}">{{ tag }}</a>
+                </div>
+                {% endfor %}
+            </li>
+        </ul>
+    </div>
+</div>
+{% endif %}
+
+{% if show_as and show_as == 'list' %}
+    {% include 'list.html' %}
+{% else %}
+    {% include 'cards.html' %}
+{% endif %}
+
+<div class="fixed-action-btn" style="bottom: 20px; right: 20px;">
+    <a class="btn-floating btn-large {{ theme.FAB }}" href="{{ url_for('addbookmark', userkey=userkey) }}">
+      <i class="large material-icons">add</i>
+    </a>
+</div>
+{% endblock %}
+{% block extrajs %}
+<script>
+    function submitFilter() {
+        document.filterForm.submit();
+    }
+    /* Search filter autocomplete */
+    var options = {
+        onAutocomplete: submitFilter,
+        minLength: 3,
+        limit: 10,
+        data: {
+        },
+    }
+    var elem = document.querySelector('.autocomplete');
+    var instance = M.Autocomplete.init(elem, options);
+    /* TODO: fetch from API
+    instance.updateData({
+    });
+    */
+</script>
+<script src="{{ url_for('bookmarks_js', userkey=userkey) }}" ></script>
+{% endblock %}

digimarks/templates/bookmarks.js

@@ -0,0 +1,11 @@
+var elem = document.querySelector('.autocomplete');
+var instance = M.Autocomplete.getInstance(elem);
+instance.updateData({
+    {% for bookmark in bookmarks %}
+        {% if bookmark.favicon %}
+            "{{ bookmark.title | replace('"', '\\"') | replace('\n', '') | replace('\r', '') }}": "{{ url_for('static', filename='favicons/' + bookmark.favicon) }}",
+        {% else %}
+            "{{ bookmark.title | replace('"', '\\"') | replace('\n', '') | replace('\r', '') }}": null,
+        {% endif %}
+    {% endfor %}
+});

digimarks/templates/cards.html

@@ -0,0 +1,65 @@
+<div class="row">
+    {% for bookmark in bookmarks %}
+    <div class="col s12 m6 l4">
+        <div class="card horizontal tiny {{ theme.CARD_BACKGROUND }}">
+            <div class="card-image">
+                {% if bookmark.favicon %}
+                    <div><img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" class="favicon" /></div>
+                {% else %}
+                    <div><img src="{{ url_for('static', filename='faviconfallback.png') }}" class="favicon" /></div>
+                {% endif %}
+                {% if bookmark.http_status != 200 and bookmark.http_status != 304 %}
+                    <div><i class="small material-icons {{ theme.PROBLEM }}" title="HTTP status {{ bookmark.http_status }}">report_problem</i></div>
+                {% endif %}
+                {% if bookmark.starred == True %}
+                    <div><i class="small material-icons {{ theme.STAR }}">star</i></div>
+                {% endif %}
+                {% if bookmark.note %}
+                    <div><i class="small material-icons {{ theme.CARD_TEXT }}" title="{{ bookmark.note|truncate(100) }}">comment</i></div>
+                {% endif %}
+            </div>
+            <div class="card-stacked">
+                <div class="card-content {{ theme.CARD_TEXT }}">
+                    <span class="digimark-card-header activator">
+                        <i class="material-icons right">more_vert</i>
+                    </span>
+                    <div class="digimark-card-content">
+                        <a href="{{ bookmark.url }}" title="{{ bookmark.url }}" rel="noreferrer noopener" target="_blank">
+                            {% if bookmark.title %}
+                                {{ bookmark.title }}
+                            {% else %}
+                                {{ bookmark.get_uri_domain() }} (no title)
+                            {% endif %}
+                        </a>
+                    </div>
+                </div>
+            </div>
+            <div class="card-reveal {{ theme.CARD_BACKGROUND }}">
+                <span class="card-title {{ theme.CARD_TEXT }}">Added @ {{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}<i class="material-icons right">close</i></span>
+                {% if editable %}
+                <div class="{{ theme.CARD_TEXT }}" style="padding-top: 10px;">
+                <a href="{{ url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" style="padding: 3px"><i class="tiny material-icons">mode_edit</i> EDIT</a>
+                <a href="{{ url_for('deletingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" style="padding: 3px" class="red-text"><i class="tiny material-icons">delete</i> DELETE</a>
+                </div>
+                {% endif %}
+                {% if showtags %}
+                <div class="digimark-card-header-tags">
+                {% for tag in bookmark.tags_list %}
+                    <div class="chip">
+                        <a href="{{ url_for('tag_page', userkey=userkey, tag=tag) }}">{{ tag }}</a>
+                    </div>
+                {% endfor %}
+                </div>
+                {% endif %}
+            </div>
+        </div>
+    </div>
+    {% endfor %}
+
+    {#
+    <div class="pagination">
+        {% if page > 1 %}<a href="./?page={{ page - 1 }}">Previous</a>{% endif %}
+        {% if pagination.get_pages() > page %}<a href="./?page={{ page + 1 }}">Next</a>{% endif %}
+    </div>
+    #}
+</div>

digimarks/templates/edit.html

@@ -0,0 +1,175 @@
+{% extends "base.html" %}
+{% block title %}{{ action }}{% endblock %}
+{% block pageheader %}{{ action }}{% endblock %}
+{% block pagecontent %}
+
+{% if bookmark.http_status != 200 and bookmark.http_status != 202 and bookmark.http_status != 304 %}
+<div class="row">
+    <div class="col s12">
+        <div class="card-panel {{ theme.ERRORMESSAGE_BACKGROUND }}">
+            <span class="{{ theme.ERRORMESSAGE_TEXT }}">
+                {% if bookmark.http_status == 404 %}
+                <i class="material-icons">report_problem</i>&nbsp;&nbsp;URL not found (404), broken/outdated link?
+                {% elif bookmark.http_status == 301 %}
+                <i class="material-icons">report_problem</i>&nbsp;&nbsp;HTTP status (301), moved permanently. Use button for new target
+                {% elif bookmark.http_status == 302 %}
+                <i class="material-icons">report_problem</i>&nbsp;&nbsp;HTTP status (302), moved temporarily. Use button for new target
+                {% elif bookmark.http_status == bookmark.HTTP_CONNECTIONERROR %}
+                <i class="material-icons">report_problem</i>&nbsp;&nbsp;Connection error, server might have been offline at the time of last edit
+                {% else %}
+                <i class="material-icons">report_problem</i>&nbsp;&nbsp;HTTP status {{ bookmark.http_status }}
+                {% endif %}
+            </span>
+        </div>
+    </div>
+    </div>
+{% endif %}
+
+{% if message %}
+<div class="row">
+    <div class="col s12">
+        <div class="card-panel {{ theme.MESSAGE_BACKGROUND }}">
+            <span class="{{ theme.MESSAGE_TEXT }}">
+                {{ message }}
+            </span>
+        </div>
+    </div>
+</div>
+{% endif %}
+
+{% if formaction and formaction == 'edit' %}
+<form class="digimark" id="digimark" action="{{ url_for('editingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" method="POST" onsubmit="return onSubmitForm();" autocomplete="off">
+{% else %}
+<form class="digimark" id="digimark" action="{{ url_for('addingbookmark', userkey=userkey) }}" method="POST" onsubmit="return onSubmitForm();" autocomplete="off">
+{% endif %}
+
+    <div class="row">
+        <div class="input-field col s12">
+            <i class="material-icons prefix">description</i>
+            <input placeholder="title (leave empty for autofetch)" type="text" name="title" id="title" value="{{ bookmark.title }}" autocomplete="false" />
+            <label for="title">Title</label>
+{#            <span class="helper-text">Leave title empty for autofetching from the page</span>#}
+        </div>
+
+        <div class="input-field col s12">
+            <i class="material-icons prefix">turned_in</i>
+            <input placeholder="url" type="text" name="url" id="url" value="{{ bookmark.url }}" autocomplete="false" />
+            <label for="url">URL</label>
+            {% if bookmark.get_redirect_uri() %}
+            <div>
+                <a class="waves-effect waves-light btn" id="btn_urlupdate" onclick="updateURL()"><i class="material-icons left">turned_in</i>{{ bookmark.get_redirect_uri() }}</a>
+            </div>
+            <script type="text/javascript">
+            function updateURL() {
+                var text = document.getElementById('url');
+                text.value = '{{ bookmark.get_redirect_uri() }}';
+            }
+            </script>
+            {% endif %}
+        </div>
+
+        <div class="input-field col s12">
+            <i class="material-icons prefix">comment</i>
+            <input placeholder="note" type="text" name="note" id="note" value="{{ bookmark.note }}" autocomplete="false" />
+            <label for="note">Note</label>
+        </div>
+
+        <div class="input-field col s12">
+            <i class="material-icons prefix">label</i>
+            <input placeholder="tags, divided by comma's" type="text" name="tags" id="tags" value="{{ bookmark.tags }}" autocomplete="false" />
+            <label for="tags">Tags</label>
+        </div>
+    </div>
+    {% if tags %}
+    <div class="row">
+        <div class="col s12">
+            <ul class="collapsible" data-collapsible="expandable">
+                <li>
+                  <div class="collapsible-header"><i class="material-icons">label</i>Existing tags</div>
+                  <div class="collapsible-body" style="padding: 10px;">
+                    {% for tag in tags %}
+                    <div class="chip clickable" id="chip_{{ tag }}" onclick="addTag('{{ tag }}');">
+                        {{ tag }}
+                    </div>
+                    {% endfor %}
+                  </div>
+                </li>
+            </ul>
+        </div>
+    </div>
+    {% endif %}
+    <div class="row">
+
+        <div class="col s12">
+            {#<i class="material-icons prefix">star</i>#}
+            <label>
+                <input type="checkbox" name="starred" id="starred" {% if bookmark.starred == True %}checked{% endif %} />
+                <span>Starred</span>
+            </label>
+        </div>
+
+        <div class="col s12">
+            <label>
+                <input type="checkbox" name="strip" id="strip" />
+                <span>Strip parameters from url (like <em>?utm_source=social</em> - can break the link!)</span>
+            </label>
+        </div>
+
+    {% if bookmark.url_hash %}
+    </div>
+    <div class="row">
+        <div class="col l4 m6 s12">
+            <table>
+                <tr>
+                    <th>Added</th>
+                    <td>{{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}</td>
+                </tr>
+                {% if bookmark.modified_date %}
+                <tr>
+                    <th>Modified</th>
+                    <td>{{ bookmark.modified_date.strftime('%Y-%m-%d %H:%M') }}</td>
+                </tr>
+                {% endif %}
+                {% if bookmark.deleted_date %}
+                <tr>
+                    <th>Deleted</th>
+                    <td>{{ bookmark.deleted_date.strftime('%Y-%m-%d %H:%M') }}</td>
+                </tr>
+                {% endif %}
+            </table>
+        </div>
+    </div>
+    <div class="row">
+    {% endif %}
+
+        <div class="input-field col l2 m3 s4">
+            <p class="left-align"><button class="btn waves-effect waves-light" type="submit" name="submit" id="submit">Save <i class="material-icons right">send</i></button></p>
+        </div>
+    {% if bookmark.url_hash %}
+</form>
+        <div class="input-field col l4 m4 s6">
+        <form action="{{ url_for('deletingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" method="POST">
+            <p class="left-align"><button class="btn waves-effect waves-light deletebtn" type="submit" name="delete">Delete <i class="material-icons right">delete</i></button></p>
+        </form>
+        </div>
+    </div>
+    {% else %}
+    </div>
+</form>
+    {% endif %}
+
+<script>
+function onSubmitForm()
+{
+    var theForm = document.getElementById('digimark');
+    var submitButton = document.getElementById('submit');
+    theForm.onsubmit = submitButton.setAttribute("disabled", true);
+    return true;
+}
+function addTag(tagText)
+{
+    var text = document.getElementById('tags');
+    text.value = text.value + ', ' + tagText;
+}
+</script>
+{% endblock %}

digimarks/templates/list.html

@@ -0,0 +1,62 @@
+<div class="row">
+    <table>
+        <thead>
+            <tr>
+                <th>&nbsp;</th>
+                <th>Bookmark</th>
+                <th>Added</th>
+                {% if showtags %}
+                <th>Tags</th>
+                {% endif %}
+                <th>&nbsp;</th>
+            </tr>
+        </thead>
+        <tbody>
+        {% for bookmark in bookmarks %}
+            <tr>
+                <td class="list-image">
+                {% if bookmark.favicon %}
+                    <img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" class="favicon" />
+                {% else %}
+                    <img src="{{ url_for('static', filename='faviconfallback.png') }}" class="favicon" />
+                {% endif %}
+                {% if bookmark.http_status != 200 and bookmark.http_status != 304 %}
+                    <i class="small material-icons {{ theme.PROBLEM }}" title="HTTP status {{ bookmark.http_status }}">report_problem</i>
+                {% endif %}
+                {% if bookmark.starred == True %}
+                    <i class="small material-icons {{ theme.STAR }}">star</i>
+                {% endif %}
+                {% if bookmark.note %}
+                    <i class="small material-icons {{ theme.CARD_TEXT }}" title="{{ bookmark.note|truncate(100) }}">comment</i>
+                {% endif %}
+                </td>
+                <td>
+                    <a href="{{ bookmark.url }}" title="{{ bookmark.url }}" rel="noreferrer noopener" target="_blank">
+                        {% if bookmark.title %}
+                            {{ bookmark.title }}
+                        {% else %}
+                            {{ bookmark.get_uri_domain() }} (no title)
+                        {% endif %}
+                    </a>
+                </td>
+                <td>{{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}</td>
+                {% if showtags %}
+                <td>
+                {% for tag in bookmark.tags_list %}
+                    <div class="chip">
+                        <a href="{{ url_for('tag_page', userkey=userkey, tag=tag) }}">{{ tag }}</a>
+                    </div>
+                {% endfor %}
+                </td>
+                {% endif %}
+                <td>
+                    {% if editable %}
+                    <a href="{{ url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" class="waves-effect waves-light btn" title="Edit"><i class="tiny material-icons">mode_edit</i></a>
+                    <a href="{{ url_for('deletingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" class="waves-effect waves-light btn red" title="DELETE"><i class="tiny material-icons">delete</i></a>
+                    {% endif %}
+                </td>
+            </tr>
+        {% endfor %}
+        </tbody>
+    </table>
+</div>

digimarks/templates/publicbookmarks.html

@@ -0,0 +1,29 @@
+{% extends "base.html" %}
+{% if not action %}
+{% set action = 'Bookmarks' %}
+{% endif %}
+{% block title %}{{ action }}{% endblock %}
+{% block pageheader %}{{ action }}{% endblock %}
+{% block pagecontent %}
+
+{% if message %}
+<div class="row">
+    <div class="col s12">
+        <div class="card-panel orange lighten-2">
+            <span class="white-text">
+                {{ message }}
+            </span>
+        </div>
+    </div>
+</div>
+{% endif %}
+
+<div class="row">
+    <div class="col s12">
+        <a href="{{ url_for('publictag_feed', tagkey=tagkey) }}"><i class="material-icons tiny">rss_feed</i> feed</a>
+    </div>
+</div>
+
+{% include 'cards.html' %}
+
+{% endblock %}

digimarks/templates/redirect.html

@@ -0,0 +1,16 @@
+<!doctype html>
+<html>
+    <head>
+        <title>Redirecting - digimarks</title>
+        <meta name="referrer" content="never">
+        <meta name="robots" content="noindex, nofollow">
+        <meta http-equiv=refresh content="3; URL={{ url }}">
+        <style>
+            body { background-color: #000; color: #FFF; }
+            a { color: #fb8c00; }
+        </style>
+    </head>
+    <body>
+        <p>You're being redirected. If nothing happens, <a href="{{ url }}">click here instead</a>.</p>
+    </body>
+</html>

digimarks/templates/tags.html

@@ -0,0 +1,65 @@
+{% extends "base.html" %}
+{% block title %}Tags{% endblock %}
+{% block pageheader %}Tags{% endblock %}
+{% block pagecontent %}
+<div class="row">
+
+    <div class="col s12">
+    <table class="centered">
+        <thead>
+            <tr>
+                <th><i class="material-icons" title="Unique labels">label</i></th>
+                <th><i class="material-icons green-text" title="Public tag pages">present_to_all</i></th>
+                <th><i class="material-icons" title="Total bookmarks">turned_in</i></th>
+                <th><i class="material-icons" title="Bookmarks with notes">comment</i></th>
+                <th><i class="material-icons yellow-text" title="Starred bookmarks">star</i></th>
+                <th><i class="material-icons orange-text" title="HTTP status is not 200 OK">warning</i></th>
+                <th><i class="material-icons red-text" title="Deleted bookmarks">delete</i></th>
+            </tr>
+        </thead>
+        <tbody>
+            <tr>
+                <td>{{ totaltags }}</td>
+                <td>{{ totalpublic }}</td>
+                <td>{{ totalbookmarks }}</td>
+                <td>{{ totalnotes }}</td>
+                <td>{{ totalstarred }}</td>
+                <td>{{ totalhttperrorstatus }}</td>
+                <td>{{ totaldeleted }}</td>
+            </tr>
+        </tbody>
+    </table>
+
+    <br /><br />
+
+    <table>
+        <thead>
+            <tr>
+                <th>Tag</th>
+                <th>Public link</th>
+                <th>Number of bookmarks</th>
+            </tr>
+        </thead>
+        <tbody>
+            {% for tag in tags %}
+            <tr>
+                <td>
+                    <a href="{{ url_for('tag_page', userkey=userkey, tag=tag['tag']) }}">{{ tag['tag'] }}</a>
+                </td>
+                <td>
+                    {% if tag['publictag'] %}
+                        <a href="{{ url_for('publictag_page', tagkey=tag['publictag'].tagkey) }}">Public link</a> (<a href="{{ url_for('removepublictag', tag=tag['tag'], tagkey=tag['publictag'].tagkey, userkey=userkey) }}">Delete</a> <i class="tiny material-icons red-text">warning</i>)
+                    {% else %}
+                        <a href="{{ url_for('addpublictag', userkey=userkey, tag=tag['tag']) }}">Create</a>
+                    {% endif %}
+                </td>
+                <td>
+                    {{ tag['total'] }}
+                </td>
+            </tr>
+            {% endfor %}
+        </tbody>
+    </table>
+    </div>
+</div>
+{% endblock %}

digimarks/themes.py

@@ -0,0 +1,125 @@
+"""digimarks theme definitions"""
+
+DEFAULT_THEME = 'freshgreen'
+themes = {
+    'green': {
+        'BROWSERCHROME': '#2e7d32',  # green darken-2
+        'BODY': 'grey lighten-4',
+        'TEXT': 'black-text',
+        'TEXTHEX': '#000',
+        'NAV': 'green darken-3',
+        'PAGEHEADER': 'grey-text lighten-5',
+        'MESSAGE_BACKGROUND': 'orange lighten-2',
+        'MESSAGE_TEXT': 'white-text',
+        'ERRORMESSAGE_BACKGROUND': 'red darken-1',
+        'ERRORMESSAGE_TEXT': 'white-text',
+        'BUTTON': '#1b5e20', # green darken-4
+        'BUTTON_ACTIVE': '#43a047',  # green darken-1
+        'LINK_TEXT': '#1b5e20',  # green darken-4
+        'CARD_BACKGROUND': 'green darken-3',
+        'CARD_TEXT': 'white-text',
+        'CARD_LINK': '#FFF',  # white-text
+        'CHIP_TEXT': '#1b5e20',  # green darken-4
+        'FAB': 'red',
+
+        'STAR': 'yellow-text',
+        'PROBLEM': 'red-text',
+        'COMMENT': '',
+    },
+    'freshgreen': {
+        'BROWSERCHROME': '#43a047',  # green darken-1
+        'BODY': 'grey lighten-5',
+        'TEXT': 'black-text',
+        'TEXTHEX': '#000',
+        'NAV': 'green darken-1',
+        'PAGEHEADER': 'grey-text lighten-5',
+        'MESSAGE_BACKGROUND': 'orange lighten-2',
+        'MESSAGE_TEXT': 'white-text',
+        'ERRORMESSAGE_BACKGROUND': 'red darken-1',
+        'ERRORMESSAGE_TEXT': 'white-text',
+        'BUTTON': '#1b5e20', # green darken-4
+        'BUTTON_ACTIVE': '#43a047',  # green darken-1
+        'LINK_TEXT': '#1b5e20',  # green darken-4
+        'CARD_BACKGROUND': 'green darken-1',
+        'CARD_TEXT': 'white-text',
+        'CARD_LINK': '#FFF',  # white-text
+        'CHIP_TEXT': '#1b5e20',  # green darken-4
+        'FAB': 'red',
+
+        'STAR': 'yellow-text',
+        'PROBLEM': 'red-text',
+        'COMMENT': '',
+    },
+    'lightblue': {
+        'BROWSERCHROME': '#0288d1',  # light-blue darken-2
+        'BODY': 'white',
+        'TEXT': 'black-text',
+        'TEXTHEX': '#000',
+        'NAV': 'light-blue darken-2',
+        'PAGEHEADER': 'grey-text lighten-5',
+        'MESSAGE_BACKGROUND': 'orange lighten-2',
+        'MESSAGE_TEXT': 'white-text',
+        'ERRORMESSAGE_BACKGROUND': 'red darken-1',
+        'ERRORMESSAGE_TEXT': 'white-text',
+        'BUTTON': '#fb8c00', # orange darken-1
+        'BUTTON_ACTIVE': '#ffa726',  # orange lighten-1
+        'LINK_TEXT': '#FFF',  # white
+        'CARD_BACKGROUND': 'light-blue lighten-2',
+        'CARD_TEXT': 'black-text',
+        'CARD_LINK': '#263238',  # blue-grey-text darken-4
+        'CHIP_TEXT': '#FFF',  # white
+        'FAB': 'light-blue darken-4',
+
+        'STAR': 'yellow-text',
+        'PROBLEM': 'red-text',
+        'COMMENT': '',
+    },
+    'dark': {
+        'BROWSERCHROME': '#212121',  # grey darken-4
+        'BODY': 'grey darken-4',
+        'TEXT': 'grey-text lighten-1',
+        'TEXTHEX': '#bdbdbd',
+        'NAV': 'grey darken-3',
+        'PAGEHEADER': 'grey-text lighten-1',
+        'MESSAGE_BACKGROUND': 'orange lighten-2',
+        'MESSAGE_TEXT': 'white-text',
+        'ERRORMESSAGE_BACKGROUND': 'red darken-1',
+        'ERRORMESSAGE_TEXT': 'white-text',
+        'BUTTON': '#fb8c00', # orange darken-1
+        'BUTTON_ACTIVE': '#ffa726',  # orange lighten-1
+        'LINK_TEXT': '#fb8c00',  # orange-text darken-1
+        'CARD_BACKGROUND': 'grey darken-3',
+        'CARD_TEXT': 'grey-text lighten-1',
+        'CARD_LINK': '#fb8c00',  # orange-text darken-1
+        'CHIP_TEXT': '#fb8c00',  # orange-text darken-1
+        'FAB': 'red',
+
+        'STAR': 'yellow-text',
+        'PROBLEM': 'red-text',
+        'COMMENT': '',
+    },
+    'amoled': {
+        'BROWSERCHROME': '#000',  # grey darken-4
+        'BODY': 'black',
+        'TEXT': 'grey-text lighten-1',
+        'TEXTHEX': '#bdbdbd',
+        'NAV': 'grey darken-3',
+        'PAGEHEADER': 'grey-text lighten-1',
+        'MESSAGE_BACKGROUND': 'orange lighten-2',
+        'MESSAGE_TEXT': 'white-text',
+        'ERRORMESSAGE_BACKGROUND': 'red darken-1',
+        'ERRORMESSAGE_TEXT': 'white-text',
+        'BUTTON': '#fb8c00', # orange darken-1
+        'BUTTON_ACTIVE': '#ffa726',  # orange lighten-1
+        'LINK_TEXT': '#fb8c00',  # orange-text darken-1
+        'CARD_BACKGROUND': 'grey darken-3',
+        'CARD_TEXT': 'grey-text lighten-1',
+        'CARD_LINK': '#fb8c00',  # orange-text darken-1
+        'CHIP_TEXT': '#fb8c00',  # orange-text darken-1
+        'FAB': 'red',
+
+        'STAR': 'yellow-text',
+        'PROBLEM': 'red-text',
+        'COMMENT': '',
+    }
+}

example_config/settings.py

@@ -1,4 +1,4 @@
-# Virtualenv to use with the wsgi file
+# Virtualenv to use with the wsgi file (optional)
 VENV = '/srv/marks.example.com/venv/bin/activate_this.py'
 
 PORT = 8086

requirements-dev.in

@@ -0,0 +1,3 @@
+-r requirements.in
+
+pylint

requirements-dev.txt

@@ -0,0 +1,29 @@
+#
+# This file is autogenerated by pip-compile
+# To update, run:
+#
+#    pip-compile requirements-dev.in
+#
+astroid==2.3.3            # via pylint
+beautifulsoup4==4.8.2     # via bs4
+bs4==0.0.1
+certifi==2019.11.28       # via requests
+chardet==3.0.4            # via requests
+click==7.0                # via flask
+flask==1.1.1
+idna==2.8                 # via requests
+isort==4.3.21             # via pylint
+itsdangerous==1.1.0       # via flask
+jinja2==2.10.3            # via flask
+lazy-object-proxy==1.4.3  # via astroid
+markupsafe==1.1.1         # via jinja2
+mccabe==0.6.1             # via pylint
+peewee==3.13.1
+pylint==2.4.4
+requests==2.22.0
+six==1.13.0               # via astroid
+soupsieve==1.9.5          # via beautifulsoup4
+typed-ast==1.4.0          # via astroid
+urllib3==1.25.7           # via requests
+werkzeug==0.16.0          # via flask
+wrapt==1.11.2             # via astroid

requirements.in

@@ -0,0 +1,4 @@
+flask
+peewee
+bs4
+requests

requirements.txt

@@ -1,7 +1,21 @@
-flask
-peewee
-flask-peewee
-bs4
-more_itertools
-requests
-utilkit
+#
+# This file is autogenerated by pip-compile
+# To update, run:
+#
+#    pip-compile requirements.in --upgrade
+#
+beautifulsoup4==4.7.1     # via bs4
+bs4==0.0.1
+certifi==2019.3.9         # via requests
+chardet==3.0.4            # via requests
+click==7.0                # via flask
+flask==1.0.2
+idna==2.8                 # via requests
+itsdangerous==1.1.0       # via flask
+jinja2==2.10.1            # via flask
+markupsafe==1.1.1         # via jinja2
+peewee==3.9.3
+requests==2.21.0
+soupsieve==1.9            # via beautifulsoup4
+urllib3==1.24.1           # via requests
+werkzeug==0.15.2          # via flask

setup.py

@@ -5,20 +5,22 @@ https://packaging.python.org/en/latest/distributing.html
 https://github.com/pypa/sampleproject
 """
 
-from setuptools import setup
 # To use a consistent encoding
-from codecs import open
+from codecs import open as codecopen
 from os import path
 
+from setuptools import setup
+
 here = path.abspath(path.dirname(__file__))
 
 # Get the long description from the relevant file
-with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
+with codecopen(path.join(here, 'README.rst'), encoding='utf-8') as f:
     long_description = f.read()
 
 setup(
     name='digimarks', # pip install digimarks
-    description='Simple bookmarking service, using a SQLite database to store bookmarks, supporting tags and automatic title fetching.',
+    description='Simple bookmarking service, using a SQLite database to store bookmarks, supporting tags, automatic '
+                'title fetching and REST API calls.',
     #long_description=open('README.md', 'rt').read(),
     long_description=long_description,
 
@@ -26,7 +28,7 @@ setup(
     # third part for minor release
     # second when api changes
     # first when it becomes stable someday
-    version='0.1.0',
+    version='1.1.99',
     author='Michiel Scholten',
     author_email='michiel@diginaut.net',
 
@@ -35,7 +37,7 @@ setup(
 
     # as a practice no need to hard code version unless you know program wont
     # work unless the specific versions are used
-    install_requires=['Flask', 'Peewee', 'Flask-Peewee', 'requests', 'utilkit'],
+    install_requires=['Flask', 'Peewee', 'requests', 'bs4'],
 
     py_modules=['digimarks'],
 

static/css/digimarks.css

@@ -1,59 +0,0 @@
-
-  /* label color */
-   .input-field label {
-     color: #000;
-   }
-   /* label focus color */
-   .input-field input[type=text]:focus + label {
-     color: #000;
-   }
-   /* label underline focus color */
-   .input-field input[type=text]:focus {
-     border-bottom: 1px solid #000;
-     box-shadow: 0 1px 0 0 #000;
-   }
-   /* valid color */
-   .input-field input[type=text].valid {
-     border-bottom: 1px solid #000;
-     box-shadow: 0 1px 0 0 #000;
-   }
-   /* invalid color */
-   .input-field input[type=text].invalid {
-     border-bottom: 1px solid #000;
-     box-shadow: 0 1px 0 0 #000;
-   }
-   /* icon prefix focus color */
-   .input-field .prefix.active {
-     color: #000;
-   }
-
-
-   /* Card title anchor colour */
-   .white-text .card-title a,
-   .white-text a
-   {
-       color: #FFF;
-   }
-
-   .chip a,
-   .white-text .chip a
-   {
-       color: #1b5e20; /* green darken-4 */
-   }
-
-   .card.tiny
-   {
-       height: 140px;
-   }
-
-   .card .card-reveal .digimark-card-header,
-   .card .digimark-card-header.activator
-   {
-       cursor: pointer;
-       /*display: block;*/
-   }
-
-   .card .digimark-card-content
-   {
-       padding-top: 10px;
-   }

static/js/init.js

@@ -1,7 +0,0 @@
-(function($){
-  $(function(){
-
-    $('.button-collapse').sideNav();
-
-  }); // end of document ready
-})(jQuery); // end of jQuery name space

templates/base.html

@@ -1,61 +0,0 @@
-<!doctype html>
-<html>
-    <head>
-        <title>{% block title %}{% endblock %} - digimarks</title>
-
-        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
-        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0"/>
-
-        <!-- Chrome, Firefox OS and Opera -->
-        <meta name="theme-color" content="#2e7d32" />
-        <!-- Windows Phone -->
-        <meta name="msapplication-navbutton-color" content="#2e7d32">
-        <!-- iOS Safari -->
-        <meta name="apple-mobile-web-app-capable" content="yes">
-        <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
-
-        <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
-        <link href='https://fonts.googleapis.com/css?family=Roboto+Mono&subset=latin,latin-ext' rel='stylesheet' type='text/css'>
-        <link href="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.97.6/css/materialize.min.css" type="text/css" rel="stylesheet" media="screen,projection"/>
-        <link href="{{ url_for('static', filename='css/digimarks.css') }}" type="text/css" rel="stylesheet" media="screen,projection"/>
-    </head>
-    <body class="grey lighten-4">
-        <nav class="green darken-3" role="navigation">
-            <div class="nav-wrapper container"><a id="logo-container" href="{% if userkey %}{{ url_for('bookmarks', userkey=userkey) }}{% else %}{{ url_for('index') }}{% endif %}" class="brand-logo">digimarks</a>
-                <ul class="right hide-on-med-and-down">
-                    {% if userkey %}
-                    <li><a href="{{ url_for('tags', userkey=userkey) }}">Tags</a></li>
-                    <li><a href="{{ url_for('addbookmark', userkey=userkey) }}">Add bookmark</a></li>
-                    {% endif %}
-                </ul>
-
-                <ul id="nav-mobile" class="side-nav">
-                    {% if userkey %}
-                    <li><a href="{{ url_for('tags', userkey=userkey) }}">Tags</a></li>
-                    <li><a href="{{ url_for('addbookmark', userkey=userkey) }}">Add bookmark</a></li>
-                    {% endif %}
-                </ul>
-                <a href="#" data-activates="nav-mobile" class="button-collapse"><i class="material-icons">menu</i></a>
-            </div>
-        </nav>
-        <div class="section no-pad-bot" id="index-banner">
-            <div class="container">
-                <div class="header grey-text lighten-5">
-                    <h1>{% block pageheader %}Bookmarks{% endblock %}</h1>
-                </div>
-            </div>
-        </div>
-        <div class="container">
-            <div class="section">
-                {% block pagecontent %}
-                {% endblock %}
-            </div>
-        </div>
-
-        <!-- Scripts -->
-        <script src="https://code.jquery.com/jquery-2.2.3.min.js"></script>
-        <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/0.97.6/js/materialize.min.js"></script>
-        <script src="{{ url_for('static', filename='js/init.js') }}"></script>
-
-    </body>
-</html>

templates/bookmarks.html

@@ -1,121 +0,0 @@
-{% extends "base.html" %}
-{% if not action %}
-{% set action = 'Bookmarks' %}
-{% endif %}
-{% block title %}{{ action }}{% endblock %}
-{% block pageheader %}{{ action }}{% endblock %}
-{% block pagecontent %}
-
-{% if tag and not publictag %}
-<div class="row">
-    <div class="col s12">
-        <a href="{{ url_for('addpublictag', userkey=userkey, tag=tag) }}">Create public page <i class="material-icons right">tag</i></a>
-    </div>
-</div>
-{% endif %}
-
-{% if tag and publictag %}
-<div class="row">
-    <div class="col s12"><a href="{{ url_for('publictag', tagkey=publictag.tagkey) }}">Public link</a></div>
-</div>
-{% endif %}
-
-{% if message %}
-<div class="row">
-    <div class="col s12">
-        <div class="card-panel orange lighten-2">
-            <span class="white-text">
-                {{ message }}
-            </span>
-        </div>
-    </div>
-</div>
-{% endif %}
-
-<div class="row">
-    <form action="{{ url_for('bookmarks', userkey=userkey) }}" method="POST">
-        <div class="input-field col l10 m10 s8">
-            <input placeholder="filter" type="text" name="filter" id="filter" value="{{ filter }}" class="validate" />
-        </div>
-
-        <div class="input-field col l2 m2 s4">
-            <p class="left-align"><button class="btn btn-large waves-effect waves-light" type="submit" name="submit">Filter</button></p>
-        </div>
-    </form>
-</div>
-
-{% if tags %}
-<div class="row">
-    <p>
-    {% for tag in tags %}
-    <div class="chip">
-        <a href="{{ url_for('tag', userkey=userkey, tag=tag) }}">{{ tag }}</a>
-    </div>
-    {% endfor %}
-    </p>
-</div>
-{% endif %}
-
-<div class="row">
-    {% for bookmark in bookmarks %}
-    <div class="col s12 m6 l4">
-        {#
-        <div class="thumbnail">
-            <a href="{{ bookmark.url }}" title="{{ bookmark.url }}">
-                <img style="width:450px;" src="{{ bookmark.image }}" />
-            </a>
-            <p><a href="{{ bookmark.url }}">{{ bookmark.url|urlize(25) }}</a></p>
-            <p>{{ bookmark.created_date.strftime("%m/%d/%Y %H:%M") }}</p>
-        </div>
-        #}
-        <div class="card tiny green darken-3">
-            <div class="card-content white-text">
-                <span class="digimark-card-header activator">
-                    {% for tag in bookmark.tags_list %}
-                        <div class="chip">
-                            <a href="{{ url_for('tag', userkey=userkey, tag=tag) }}">{{ tag }}</a>
-                        </div>
-                    {% endfor %}
-                    <i class="material-icons right">more_vert</i>
-                </span>
-                    <div class="valign-wrapper digimark-card-content">
-                    <div class="valign">
-                        {% if bookmark.http_status != 200 %}
-                            <i class="tiny material-icons" title="HTTP status {{ bookmark.http_status }}">report_problem</i>
-                        {% endif %}
-                        {% if bookmark.favicon %}
-                            <img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" />&nbsp;
-                        {% endif %}
-                        <a href="{{ bookmark.url }}" title="{{ bookmark.url }}" rel="noreferrer">
-                        {% if bookmark.starred == True %}
-                            <i class="tiny material-icons yellow-text">star</i>
-                        {% endif %}
-                        {% if bookmark.title %}
-                            {{ bookmark.title }}
-                        {% else %}
-                            [ no title ]
-                        {% endif %}
-                    </a></div>
-                    </div>
-            </div>
-            <div class="card-reveal green darken-3">
-                <span class="card-title white-text valign-wrapper">Added @ {{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}<i class="material-icons right">close</i></span>
-                <div class="white-text valign">
-                {% if bookmark.favicon %}
-                    <img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" />&nbsp;&nbsp;&nbsp;
-                {% endif %}
-                <a href="{{ url_for('editbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" style="padding: 3px"><i class="tiny material-icons">mode_edit</i> EDIT</a>
-                </div>
-            </div>
-        </div>
-    </div>
-    {% endfor %}
-
-    {#
-    <div class="pagination">
-        {% if page > 1 %}<a href="./?page={{ page - 1 }}">Previous</a>{% endif %}
-        {% if pagination.get_pages() > page %}<a href="./?page={{ page + 1 }}">Next</a>{% endif %}
-    </div>
-    #}
-</div>
-{% endblock %}

templates/edit.html

@@ -1,110 +0,0 @@
-{% extends "base.html" %}
-{% block title %}{{ action }}{% endblock %}
-{% block pageheader %}{{ action }}{% endblock %}
-{% block pagecontent %}
-
-{% if bookmark.http_status != 200 %}
-<div class="row">
-    <div class="col s12">
-        <div class="card-panel red darken-1">
-            <span class="white-text">
-                {% if bookmark.http_status == 404 %}
-                <i class="material-icons">report_problem</i>&nbsp;&nbsp;URL not found (404), broken/outdated link?
-                {% else %}
-                <i class="material-icons">report_problem</i>&nbsp;&nbsp;HTTP status {{ bookmark.http_status }}
-                {% endif %}
-            </span>
-        </div>
-    </div>
-    </div>
-{% endif %}
-
-{% if message %}
-<div class="row">
-    <div class="col s12">
-        <div class="card-panel orange lighten-2">
-            <span class="white-text">
-                {{ message }}
-            </span>
-        </div>
-    </div>
-</div>
-{% endif %}
-
-    {% if formaction and formaction == 'edit' %}
-    <form action="{{ url_for('editingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" method="POST">
-    {% else %}
-    <form action="{{ url_for('addingbookmark', userkey=userkey) }}" method="POST">
-    {% endif %}
-
-<div class="row">
-
-        <div class="input-field col s12">
-            <i class="material-icons prefix">description</i>
-            <input placeholder="title" type="text" name="title" id="title" value="{{ bookmark.title }}" class="validate" />
-            <label for="title">Title</label>
-        </div>
-
-        <div class="input-field col s12">
-            <i class="material-icons prefix">turned_in</i>
-            <input placeholder="url" type="text" name="url" id="url" value="{{ bookmark.url }}" class="validate" />
-            <label for="url">URL</label>
-        </div>
-
-        <div class="input-field col s12">
-            <i class="material-icons prefix">label</i>
-            <input placeholder="tags, divided by comma's" type="text" name="tags" id="tags" value="{{ bookmark.tags }}" class="validate" />
-            <label for="tags">Tags</label>
-        </div>
-
-        <div class="input-field col s12">
-            {#<i class="material-icons prefix">star</i>#}
-            <input type="checkbox" name="starred" id="starred" {% if bookmark.starred == True %}checked{% endif %} />
-            <label for="starred">Starred</label>
-        </div>
-
-        <div class="input-field col s12">
-            <input type="checkbox" name="strip" id="strip" />
-            <label for="strip">Strip parameters from url</label>
-        </div>
-
-    {% if bookmark.url_hash %}
-    </div>
-    <div class="row">
-        <div class="col l4 m6 s12">
-            <table>
-                <tr>
-                    <th>Added</th>
-                    <td>{{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}</td>
-                </tr>
-                {% if bookmark.modified_date %}
-                <tr>
-                    <th>Modified</th>
-                    <td>{{ bookmark.modified_date.strftime('%Y-%m-%d %H:%M') }}</td>
-                </tr>
-                {% endif %}
-                {% if bookmark.deleted_date %}
-                <tr>
-                    <th>Deleted</th>
-                    <td>{{ bookmark.deleted_date.strftime('%Y-%m-%d %H:%M') }}</td>
-                </tr>
-                {% endif %}
-            </table>
-        </div>
-    </div>
-    <div class="row">
-    {% endif %}
-
-        <div class="input-field col l2 m3 s4">
-            <p class="left-align"><button class="btn btn-large waves-effect waves-light" type="submit" name="submit">Save <i class="material-icons right">send</i></button></p>
-        </div>
-    </form>
-    {% if bookmark.url_hash %}
-        <div class="input-field col l2 m3 s4">
-        <form action="{{ url_for('deletingbookmark', userkey=userkey, urlhash=bookmark.url_hash) }}" method="POST">
-            <p class="left-align"><button class="btn btn-large waves-effect waves-light" type="submit" name="delete">Delete <i class="material-icons right">delete</i></button></p>
-        </form>
-        </div>
-    {% endif %}
-</div>
-{% endblock %}

templates/publicbookmarks.html

@@ -1,56 +0,0 @@
-{% extends "base.html" %}
-{% if not action %}
-{% set action = 'Bookmarks' %}
-{% endif %}
-{% block title %}{{ action }}{% endblock %}
-{% block pageheader %}{{ action }}{% endblock %}
-{% block pagecontent %}
-
-{% if message %}
-<div class="row">
-    <div class="col s12">
-        <div class="card-panel orange lighten-2">
-            <span class="white-text">
-                {{ message }}
-            </span>
-        </div>
-    </div>
-</div>
-{% endif %}
-
-<div class="row">
-    {% for bookmark in bookmarks %}
-    <div class="col s12 m6 l4">
-        <div class="card green darken-3">
-            <div class="card-content white-text">
-                <span class="card-title activator"><i class="material-icons right">more_vert</i></span>
-                    {% if bookmark.http_status != 200 %}
-                        <i class="material-icons" title="HTTP status {{ bookmark.http_status }}">report_problem</i>
-                    {% endif %}
-                    {% if bookmark.favicon %}
-                        <img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" />&nbsp;&nbsp;&nbsp;
-                    {% endif %}
-                    <p><a href="{{ bookmark.url }}" title="{{ bookmark.url }}" rel="noreferrer">
-                        {% if bookmark.starred == True %}
-                            <i class="material-icons">star</i>
-                        {% endif %}
-                        {% if bookmark.title %}
-                            {{ bookmark.title }}
-                        {% else %}
-                            [ no title ]
-                        {% endif %}
-                    </a></p>
-            </div>
-            <div class="card-reveal green darken-3">
-                <span class="card-title white-text">Added @ {{ bookmark.created_date.strftime('%Y-%m-%d %H:%M') }}<i class="material-icons right">close</i></span>
-                <div class="white-text">
-                {% if bookmark.favicon %}
-                    <img src="{{ url_for('static', filename='favicons/' + bookmark.favicon) }}" />&nbsp;&nbsp;&nbsp;
-                {% endif %}
-                </div>
-            </div>
-        </div>
-    </div>
-    {% endfor %}
-</div>
-{% endblock %}

templates/tags.html

@@ -1,14 +0,0 @@
-{% extends "base.html" %}
-{% block title %}Tags{% endblock %}
-{% block pageheader %}Tags{% endblock %}
-{% block pagecontent %}
-<div class="row">
-
-    {% for tag in tags %}
-    <div class="chip">
-        <a href="{{ url_for('tag', userkey=userkey, tag=tag) }}">{{ tag }}</a>
-    </div>
-    {% endfor %}
-
-</div>
-{% endblock %}

wsgi.py

@@ -1,7 +1,13 @@
 # Activate virtualenv
 import settings
-activate_this = settings.VENV
-execfile(activate_this, dict(__file__=activate_this))
+activate_this = getattr(settings, 'VENV', None)
+# FIXME: python 2 *and* python 3 compatibility
+# Python 2
+#if activate_this:
+#    execfile(activate_this, dict(__file__=activate_this))
+# Python 3
+with open(activate_this) as file_:
+    exec(file_.read(), dict(__file__=activate_this))
 
 from digimarks import app as application