docstrings for module files

Implement alembic migrations and better DB models module, including some async rework

CHANGELOG.md

@@ -7,13 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## TODO
 
-- Sorting of bookmarks
-  - Sort by title
-  - Sort by date
-- Logging of actions
 - Add new way of authentication and editing bookmark collections:
   https://github.com/aquatix/digimarks/issues/8 and https://github.com/aquatix/digimarks/issues/9
-- Change adding tags to use the MaterializeCSS tags: https://materializecss.com/chips.html
+- Change adding tags to use ~~the MaterializeCSS tags: https://materializecss.com/chips.html~~ a nice tags lib/styling
 - Do calls to the API endpoint of an existing bookmark when editing properties
   (for example to update tags, title and such, also to already suggest title)
 - Look into compatibility with del.icio.us, so we can make use of existing browser integration
@@ -23,7 +19,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- Settings through Pydantic Settings
+- Settings now work through Pydantic Settings
+- New UI theme(s) through digui
+- Caching of the bookmarks, tags and more in the browser, for fast filtering and lookups
+- Sorting of bookmarks
+  - Sort by title
+  - Sort by date
+- Logging of actions
+- Recognise when a url already is in the list of known bookmarks and fill in the form with already-known data
 
 ### Changed
 - Moved from Flask to FastAPI

README.rst

@@ -30,21 +30,42 @@ necessary packages:
     git clone https://github.com/aquatix/digimarks.git
     cd digimarks
     mkvirtualenv digimarks # or whatever project you are working on
-    pip install -r requirements.txt
+    # If you just want to run it, no need for development dependencies
+    uv sync --active --no-dev
+    # Otherwise, install everything in the active virtualenv
+    uv sync --active
+
+
+Migrating from version 1
+------------------------
+
+To be able to use the new database schema's, you will need to migrate your existing ``bookmarks.db`` to one under the control of the ``alembic`` migrations tool.
+
+To do so, start with making a backup of this ``bookmarks.db`` file to a safe place.
+
+Then, stamp the initial migration into the database, and migrate to the latest version:
+
+.. code-block:: bash
+
+   alembic stamp 115bcd2e1a38
+
+   alembic upgrade head
 
 
 Usage / example configuration
 -----------------------------
 
+OUT OF DATE!
+
 Copy ``settings.py`` from example_config to the parent directory and
 configure to your needs (*at the least* change the value of `SYSTEMKEY`).
 
-Do not forget to fill in the `MASHAPE_API_KEY` value, which you [can request on the RapidAPI website](https://rapidapi.com/realfavicongenerator/api/realfavicongenerator).
+Do not forget to fill in the `MASHAPE_API_KEY` value, which you ``can request on the RapidAPI website <https://rapidapi.com/realfavicongenerator/api/realfavicongenerator>`_.
 
 Run digimarks as a service under nginx or apache and call the appropriate
 url's when wanted.
 
-Url's are of the form https://marks.example.com/<userkey>/<action>
+Url's are of the form ``https://marks.example.com/<userkey>/<action>``
 
 digimarks can also be run from the command line: ``uvicorn digimarks:app --reload``
 

__init__.py

@@ -0,0 +1 @@
+"""Digimarks project."""

alembic.ini

@@ -0,0 +1,147 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts.
+# this is typically a path given in POSIX (e.g. forward slashes)
+# format, relative to the token %(here)s which refers to the location of this
+# ini file
+script_location = %(here)s/migrations
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.  for multiple paths, the path separator
+# is defined by "path_separator" below.
+prepend_sys_path = .
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the python>=3.9 or backports.zoneinfo library and tzdata library.
+# Any required deps can installed by adding `alembic[tz]` to the pip requirements
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to <script_location>/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "path_separator"
+# below.
+# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
+
+# path_separator; This indicates what character is used to split lists of file
+# paths, including version_locations and prepend_sys_path within configparser
+# files such as alembic.ini.
+# The default rendered in new alembic.ini files is "os", which uses os.pathsep
+# to provide os-dependent path splitting.
+#
+# Note that in order to support legacy alembic.ini files, this default does NOT
+# take place if path_separator is not present in alembic.ini.  If this
+# option is omitted entirely, fallback logic is as follows:
+#
+# 1. Parsing of the version_locations option falls back to using the legacy
+#    "version_path_separator" key, which if absent then falls back to the legacy
+#    behavior of splitting on spaces and/or commas.
+# 2. Parsing of the prepend_sys_path option falls back to the legacy
+#    behavior of splitting on spaces, commas, or colons.
+#
+# Valid values for path_separator are:
+#
+# path_separator = :
+# path_separator = ;
+# path_separator = space
+# path_separator = newline
+#
+# Use os.pathsep. Default configuration used for new projects.
+path_separator = os
+
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+# recursive_version_locations = false
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# database URL.  This is consumed by the user-maintained env.py script only.
+# other means of configuring database URLs may be customized within the env.py
+# file.
+sqlalchemy.url = sqlite+aiosqlite:///bookmarks.db
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
+# hooks = ruff
+# ruff.type = module
+# ruff.module = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Alternatively, use the exec runner to execute a binary found on your PATH
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration.  This is also consumed by the user-maintained
+# env.py script only.
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARNING
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARNING
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

migrations/README

@@ -0,0 +1 @@
+Generic single-database configuration with an async dbapi.

migrations/env.py

@@ -0,0 +1,94 @@
+"""Alembic environment file for SQLAlchemy."""
+
+import asyncio
+from logging.config import fileConfig
+
+from alembic import context
+from sqlalchemy import pool
+from sqlalchemy.engine import Connection
+from sqlalchemy.ext.asyncio import async_engine_from_config
+from sqlmodel import SQLModel
+
+from src.digimarks.models import Bookmark, PublicTag, User  # noqa
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+if config.config_file_name is not None:
+    fileConfig(config.config_file_name)
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+# from myapp import mymodel
+# target_metadata = mymodel.Base.metadata
+target_metadata = SQLModel.metadata
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline() -> None:
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    url = config.get_main_option('sqlalchemy.url')
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={'paramstyle': 'named'},
+        render_as_batch=True,
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def do_run_migrations(connection: Connection) -> None:
+    """Run the migrations."""
+    context.configure(
+        connection=connection,
+        target_metadata=target_metadata,
+        render_as_batch=True,
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+async def run_async_migrations() -> None:
+    """In this scenario we need to create an Engine and associate a connection with the context."""
+    connectable = async_engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix='sqlalchemy.',
+        poolclass=pool.NullPool,
+    )
+
+    async with connectable.connect() as connection:
+        await connection.run_sync(do_run_migrations)
+
+    await connectable.dispose()
+
+
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode."""
+    asyncio.run(run_async_migrations())
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

migrations/script.py.mako

@@ -0,0 +1,29 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
+branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
+depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    ${downgrades if downgrades else "pass"}

migrations/versions/115bcd2e1a38_initial_migration.py

@@ -0,0 +1,68 @@
+"""Initial migration.
+
+Revision ID: 115bcd2e1a38
+Revises:
+Create Date: 2025-09-12 16:06:16.479075
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = '115bcd2e1a38'
+down_revision: Union[str, Sequence[str], None] = None
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        'bookmark',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('userkey', sa.String(length=255), nullable=False),
+        sa.Column('title', sa.String(length=255), nullable=False),
+        sa.Column('url', sa.String(length=255), nullable=False),
+        sa.Column('created_date', sa.DateTime(), nullable=False),
+        sa.Column('url_hash', sa.String(length=255), nullable=False),
+        sa.Column('tags', sa.String(length=255), nullable=False),
+        sa.Column('http_status', sa.Integer(), nullable=False),
+        sa.Column('modified_date', sa.DateTime(), nullable=True),
+        sa.Column('favicon', sa.String(length=255), nullable=True),
+        sa.Column('starred', sa.Boolean(), server_default=sa.text('0'), nullable=True),
+        sa.Column('deleted_date', sa.DateTime(), server_default=sa.text('(null)'), nullable=True),
+        sa.Column('status', sa.Integer(), server_default=sa.text('0'), nullable=True),
+        sa.Column('note', sa.Text(), server_default=sa.text('(null)'), nullable=True),
+        sa.PrimaryKeyConstraint('id'),
+    )
+    op.create_table(
+        'publictag',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('tagkey', sa.String(length=255), nullable=False),
+        sa.Column('userkey', sa.String(length=255), nullable=False),
+        sa.Column('tag', sa.String(length=255), nullable=False),
+        sa.Column('created_date', sa.DateTime(), server_default=sa.text('(null)'), nullable=True),
+        sa.PrimaryKeyConstraint('id'),
+    )
+    op.create_table(
+        'user',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('username', sa.String(length=255), nullable=False),
+        sa.Column('key', sa.String(length=255), nullable=False),
+        sa.Column('created_date', sa.DateTime(), nullable=False),
+        sa.Column('theme', sa.String(length=20), server_default=sa.text("'green'"), nullable=True),
+        sa.PrimaryKeyConstraint('id'),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('user')
+    op.drop_table('publictag')
+    op.drop_table('bookmark')
+    # ### end Alembic commands ###

migrations/versions/a8d8e45f60a1_migrate_to_sqlmodel.py

@@ -0,0 +1,96 @@
+"""Migrate to sqlmodel.
+
+Revision ID: a8d8e45f60a1
+Revises: 115bcd2e1a38
+Create Date: 2025-09-12 16:10:41.378716
+"""
+
+from datetime import UTC, datetime
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+import sqlmodel
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = 'a8d8e45f60a1'
+down_revision: Union[str, Sequence[str], None] = '115bcd2e1a38'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('bookmark', schema=None) as batch_op:
+        batch_op.alter_column(
+            'note',
+            existing_type=sa.TEXT(),
+            type_=sqlmodel.sql.sqltypes.AutoString(),
+            nullable=True,
+            existing_server_default=sa.text('(null)'),
+        )
+        batch_op.alter_column(
+            'starred', existing_type=sa.BOOLEAN(), nullable=False, existing_server_default=sa.text('0')
+        )
+        batch_op.alter_column('modified_date', existing_type=sa.DATETIME(), nullable=True)
+        batch_op.alter_column(
+            'deleted_date', existing_type=sa.DATETIME(), nullable=True, existing_server_default=sa.text('(null)')
+        )
+        batch_op.alter_column(
+            'status', existing_type=sa.INTEGER(), nullable=False, existing_server_default=sa.text('0')
+        )
+        batch_op.create_foreign_key('bookmark_user', 'user', ['userkey'], ['key'])
+    with op.batch_alter_table('publictag', schema=None) as batch_op:
+        batch_op.alter_column(
+            'created_date',
+            existing_type=sa.DATETIME(),
+            nullable=True,
+            existing_server_default=sa.text(str(datetime.now(UTC))),
+        )
+        batch_op.create_foreign_key('publictag_user', 'user', ['userkey'], ['key'])
+    with op.batch_alter_table('user', schema=None) as batch_op:
+        batch_op.alter_column(
+            'theme', existing_type=sa.VARCHAR(length=20), nullable=False, existing_server_default=sa.text("'green'")
+        )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.alter_column(
+        'user', 'theme', existing_type=sa.VARCHAR(length=20), nullable=True, existing_server_default=sa.text("'green'")
+    )
+    op.drop_constraint(None, 'publictag', type_='foreignkey')
+    op.alter_column(
+        'publictag',
+        'created_date',
+        existing_type=sa.DATETIME(),
+        nullable=True,
+        existing_server_default=sa.text('(null)'),
+    )
+    op.drop_constraint(None, 'bookmark', type_='foreignkey')
+    op.alter_column(
+        'bookmark', 'status', existing_type=sa.INTEGER(), nullable=True, existing_server_default=sa.text('0')
+    )
+    op.alter_column(
+        'bookmark',
+        'deleted_date',
+        existing_type=sa.DATETIME(),
+        nullable=True,
+        existing_server_default=sa.text('(null)'),
+    )
+    op.alter_column('bookmark', 'modified_date', existing_type=sa.DATETIME(), nullable=True)
+    op.alter_column(
+        'bookmark', 'starred', existing_type=sa.BOOLEAN(), nullable=True, existing_server_default=sa.text('0')
+    )
+    op.alter_column(
+        'bookmark',
+        'note',
+        existing_type=sqlmodel.sql.sqltypes.AutoString(),
+        type_=sa.TEXT(),
+        nullable=True,
+        existing_server_default=sa.text('(null)'),
+    )
+    # ### end Alembic commands ###

migrations/versions/b8cbc6957df5_renamed_keys.py

@@ -0,0 +1,50 @@
+"""Renamed keys.
+
+Revision ID: b8cbc6957df5
+Revises: a8d8e45f60a1
+Create Date: 2025-09-12 22:26:38.684120
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = 'b8cbc6957df5'
+down_revision: Union[str, Sequence[str], None] = 'a8d8e45f60a1'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('bookmark', schema=None) as batch_op:
+        batch_op.drop_constraint(batch_op.f('bookmark_user'), type_='foreignkey')
+        batch_op.alter_column('userkey', new_column_name='user_key')
+        batch_op.create_foreign_key('bookmark_user', 'user', ['user_key'], ['key'])
+
+    with op.batch_alter_table('publictag', schema=None) as batch_op:
+        batch_op.drop_constraint(batch_op.f('publictag_user'), type_='foreignkey')
+        batch_op.alter_column('userkey', new_column_name='user_key')
+        batch_op.alter_column('tagkey', new_column_name='tag_key')
+        batch_op.create_foreign_key('publictag_user', 'user', ['user_key'], ['key'])
+
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('publictag', schema=None) as batch_op:
+        batch_op.drop_constraint(batch_op.f('publictag_user'), type_='foreignkey')
+        batch_op.alter_column('user_key', new_column_name='userkey')
+        batch_op.alter_column('tag_key', new_column_name='tagkey')
+        batch_op.create_foreign_key('publictag_user', 'user', ['userkey'], ['key'])
+
+    with op.batch_alter_table('bookmark', schema=None) as batch_op:
+        batch_op.drop_constraint(batch_op.f('bookmark_user'), type_='foreignkey')
+        batch_op.alter_column('user_key', new_column_name='userkey')
+        batch_op.create_foreign_key('bookmark_user', 'user', ['userkey'], ['key'])
+
+    # ### end Alembic commands ###

pyproject.toml

@@ -6,25 +6,51 @@ build-backend = "setuptools.build_meta"
 name = "digimarks"
 version = "1.1.99"
 authors = [
-    {name = "Michiel Scholten", email = "michiel@diginaut.net"},
+    { name = "Michiel Scholten", email = "michiel@diginaut.net" },
 ]
-description='Simple bookmarking service, using a SQLite database to store bookmarks, supporting tags, automatic title fetching and REST API calls.'
+description = 'Simple bookmarking service, using a SQLite database to store bookmarks, supporting tags, automatic title fetching and REST API calls.'
 readme = "README.rst"
-requires-python = ">=3.7"
+requires-python = ">=3.10"
 keywords = ["bookmarks", "api"]
-license = {text = "Apache"}
+license = { text = "Apache" }
 classifiers = [
     "Framework :: FastAPI",
     "Programming Language :: Python :: 3",
     "License :: OSI Approved :: Apache Software License",
 ]
 dependencies = [
-    "importlib-metadata; python_version<'3.8'",
     "fastapi[all]",
+    "sqlmodel",
+    "alembic",
+    "aiosqlite",
     "pydantic>2.0",
-    "requests",
-    "bs4",
-    "feedgen"
+    "httpx",
+    "beautifulsoup4",
+    "extract_favicon",
+    "feedgen",
+]
+
+[dependency-groups]
+dev = [
+  {include-group = "lint"},
+  {include-group = "pub"},
+  {include-group = "test"}
+]
+test = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+]
+lint = [
+    "ruff>=0.1.0",
+    "mypy>=1.0.0",
+]
+# Publishing on PyPI
+pub = [
+    "build",
+    "twine"
+]
+server = [
+    "gunicorn>=23.0.0",
 ]
 # dynamic = ["version"]
 

requirements-dev.txt

@@ -1,223 +1,11 @@
-#
-# This file is autogenerated by pip-compile with Python 3.10
-# by the following command:
-#
-#    pip-compile requirements-dev.in
-#
-annotated-types==0.5.0
-    # via pydantic
-anyio==3.7.1
-    # via
-    #   httpcore
-    #   starlette
-    #   watchfiles
-astroid==2.15.6
-    # via pylint
-beautifulsoup4==4.12.2
-    # via bs4
-black==23.7.0
-    # via -r requirements-dev.in
-bleach==6.0.0
-    # via readme-renderer
-bs4==0.0.1
-    # via -r requirements.in
-build==0.10.0
-    # via -r requirements-dev.in
-certifi==2023.7.22
-    # via
-    #   httpcore
-    #   httpx
-    #   requests
-cffi==1.15.1
-    # via cryptography
-charset-normalizer==3.2.0
-    # via requests
-click==8.1.6
-    # via
-    #   black
-    #   uvicorn
-cryptography==41.0.2
-    # via secretstorage
-dill==0.3.7
-    # via pylint
-dnspython==2.4.1
-    # via email-validator
-docutils==0.20.1
-    # via readme-renderer
-email-validator==2.0.0.post2
-    # via fastapi
-exceptiongroup==1.1.2
-    # via anyio
-fastapi[all]==0.100.1
-    # via -r requirements.in
-feedgen==0.9.0
-    # via -r requirements.in
-greenlet==2.0.2
-    # via sqlalchemy
-h11==0.14.0
-    # via
-    #   httpcore
-    #   uvicorn
-httpcore==0.17.3
-    # via httpx
-httptools==0.6.0
-    # via uvicorn
-httpx==0.24.1
-    # via fastapi
-idna==3.4
-    # via
-    #   anyio
-    #   email-validator
-    #   httpx
-    #   requests
-importlib-metadata==6.8.0
-    # via
-    #   keyring
-    #   twine
-isort==5.12.0
-    # via pylint
-itsdangerous==2.1.2
-    # via fastapi
-jaraco-classes==3.3.0
-    # via keyring
-jeepney==0.8.0
-    # via
-    #   keyring
-    #   secretstorage
-jinja2==3.1.2
-    # via fastapi
-keyring==24.2.0
-    # via twine
-lazy-object-proxy==1.9.0
-    # via astroid
-lxml==4.9.3
-    # via feedgen
-markdown-it-py==3.0.0
-    # via rich
-markupsafe==2.1.3
-    # via jinja2
-mccabe==0.7.0
-    # via pylint
-mdurl==0.1.2
-    # via markdown-it-py
-more-itertools==10.0.0
-    # via jaraco-classes
-mypy-extensions==1.0.0
-    # via black
-orjson==3.9.2
-    # via fastapi
-packaging==23.1
-    # via
-    #   black
-    #   build
-pathspec==0.11.2
-    # via black
-pkginfo==1.9.6
-    # via twine
-platformdirs==3.10.0
-    # via
-    #   black
-    #   pylint
-pycparser==2.21
-    # via cffi
-pydantic==2.1.1
-    # via
-    #   fastapi
-    #   pydantic-extra-types
-    #   pydantic-settings
-pydantic-core==2.4.0
-    # via pydantic
-pydantic-extra-types==2.0.0
-    # via fastapi
-pydantic-settings==2.0.2
-    # via fastapi
-pygments==2.15.1
-    # via
-    #   readme-renderer
-    #   rich
-pylint==2.17.5
-    # via -r requirements-dev.in
-pyproject-hooks==1.0.0
-    # via build
-python-dateutil==2.8.2
-    # via feedgen
-python-dotenv==1.0.0
-    # via
-    #   pydantic-settings
-    #   uvicorn
-python-multipart==0.0.6
-    # via fastapi
-pyyaml==6.0.1
-    # via
-    #   fastapi
-    #   uvicorn
-readme-renderer==40.0
-    # via twine
-requests==2.31.0
-    # via
-    #   -r requirements.in
-    #   requests-toolbelt
-    #   twine
-requests-toolbelt==1.0.0
-    # via twine
-rfc3986==2.0.0
-    # via twine
-rich==13.5.0
-    # via twine
-ruff==0.0.280
-    # via -r requirements-dev.in
-secretstorage==3.3.3
-    # via keyring
-six==1.16.0
-    # via
-    #   bleach
-    #   python-dateutil
-sniffio==1.3.0
-    # via
-    #   anyio
-    #   httpcore
-    #   httpx
-soupsieve==2.4.1
-    # via beautifulsoup4
-sqlalchemy==2.0.19
-    # via -r requirements.in
-starlette==0.27.0
-    # via fastapi
-tomli==2.0.1
-    # via
-    #   black
-    #   build
-    #   pylint
-    #   pyproject-hooks
-tomlkit==0.12.1
-    # via pylint
-twine==4.0.2
-    # via -r requirements-dev.in
-typing-extensions==4.7.1
-    # via
-    #   astroid
-    #   fastapi
-    #   pydantic
-    #   pydantic-core
-    #   sqlalchemy
-    #   uvicorn
-ujson==5.8.0
-    # via fastapi
-urllib3==2.0.4
-    # via
-    #   requests
-    #   twine
-uvicorn[standard]==0.23.1
-    # via fastapi
-uvloop==0.17.0
-    # via uvicorn
-watchfiles==0.19.0
-    # via uvicorn
-webencodings==0.5.1
-    # via bleach
-websockets==11.0.3
-    # via uvicorn
-wrapt==1.15.0
-    # via astroid
-zipp==3.16.2
-    # via importlib-metadata
+-r requirements.txt
+
+# Linting and fixing, including isort
+ruff
+
+# Test suite
+pytest
+
+# Publishing on PyPI
+build
+twine

requirements.in

@@ -1,6 +1,14 @@
 # Core application
 fastapi[all]
 sqlmodel
+sqlalchemy
+pydantic
+pydantic_settings
+alembic
+aiosqlite
+
+# Fetch external resources
+httpx
 
 # Fetch title etc from links
 beautifulsoup4

requirements.txt

@@ -1,121 +1,20 @@
-#
-# This file is autogenerated by pip-compile with Python 3.10
-# by the following command:
-#
-#    pip-compile requirements.in
-#
-annotated-types==0.5.0
-    # via pydantic
-anyio==3.7.1
-    # via
-    #   httpcore
-    #   starlette
-    #   watchfiles
-beautifulsoup4==4.12.2
-    # via bs4
-bs4==0.0.1
-    # via -r requirements.in
-certifi==2023.7.22
-    # via
-    #   httpcore
-    #   httpx
-    #   requests
-charset-normalizer==3.2.0
-    # via requests
-click==8.1.6
-    # via uvicorn
-dnspython==2.4.1
-    # via email-validator
-email-validator==2.0.0.post2
-    # via fastapi
-exceptiongroup==1.1.2
-    # via anyio
-fastapi[all]==0.100.1
-    # via -r requirements.in
-feedgen==0.9.0
-    # via -r requirements.in
-greenlet==2.0.2
-    # via sqlalchemy
-h11==0.14.0
-    # via
-    #   httpcore
-    #   uvicorn
-httpcore==0.17.3
-    # via httpx
-httptools==0.6.0
-    # via uvicorn
-httpx==0.24.1
-    # via fastapi
-idna==3.4
-    # via
-    #   anyio
-    #   email-validator
-    #   httpx
-    #   requests
-itsdangerous==2.1.2
-    # via fastapi
-jinja2==3.1.2
-    # via fastapi
-lxml==4.9.3
-    # via feedgen
-markupsafe==2.1.3
-    # via jinja2
-orjson==3.9.2
-    # via fastapi
-pydantic==2.1.1
-    # via
-    #   fastapi
-    #   pydantic-extra-types
-    #   pydantic-settings
-pydantic-core==2.4.0
-    # via pydantic
-pydantic-extra-types==2.0.0
-    # via fastapi
-pydantic-settings==2.0.2
-    # via fastapi
-python-dateutil==2.8.2
-    # via feedgen
-python-dotenv==1.0.0
-    # via
-    #   pydantic-settings
-    #   uvicorn
-python-multipart==0.0.6
-    # via fastapi
-pyyaml==6.0.1
-    # via
-    #   fastapi
-    #   uvicorn
-requests==2.31.0
-    # via -r requirements.in
-six==1.16.0
-    # via python-dateutil
-sniffio==1.3.0
-    # via
-    #   anyio
-    #   httpcore
-    #   httpx
-soupsieve==2.4.1
-    # via beautifulsoup4
-sqlalchemy==2.0.19
-    # via -r requirements.in
-starlette==0.27.0
-    # via fastapi
-typing-extensions==4.7.1
-    # via
-    #   fastapi
-    #   pydantic
-    #   pydantic-core
-    #   sqlalchemy
-    #   uvicorn
-ujson==5.8.0
-    # via fastapi
-urllib3==2.0.4
-    # via requests
-uvicorn[standard]==0.23.1
-    # via fastapi
-uvloop==0.17.0
-    # via uvicorn
-watchfiles==0.19.0
-    # via uvicorn
-websockets==11.0.3
-    # via uvicorn
+# Core application
+fastapi[all]
+sqlmodel
+sqlalchemy
+pydantic
+pydantic_settings
+alembic
+aiosqlite
+
+# Fetch external resources
+httpx
+
+# Fetch title etc from links
+beautifulsoup4
+
+# Fetch favicons
+extract_favicon
+
+# Generate (atom) feeds for tags and such
+feedgen

src/__init__.py

@@ -0,0 +1 @@
+"""digimarks main module."""

src/digimarks/__init__.py

@@ -0,0 +1,3 @@
+"""Top-level package for Digimarks."""
+
+__author__ = """Michiel Scholten"""

src/digimarks/bookmarks_service.py

@@ -0,0 +1,214 @@
+"""Bookmark helper functions, like content scrapers, favicon extractor, updater functions."""
+
+import logging
+from collections.abc import Sequence
+from datetime import UTC, datetime
+from typing import Annotated
+from urllib.parse import urlparse, urlunparse
+
+import bs4
+import httpx
+from extract_favicon import from_html
+from fastapi import Query, Request
+from pydantic import AnyUrl
+from sqlmodel import select
+
+from digimarks import tags_service, utils
+from digimarks.exceptions import BookmarkNotFound
+from digimarks.models import Bookmark, Visibility
+
+DIGIMARKS_USER_AGENT = 'digimarks/2.0.0-dev'
+
+logger = logging.getLogger('digimarks')
+
+
+def get_favicon(html_content: str, root_url: str) -> str:
+    """Fetch the favicon from `html_content` using `root_url`."""
+    favicons = from_html(html_content, root_url=root_url, include_fallbacks=True)
+    for favicon in favicons:
+        print(favicon.url, favicon.width, favicon.height)
+    # TODO: save the preferred image to file and return
+
+
+async def set_information_from_source(bookmark: Bookmark, request: Request) -> Bookmark:
+    """Request the title by requesting the source url."""
+    logger.info('Extracting information from url %s', bookmark.url)
+    try:
+        result = await request.app.requests_client.get(bookmark.url, headers={'User-Agent': DIGIMARKS_USER_AGENT})
+        bookmark.http_status = result.status_code
+    except httpx.HTTPError as err:
+        # For example, "MissingSchema: Invalid URL 'abc': No schema supplied. Perhaps you meant http://abc?"
+        logger.error('Exception when trying to retrieve title for %s. Error: %s', bookmark.url, str(err))
+        bookmark.http_status = 404
+        bookmark.title = ''
+        return bookmark
+    if bookmark.http_status == 200 or bookmark.http_status == 202:
+        html = bs4.BeautifulSoup(result.text, 'html.parser')
+        try:
+            bookmark.title = html.title.text.strip()
+        except AttributeError:
+            bookmark.title = ''
+
+        url_parts = urlparse(str(bookmark.url))
+        root_url = url_parts.scheme + '://' + url_parts.netloc
+        favicon = get_favicon(result.text, root_url)
+        # filename = os.path.join(settings.media_dir, 'favicons/', domain + file_extension)
+        # with open(filename, 'wb') as out_file:
+        #     shutil.copyfileobj(response.raw, out_file)
+
+    # Extraction was successful
+    logger.info('Extracting information was successful')
+    return bookmark
+
+
+def strip_url_params(url: str) -> str:
+    """Strip URL params from URL.
+
+    :param url: URL to strip URL params from.
+    :return: clean URL
+    :rtype: str
+    """
+    parsed = urlparse(url)
+    return urlunparse((parsed.scheme, parsed.netloc, parsed.path, parsed.params, '', parsed.fragment))
+
+
+def update_bookmark_with_info(bookmark: Bookmark, request: Request, strip_params: bool = False):
+    """Automatically update title, favicon, etc."""
+    if not bookmark.title:
+        # Title was empty, automatically fetch it from the url, will also update the status code
+        set_information_from_source(bookmark, request)
+
+    if strip_params:
+        # Strip URL parameters, e.g., tracking params
+        bookmark.url = AnyUrl(strip_url_params(str(bookmark.url)))
+
+    # Sort and deduplicate tags
+    tags_service.set_tags(bookmark, bookmark.tags)
+
+
+async def list_bookmarks_for_user(
+    session,
+    user_key: str,
+    offset: int = 0,
+    limit: Annotated[int, Query(le=10000)] = 100,
+) -> Sequence[Bookmark]:
+    """List all bookmarks in the database. By default, 100 items are returned."""
+    result = await session.exec(
+        select(Bookmark)
+        .where(Bookmark.user_key == user_key, Bookmark.status != Visibility.DELETED)
+        .offset(offset)
+        .limit(limit)
+    )
+    bookmarks = result.all()
+    return bookmarks
+
+
+async def get_bookmark_for_user_with_url_hash(session, user_key: str, url_hash: str) -> Bookmark:
+    """Get a bookmark from the database by its URL hash."""
+    result = await session.exec(
+        select(Bookmark).where(
+            Bookmark.user_key == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
+        )
+    )
+    if not result.first():
+        raise BookmarkNotFound(f'url_hash: {url_hash}')
+    return result.first()
+
+
+async def autocomplete_bookmark(
+    session,
+    request: Request,
+    user_key: str,
+    bookmark: Bookmark,
+    strip_params: bool = False,
+):
+    """Autofill some fields for this (new) bookmark for user `user_key`."""
+    bookmark.user_key = user_key
+
+    # Auto-fill title, fix tags etc.
+    update_bookmark_with_info(bookmark, request, strip_params)
+
+    url_hash = utils.generate_hash(str(bookmark.url))
+    result = await session.exec(
+        select(Bookmark).where(
+            Bookmark.user_key == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
+        )
+    )
+    bookmark_db = result.first()
+    if bookmark_db:
+        # Bookmark with this URL already exists, provide the hash so the frontend can look it up and the user can
+        # merge them if so wanted
+        bookmark.url_hash = url_hash
+
+    return bookmark
+
+
+async def add_bookmark(
+    session,
+    request: Request,
+    user_key: str,
+    bookmark: Bookmark,
+    strip_params: bool = False,
+):
+    """Add new bookmark for user `user_key`."""
+    bookmark.user_key = user_key
+
+    # Auto-fill title, fix tags etc.
+    update_bookmark_with_info(bookmark, request, strip_params)
+    bookmark.url_hash = utils.generate_hash(str(bookmark.url))
+    logger.info('Adding bookmark %s for user %s', bookmark.url_hash, user_key)
+
+    session.add(bookmark)
+    await session.commit()
+    await session.refresh(bookmark)
+    return bookmark
+
+
+async def update_bookmark(
+    session,
+    request: Request,
+    user_key: str,
+    bookmark: Bookmark,
+    url_hash: str,
+    strip_params: bool = False,
+):
+    """Update existing bookmark `bookmark_key` for user `user_key`."""
+    result = await session.exec(
+        select(Bookmark).where(
+            Bookmark.user_key == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
+        )
+    )
+    bookmark_db = result.first()
+    if not bookmark_db:
+        raise BookmarkNotFound(message='Bookmark with hash {url_hash} not found')
+
+    bookmark.modified_date = datetime.now(UTC)
+
+    # 'patch' endpoint, which means that you can send only the data that you want to update, leaving the rest intact
+    bookmark_data = bookmark.model_dump(exclude_unset=True)
+    # Merge the changed fields into the existing object
+    bookmark_db.sqlmodel_update(bookmark_data)
+
+    # Autofill title, fix tags, etc. where (still) needed
+    update_bookmark_with_info(bookmark, request, strip_params)
+
+    session.add(bookmark_db)
+    await session.commit()
+    await session.refresh(bookmark_db)
+    return bookmark_db
+
+
+async def delete_bookmark(
+    session,
+    user_key: str,
+    url_hash: str,
+):
+    """(Soft)Delete bookmark `bookmark_key` for user `user_key`."""
+    result = await session.get(Bookmark, {'url_hash': url_hash, 'user_key': user_key})
+    bookmark = result
+    if not bookmark:
+        raise BookmarkNotFound(message='Bookmark with hash {url_hash} not found')
+    bookmark.deleted_date = datetime.now(UTC)
+    bookmark.status = Visibility.DELETED
+    session.add(bookmark)
+    await session.commit()

src/digimarks/exceptions.py

@@ -0,0 +1,25 @@
+"""Exceptions that could be encountered managing digimarks."""
+
+
+class BookmarkNotFound(Exception):
+    """A bookmark was not found."""
+
+    def __init__(self, message: str ='Bookmark not found'):
+        """Initialise the exception.
+
+        :param str message: The message for the exception
+        """
+        super().__init__(message)
+        self.message: str = message
+
+
+class BookmarkAlreadyExists(Exception):
+    """A bookmark already exists for this URL and this user."""
+
+    def __init__(self, message: str ='Bookmark already exists'):
+        """Initialise the exception.
+
+        :param str message: The message for the exception
+        """
+        super().__init__(message)
+        self.message: str = message

src/digimarks/main.py

@@ -1,32 +1,30 @@
 """digimarks main module."""
 
-import binascii
-import hashlib
 import logging
-import os
+from collections.abc import Sequence
 from contextlib import asynccontextmanager
 from datetime import UTC, datetime
-from http import HTTPStatus
-from typing import Annotated, Optional, Sequence, Type, TypeVar
-from urllib.parse import urlparse, urlunparse
+from typing import Annotated
 
-import bs4
 import httpx
-from extract_favicon import from_html
 from fastapi import Depends, FastAPI, HTTPException, Query, Request
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import HTMLResponse
 from fastapi.staticfiles import StaticFiles
 from fastapi.templating import Jinja2Templates
-from pydantic import AnyUrl, DirectoryPath, FilePath, computed_field
+from pydantic import DirectoryPath, FilePath
 from pydantic_settings import BaseSettings
-from sqlmodel import AutoString, Field, Session, SQLModel, create_engine, desc, select
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy.orm import sessionmaker
+from sqlmodel import desc, select
+from sqlmodel.ext.asyncio.session import AsyncSession
+
+from digimarks import bookmarks_service, tags_service
+from digimarks.exceptions import BookmarkNotFound
+from digimarks.models import DEFAULT_THEME, Bookmark, User, Visibility
 
-DIGIMARKS_USER_AGENT = 'digimarks/2.0.0-dev'
 DIGIMARKS_VERSION = '2.0.0a1'
 
-DEFAULT_THEME = 'freshgreen'
-
 
 class Settings(BaseSettings):
     """Configuration needed for digimarks to find its database, favicons, API integrations."""
@@ -36,8 +34,10 @@ class Settings(BaseSettings):
     favicons_dir: DirectoryPath
 
     # inside the codebase
-    static_dir: DirectoryPath = 'static'
-    template_dir: DirectoryPath = 'templates'
+    # static_dir: DirectoryPath = Path('digimarks/static')
+    # template_dir: DirectoryPath = Path('digimarks/templates')
+    static_dir: DirectoryPath = 'digimarks/static'
+    template_dir: DirectoryPath = 'digimarks/templates'
 
     media_url: str = '/static/'
 
@@ -49,16 +49,17 @@ class Settings(BaseSettings):
 settings = Settings()
 print(settings.model_dump())
 
-engine = create_engine(f'sqlite:///{settings.database_file}', connect_args={'check_same_thread': False})
+engine = create_async_engine(f'sqlite+aiosqlite:///{settings.database_file}', connect_args={'check_same_thread': False})
 
 
-def get_session():
+async def get_session() -> AsyncSession:
     """SQLAlchemy session factory."""
-    with Session(engine) as session:
+    async_session = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
+    async with async_session() as session:
         yield session
 
 
-SessionDep = Annotated[Session, Depends(get_session)]
+SessionDep = Annotated[AsyncSession, Depends(get_session)]
 
 
 @asynccontextmanager
@@ -74,6 +75,12 @@ app.mount('/static', StaticFiles(directory=settings.static_dir), name='static')
 app.mount('/content/favicons', StaticFiles(directory=settings.favicons_dir), name='favicons')
 templates = Jinja2Templates(directory=settings.template_dir)
 
+# Set up logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S',
+)
 logger = logging.getLogger('digimarks')
 if settings.debug:
     logger.setLevel(logging.DEBUG)
@@ -88,61 +95,6 @@ app.add_middleware(
 )
 
 
-def i_filter_false(predicate, iterable):
-    """Filter an iterable if predicate returns True.
-
-    i_filter_false(lambda x: x%2, range(10)) --> 0 2 4 6 8
-    """
-    if predicate is None:
-        predicate = bool
-    for x in iterable:
-        if not predicate(x):
-            yield x
-
-
-def unique_ever_seen(iterable, key=None):
-    """List unique elements, preserving order. Remember all elements ever seen.
-
-    unique_ever_seen('AAAABBBCCDAABBB') --> A B C D
-    unique_ever_seen('ABBCcAD', str.lower) --> A B C D
-    """
-    seen = set()
-    seen_add = seen.add
-    if key is None:
-        for element in i_filter_false(seen.__contains__, iterable):
-            seen_add(element)
-            yield element
-    else:
-        for element in iterable:
-            k = key(element)
-            if k not in seen:
-                seen_add(k)
-                yield element
-
-
-def clean_tags(tags_list: list) -> list:
-    """Generate a unique list of the tags.
-
-    :param list tags_list: List with all tags
-    :return: deduplicated list of the tags, without leading or trailing whitespace
-    :rtype: list
-    """
-    tags_res = [x.strip() for x in tags_list]
-    tags_res = list(unique_ever_seen(tags_res))
-    tags_res.sort()
-    if tags_res and tags_res[0] == '':
-        del tags_res[0]
-    return tags_res
-
-
-def list_tags_for_bookmarks(bookmarks: list) -> list:
-    """Generate a unique list of the tags from the list of bookmarks."""
-    tags = []
-    for bookmark in bookmarks:
-        tags += bookmark.tags_list
-    return clean_tags(tags)
-
-
 def file_type(filename: str) -> str:
     """Try to determine the file type for the file in `filename`.
 
@@ -162,194 +114,11 @@ def file_type(filename: str) -> str:
     return 'no match'
 
 
-def generate_hash(input_text: str) -> str:
-    """Generate a hash from string `input`, e.g., for a URL."""
-    return hashlib.md5(input_text.encode('utf-8')).hexdigest()
-
-
-def generate_key() -> str:
-    """Generate a key to be used for a user or tag."""
-    return str(binascii.hexlify(os.urandom(24)))
-
-
-# Type var used for building custom types for the DB
-T = TypeVar('T')
-
-
-def build_custom_type(internal_type: Type[T]) -> Type[AutoString]:
-    """Create a type that is compatible with the database.
-
-    Based on https://github.com/fastapi/sqlmodel/discussions/847
-    """
-
-    class CustomType(AutoString):
-        def process_bind_param(self, value, dialect) -> Optional[str]:
-            if value is None:
-                return None
-
-            if isinstance(value, str):
-                # Test if value is valid to avoid `process_result_value` failing
-                try:
-                    internal_type(value)  # type: ignore[call-arg]
-                except ValueError as e:
-                    raise ValueError(f'Invalid value for {internal_type.__name__}: {e}') from e
-
-            return str(value)
-
-        def process_result_value(self, value, dialect) -> Optional[T]:
-            if value is None:
-                return None
-
-            return internal_type(value)  # type: ignore[call-arg]
-
-    return CustomType
-
-
-def get_favicon(html_content: str, root_url: str) -> str:
-    """Fetch the favicon from `html_content` using `root_url`."""
-    favicons = from_html(html_content, root_url=root_url, include_fallbacks=True)
-    for favicon in favicons:
-        print(favicon.url, favicon.width, favicon.height)
-    # TODO: save the preferred image to file and return
-
-
-class User(SQLModel, table=True):
-    """User account."""
-
-    __tablename__ = 'user'
-
-    id: int = Field(primary_key=True)
-    username: str
-    key: str
-    theme: str = Field(default=DEFAULT_THEME)
-    created_date: datetime
-
-
-class Visibility:
-    """Options for visibility of an object."""
-
-    VISIBLE = 0
-    DELETED = 1
-
-
-class Bookmark(SQLModel, table=True):
-    """Bookmark object."""
-
-    __tablename__ = 'bookmark'
-
-    id: int = Field(primary_key=True)
-    userkey: str = Field(foreign_key='user.key')
-    title: str = Field(default='')
-    url: AnyUrl = Field(default='', sa_type=build_custom_type(AnyUrl))
-    note: str = Field(default='')
-    # image: str = Field(default='')
-    url_hash: str = Field(default='')
-    tags: str = Field(default='')
-    starred: bool = Field(default=False)
-
-    favicon: str | None = Field(default=None)
-
-    http_status: int = Field(default=HTTPStatus.OK)
-
-    created_date: datetime = Field(default=datetime.now(UTC))
-    modified_date: datetime = Field(default=None)
-    deleted_date: datetime = Field(default=None)
-
-    status: int = Field(default=Visibility.VISIBLE)
-
-    @computed_field
-    @property
-    def tag_list(self) -> list:
-        """The tags but as a proper list."""
-        if self.tags:
-            return self.tags.split(',')
-        # Not tags, return empty list instead of [''] that split returns in that case
-        return []
-
-
-async def set_information_from_source(bookmark: Bookmark, request: Request) -> Bookmark:
-    """Request the title by requesting the source url."""
-    logger.info('Extracting information from url %s', bookmark.url)
-    try:
-        result = await request.app.requests_client.get(bookmark.url, headers={'User-Agent': DIGIMARKS_USER_AGENT})
-        bookmark.http_status = result.status_code
-    except httpx.HTTPError as err:
-        # For example, 'MissingSchema: Invalid URL 'abc': No schema supplied. Perhaps you meant http://abc?'
-        logger.error('Exception when trying to retrieve title for %s. Error: %s', bookmark.url, str(err))
-        bookmark.http_status = 404
-        bookmark.title = ''
-        return bookmark
-    if bookmark.http_status == 200 or bookmark.http_status == 202:
-        html = bs4.BeautifulSoup(result.text, 'html.parser')
-        try:
-            bookmark.title = html.title.text.strip()
-        except AttributeError:
-            bookmark.title = ''
-
-        url_parts = urlparse(str(bookmark.url))
-        root_url = url_parts.scheme + '://' + url_parts.netloc
-        favicon = get_favicon(result.text, root_url)
-        # filename = os.path.join(settings.media_dir, 'favicons/', domain + file_extension)
-        # with open(filename, 'wb') as out_file:
-        #     shutil.copyfileobj(response.raw, out_file)
-
-    # Extraction was successful
-    logger.info('Extracting information was successful')
-    return bookmark
-
-
-def set_tags(bookmark: Bookmark, new_tags: str) -> None:
-    """Set tags from `tags`, strip and sort them.
-
-    :param Bookmark bookmark: Bookmark to modify
-    :param str new_tags: New tags to sort and set.
-    """
-    tags_split = new_tags.split(',')
-    tags_clean = clean_tags(tags_split)
-    bookmark.tags = ','.join(tags_clean)
-
-
-def strip_url_params(url: str) -> str:
-    """Strip URL params from URL.
-
-    :param url: URL to strip URL params from.
-    :return: clean URL
-    :rtype: str
-    """
-    parsed = urlparse(url)
-    return urlunparse((parsed.scheme, parsed.netloc, parsed.path, parsed.params, '', parsed.fragment))
-
-
-def update_bookmark_with_info(bookmark: Bookmark, request: Request, strip_params: bool = False):
-    """Automatically update title, favicon, etc."""
-    if not bookmark.title:
-        # Title was empty, automatically fetch it from the url, will also update the status code
-        set_information_from_source(bookmark, request)
-
-    if strip_params:
-        # Strip URL parameters, e.g., tracking params
-        bookmark.url = AnyUrl(strip_url_params(str(bookmark.url)))
-
-    # Sort and deduplicate tags
-    set_tags(bookmark, bookmark.tags)
-
-
-class PublicTag(SQLModel, table=True):
-    """Public tag object."""
-
-    __tablename__ = 'public_tag'
-
-    id: int = Field(primary_key=True)
-    tagkey: str
-    userkey: str = Field(foreign_key='user.key')
-    tag: str
-    created_date: datetime = Field(default=datetime.now(UTC))
-
-
 @app.get('/', response_class=HTMLResponse)
 @app.head('/', response_class=HTMLResponse)
 def index(request: Request):
     """Homepage, point visitors to project page."""
+    logger.info('Root page requested')
     return templates.TemplateResponse(
         request=request,
         name='index.html',
@@ -358,20 +127,24 @@ def index(request: Request):
 
 
 @app.get('/api/v1/admin/{system_key}/users/{user_id}', response_model=User)
-def get_user(session: SessionDep, system_key: str, user_id: int) -> Type[User]:
+async def get_user(session: SessionDep, system_key: str, user_id: int) -> type[User]:
     """Show user information."""
+    logger.info('User %d requested', user_id)
     if system_key != settings.system_key:
+        logger.error('User %s requested but incorrect system key %s provided', user_id, system_key)
         raise HTTPException(status_code=404)
 
-    user = session.get(User, user_id)
+    result = await session.get(User, user_id)
+    user = result
     if not user:
+        logger.error('User %s not found', user_id)
         raise HTTPException(status_code=404, detail='User not found')
     return user
 
 
 # @app.get('/admin/{system_key}/users/', response_model=list[User])
 @app.get('/api/v1/admin/{system_key}/users/')
-def list_users(
+async def list_users(
     session: SessionDep,
     system_key: str,
     offset: int = 0,
@@ -386,48 +159,44 @@ def list_users(
     :return: list of users in the system
     :rtype: list[User]
     """
+    logger.info('User listing requested')
     if system_key != settings.system_key:
+        logger.error('User listing requested but incorrect system key %s provided', system_key)
         raise HTTPException(status_code=404)
 
-    users = session.exec(select(User).offset(offset).limit(limit)).all()
-    return users
+    result = await session.exec(select(User).offset(offset).limit(limit))
+    return result.all()
 
 
 @app.get('/api/v1/{user_key}/bookmarks/')
-def list_bookmarks(
+async def list_bookmarks(
     session: SessionDep,
     user_key: str,
     offset: int = 0,
     limit: Annotated[int, Query(le=10000)] = 100,
-) -> list[Bookmark]:
-    """List all bookmarks in the database. By default 100 items are returned."""
-    bookmarks = session.exec(
-        select(Bookmark)
-        .where(Bookmark.userkey == user_key, Bookmark.status != Visibility.DELETED)
-        .offset(offset)
-        .limit(limit)
-    ).all()
-    return bookmarks
+) -> Sequence[Bookmark]:
+    """List all bookmarks in the database. By default, 100 items are returned."""
+    logger.info('List bookmarks for user %s with offset %d, limit %d', user_key, offset, limit)
+    return await bookmarks_service.list_bookmarks_for_user(session, user_key, offset, limit)
 
 
 @app.get('/api/v1/{user_key}/bookmarks/{url_hash}')
-def get_bookmark(
+async def get_bookmark(
     session: SessionDep,
     user_key: str,
     url_hash: str,
 ) -> Bookmark:
     """Show bookmark details."""
-    bookmark = session.exec(
-        select(Bookmark).where(
-            Bookmark.userkey == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
-        )
-    ).first()
-    # bookmark = session.get(Bookmark, {'url_hash': url_hash, 'userkey': user_key})
-    return bookmark
+    logger.info('Bookmark details for user %s with url_hash %s', user_key, url_hash)
+    try:
+        return await bookmarks_service.get_bookmark_for_user_with_url_hash(session, user_key, url_hash)
+    except BookmarkNotFound as exc:
+        logger.error('Bookmark not found: %s', exc)
+        raise HTTPException(status_code=404, detail=f'Bookmark not found: {exc.message}')
 
 
 @app.post('/api/v1/{user_key}/autocomplete_bookmark/', response_model=Bookmark)
-def autocomplete_bookmark(
+async def autocomplete_bookmark(
     session: SessionDep,
     request: Request,
     user_key: str,
@@ -435,27 +204,12 @@ def autocomplete_bookmark(
     strip_params: bool = False,
 ):
     """Autofill some fields for this (new) bookmark for user `user_key`."""
-    bookmark.userkey = user_key
-
-    # Auto-fill title, fix tags etc.
-    update_bookmark_with_info(bookmark, request, strip_params)
-
-    url_hash = generate_hash(str(bookmark.url))
-    bookmark_db = session.exec(
-        select(Bookmark).where(
-            Bookmark.userkey == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
-        )
-    ).first()
-    if bookmark_db:
-        # Bookmark with this URL already exists, provide the hash so the frontend can look it up and the user can
-        # merge them if so wanted
-        bookmark.url_hash = url_hash
-
-    return bookmark
+    logger.info('Autocompleting bookmark %s for user %s', bookmark.url_hash, user_key)
+    return await bookmarks_service.autocomplete_bookmark(session, request, user_key, bookmark, strip_params)
 
 
 @app.post('/api/v1/{user_key}/bookmarks/', response_model=Bookmark)
-def add_bookmark(
+async def add_bookmark(
     session: SessionDep,
     request: Request,
     user_key: str,
@@ -463,20 +217,12 @@ def add_bookmark(
     strip_params: bool = False,
 ):
     """Add new bookmark for user `user_key`."""
-    bookmark.userkey = user_key
-
-    # Auto-fill title, fix tags etc.
-    update_bookmark_with_info(bookmark, request, strip_params)
-    bookmark.url_hash = generate_hash(str(bookmark.url))
-
-    session.add(bookmark)
-    session.commit()
-    session.refresh(bookmark)
-    return bookmark
+    logger.info('Adding bookmark %s for user %s', bookmark.url, user_key)
+    return await bookmarks_service.add_bookmark(session, request, user_key, bookmark, strip_params)
 
 
 @app.patch('/api/v1/{user_key}/bookmarks/{url_hash}', response_model=Bookmark)
-def update_bookmark(
+async def update_bookmark(
     session: SessionDep,
     request: Request,
     user_key: str,
@@ -485,63 +231,49 @@ def update_bookmark(
     strip_params: bool = False,
 ):
     """Update existing bookmark `bookmark_key` for user `user_key`."""
-    bookmark_db = session.exec(
-        select(Bookmark).where(
-            Bookmark.userkey == user_key, Bookmark.url_hash == url_hash, Bookmark.status != Visibility.DELETED
-        )
-    ).first()
-    if not bookmark_db:
+    logger.info('Updating bookmark %s for user %s', url_hash, user_key)
+    try:
+        return await bookmarks_service.update_bookmark(session, request, user_key, bookmark, url_hash, strip_params)
+    except Exception:
+        logger.exception('Failed to update bookmark %s', bookmark.id)
         raise HTTPException(status_code=404, detail='Bookmark not found')
 
-    bookmark.modified_date = datetime.now(UTC)
-
-    # 'patch' endpoint, which means that you can send only the data that you want to update, leaving the rest intact
-    bookmark_data = bookmark.model_dump(exclude_unset=True)
-    # Merge the changed fields into the existing object
-    bookmark_db.sqlmodel_update(bookmark_data)
-
-    # Autofill title, fix tags, etc. where (still) needed
-    update_bookmark_with_info(bookmark, request, strip_params)
-
-    session.add(bookmark_db)
-    session.commit()
-    session.refresh(bookmark_db)
-    return bookmark_db
-
 
 @app.delete('/api/v1/{user_key}/bookmarks/{url_hash}', response_model=Bookmark)
-def delete_bookmark(
+async def delete_bookmark(
     session: SessionDep,
     user_key: str,
     url_hash: str,
 ):
     """(Soft)Delete bookmark `bookmark_key` for user `user_key`."""
-    bookmark = session.get(Bookmark, {'url_hash': url_hash, 'userkey': user_key})
-    if not bookmark:
+    logger.info('Deleting bookmark %s for user %s', url_hash, user_key)
+    try:
+        result = await bookmarks_service.delete_bookmark(session, user_key, url_hash)
+        return {'ok': True}
+    except Exception:
+        logger.exception('Failed to delete bookmark %s', url_hash)
         raise HTTPException(status_code=404, detail='Bookmark not found')
-    bookmark.deleted_date = datetime.now(UTC)
-    bookmark.status = Visibility.DELETED
-    session.add(bookmark)
-    session.commit()
-    return {'ok': True}
 
 
 @app.get('/api/v1/{user_key}/latest_changes/')
-def bookmarks_changed_since(
+async def bookmarks_changed_since(
     session: SessionDep,
     user_key: str,
 ):
     """Last update on server, so the (browser) client knows whether to fetch an update."""
-    latest_modified_bookmark = session.exec(
+    logger.info('Retrieving latest changes for user %s', user_key)
+    result = await session.exec(
         select(Bookmark)
-        .where(Bookmark.userkey == user_key, Bookmark.status != Visibility.DELETED)
+        .where(Bookmark.user_key == user_key, Bookmark.status != Visibility.DELETED)
         .order_by(desc(Bookmark.modified_date))
-    ).first()
-    latest_created_bookmark = session.exec(
+    )
+    latest_modified_bookmark = result.first()
+    result = await session.exec(
         select(Bookmark)
-        .where(Bookmark.userkey == user_key, Bookmark.status != Visibility.DELETED)
+        .where(Bookmark.user_key == user_key, Bookmark.status != Visibility.DELETED)
         .order_by(desc(Bookmark.created_date))
-    ).first()
+    )
+    latest_created_bookmark = result.first()
 
     latest_modification = max(latest_modified_bookmark.modified_date, latest_created_bookmark.created_date)
 
@@ -554,38 +286,34 @@ def bookmarks_changed_since(
 
 
 @app.get('/api/v1/{user_key}/tags/')
-def list_tags_for_user(
+async def list_tags_for_user(
     session: SessionDep,
     user_key: str,
 ) -> list[str]:
     """List all tags in use by the user."""
-    bookmarks = session.exec(
-        select(Bookmark).where(Bookmark.userkey == user_key, Bookmark.status != Visibility.DELETED)
-    ).all()
-    tags = []
-    for bookmark in bookmarks:
-        tags += bookmark.tag_list
-    return clean_tags(tags)
+    return await tags_service.list_tags_for_user(session, user_key)
 
 
 @app.get('/api/v1/{user_key}/tags/{tag_key}')
-def list_tags_for_user(
+async def list_bookmarks_for_tag_for_user(
     session: SessionDep,
     user_key: str,
+    tag_key: str,
 ) -> list[str]:
     """List all tags in use by the user."""
-    bookmarks = session.exec(select(Bookmark).where(Bookmark.userkey == user_key)).all()
-    return list_tags_for_bookmarks(bookmarks)
+    logger.info('List bookmarks for tag "%s" by user %s', tag_key, user_key)
+    return await tags_service.list_bookmarks_for_tag_for_user(session, user_key, tag_key)
 
 
 @app.get('/{user_key}', response_class=HTMLResponse)
-def page_user_landing(
+async def page_user_landing(
     session: SessionDep,
     request: Request,
     user_key: str,
 ):
     """HTML page with the main view for the user."""
-    user = session.exec(select(User).where(User.key == user_key)).first()
+    result = await session.exec(select(User).where(User.key == user_key))
+    user = result.first()
     if not user:
         raise HTTPException(status_code=404, detail='User not found')
     language = 'en'

src/digimarks/models.py

@@ -0,0 +1,107 @@
+"""Models for digimarks.
+
+Contains the bookmarks administration, users, tags, public tags and more.
+"""
+
+from datetime import UTC, datetime
+from http import HTTPStatus
+from typing import TypeVar
+
+from pydantic import AnyUrl, computed_field
+from sqlmodel import AutoString, Field, SQLModel
+
+DEFAULT_THEME = 'freshgreen'
+
+
+class User(SQLModel, table=True):
+    """User account."""
+
+    id: int = Field(primary_key=True)
+    username: str
+    key: str
+    theme: str = Field(default=DEFAULT_THEME)
+    created_date: datetime
+
+
+class Visibility:
+    """Options for visibility of an object."""
+
+    VISIBLE: int = 0
+    DELETED: int = 1
+    HIDDEN: int = 2
+
+
+# Type var used for building custom types for the DB
+T = TypeVar('T')
+
+
+def build_custom_type(internal_type: type[T]) -> type[AutoString]:
+    """Create a type that is compatible with the database.
+
+    Based on https://github.com/fastapi/sqlmodel/discussions/847
+    """
+
+    class CustomType(AutoString):
+        def process_bind_param(self, value, dialect) -> str | None:
+            if value is None:
+                return None
+
+            if isinstance(value, str):
+                # Test if value is valid to avoid `process_result_value` failing
+                try:
+                    internal_type(value)  # type: ignore[call-arg]
+                except ValueError as e:
+                    raise ValueError(f'Invalid value for {internal_type.__name__}: {e}') from e
+
+            return str(value)
+
+        def process_result_value(self, value, dialect) -> T | None:
+            if value is None:
+                return None
+
+            return internal_type(value)  # type: ignore[call-arg]
+
+    return CustomType
+
+
+class Bookmark(SQLModel, table=True):
+    """Bookmark object."""
+
+    id: int = Field(primary_key=True)
+    user_key: str = Field(foreign_key='user.key', nullable=False)
+    title: str = Field(default='', nullable=False)
+    url: AnyUrl = Field(default='', sa_type=build_custom_type(AnyUrl))
+    note: str = Field(default='', nullable=True)
+    # image: str = Field(default='')
+    url_hash: str = Field(default='', nullable=False)
+    tags: str = Field(default='')
+    starred: bool = Field(default=False)
+
+    favicon: str | None = Field(default=None)
+
+    http_status: int = Field(default=HTTPStatus.OK)
+
+    created_date: datetime = Field(default=datetime.now(UTC))
+    modified_date: datetime = Field(default=None, nullable=True)
+    deleted_date: datetime = Field(default=None, nullable=True)
+
+    status: int = Field(default=Visibility.VISIBLE)
+
+    @computed_field
+    @property
+    def tag_list(self) -> list[str]:
+        """The tags but as a proper list."""
+        if self.tags:
+            return self.tags.split(',')
+        # Not tags, return empty list instead of [''] that split returns in that case
+        return []
+
+
+class PublicTag(SQLModel, table=True):
+    """Public tag object."""
+
+    id: int = Field(primary_key=True)
+    tag_key: str
+    user_key: str = Field(foreign_key='user.key')
+    tag: str
+    created_date: datetime = Field(default=datetime.now(UTC))

src/digimarks/static/css/digimarks.css

@@ -22,4 +22,12 @@
 .thumbnail img {
     /*width: 72px;*/
     width: 60px;
-}
+}
+
+#bookmarkEditForm fieldset {
+    border: none;
+}
+
+#bookmarkEditForm fieldset input, #bookmarkEditForm textarea, #bookmarkEditForm select, #bookmarkEditForm label {
+    width: 100%;
+}

src/digimarks/static/css/digui.css

@@ -11,6 +11,8 @@
 :root {
     --padding: .5rem;
 
+    color-scheme: only light;
+
     /* Default is nebula-light */
     --font-family: sans-serif;
     --background-color: #fff;
@@ -28,6 +30,7 @@
     --border-color: #d5d9d9;
     --border-width: 1px;
     --border-radius: 8px;
+    --chip-border-radius: 2rem;
     --shadow-color: rgba(213, 217, 217, .5);
     --global-theme-toggle-content: ' 🌞';
 
@@ -57,6 +60,8 @@ html[data-theme='nebula'] {
 }
 
 html[data-theme='nebula-dark'] {
+    color-scheme: dark;
+
     --background-color: #29292c;
     --background-color-secondary: #29292c;
     --button-color: #29292c;
@@ -73,6 +78,7 @@ html[data-theme='nebula-dark'] {
     --border-color: #333;
     --border-width: 1px;
     --border-radius: 8px;
+    --chip-border-radius: 2rem;
     --shadow-color: rgba(3, 3, 3, .5);
     --global-theme-toggle-content: ' 🌝';
 }
@@ -95,6 +101,7 @@ html[data-theme='bbs'] {
     --border-color: #333;
     --border-width: 2px;
     --border-radius: 0;
+    --chip-border-radius: 0;
     --global-theme-toggle-content: ' 🖥️';
 }
 
@@ -120,6 +127,7 @@ html[data-theme='silo'] {
     /*--border-color: #003eaa;*/
     --border-width: 2px;
     --border-radius: 0;
+    --chip-border-radius: 0;
     --global-theme-toggle-content: ' ⌨️';
 }
 
@@ -155,6 +163,10 @@ header {
     height: 3rem;
     display: flex;
     align-items: center;
+}
+
+[data-theme='nebula'] header,
+[data-theme='nebula-dark'] header {
     /*box-shadow: 0 0 5px 0 rgba(213, 217, 217, .5);*/
     /*box-shadow: 0 0 5px 0 #999;*/
     box-shadow: 0 0 5px var(--shadow-color);
@@ -221,6 +233,7 @@ ol li::marker, ul li::marker {
 .active {
     background-color: var(--color-highlight);
     color: var(--text-color);
+    transition-duration: 0.2s;
 }
 
 /* Special button */
@@ -246,6 +259,7 @@ button, .button, input, select, textarea {
     -webkit-user-select: none;
     touch-action: manipulation;
     vertical-align: middle;
+    transition-duration: 0.2s;
 }
 
 button, .button, input, select, textarea, table {
@@ -259,6 +273,7 @@ button:hover, .button:hover {
     /*background-color: #d57803;*/
     background-color: var(--color-highlight);
     filter: brightness(80%);
+    transition-duration: 0.2s;
 }
 
 button:focus, .button:focus {
@@ -295,6 +310,22 @@ button:focus, .button:focus {
     filter: brightness(80%);
 }
 
+/* Toggle buttons */
+
+.button-group {
+    display: inline-flex;
+    overflow: hidden;
+    border: var(--border-width) solid var(--border-color);
+    border-radius: var(--border-radius);
+}
+
+.button-group button {
+    /* Reset borders because the buttons are mashed together and the group has its own border */
+    border: none;
+    border-radius: 0;
+    box-shadow: none !important;
+}
+
 /* Table */
 
 th {
@@ -322,16 +353,20 @@ th, td {
     /*padding: 1em;*/
 }
 
+[data-theme='nebula'] :modal,
 [data-theme='nebula'] .card,
 [data-theme='nebula'] button,
 [data-theme='nebula'] .button,
+[data-theme='nebula'] .button-group,
 [data-theme='nebula'] input,
 [data-theme='nebula'] select,
 [data-theme='nebula'] textarea,
 [data-theme='nebula'] table,
+[data-theme='nebula-dark'] :modal,
 [data-theme='nebula-dark'] .card,
 [data-theme='nebula-dark'] button,
 [data-theme='nebula-dark'] .button,
+[data-theme='nebula-dark'] .button-group,
 [data-theme='nebula-dark'] input,
 [data-theme='nebula-dark'] select,
 [data-theme='nebula-dark'] textarea,
@@ -409,7 +444,7 @@ th, td {
 
 .chip {
     font-size: .8rem;
-    border-radius: var(--border-radius);
+    border-radius: var(--chip-border-radius);
     background-color: var(--background-color-secondary);
     color: var(--text-color-secondary);
     /*color: var(--text-color);*/
@@ -418,7 +453,7 @@ th, td {
 }
 
 .chip .button {
-    border-radius: var(--border-radius);
+    border-radius: var(--chip-border-radius);
 }
 
 /* Status */
@@ -427,6 +462,24 @@ th, td {
 }
 
 
+/** Modal, e.g. for showing info or filling in a form; on top of the other content */
+
+dialog:modal {
+    color: var(--text-color);
+    background-color: var(--background-color);
+    /*background-color: var(--nav-background-color);*/
+    border: 2px solid var(--border-color);
+    border-radius: var(--border-radius);
+    width: 90%;
+    /*height: 90%;*/
+    overflow: auto;
+}
+
+/* The umwelt of the modal, on top of the regular content */
+dialog::backdrop {
+    background-color: rgba(0, 0, 0, 0.5);
+}
+
 /** Footer */
 
 footer {

src/digimarks/static/js/digimarks.js

@@ -1,24 +1,24 @@
 document.addEventListener('alpine:init', () => {
     Alpine.store('digimarks', {
         /** Main digimarks application, state etc */
-        // userKey: Alpine.$persist(0).as('userKey'),
         userKey: -1,
         /* cache consists of cache[userKey] = {'bookmarks': [], 'tags': [], ??} */
         cache: Alpine.$persist({}).as('cache'),
 
         bookmarks: [],
 
-        /* Bookmark that is being edited, used to fill the form etc */
-        bookmark_to_edit: null,
-
-        /* nebula (dropshadows), bbs (monospace, right lines), silo (like bbs but dark) ?? */
+        /* nebula (drop-shadows), bbs (monospace, right lines), silo (like bbs but dark) ?? */
         themes: ['nebula', 'nebula-dark', 'bbs', 'silo'],
         theme: Alpine.$persist('nebula').as('theme'),
 
-        show_bookmarks: Alpine.$persist(true).as('show_bookmarks'),
-        show_bookmarks_list: Alpine.$persist(true).as('show_bookmarks_list'),
-        show_bookmarks_cards: Alpine.$persist(false).as('show_bookmarks_cards'),
-        show_tags: Alpine.$persist(false).as('show_tags'),
+        showBookmarks: Alpine.$persist(true).as('showBookmarks'),
+        showBookmarksList: Alpine.$persist(true).as('showBookmarksList'),
+        showBookmarksCards: Alpine.$persist(false).as('showBookmarksCards'),
+        showTags: Alpine.$persist(false).as('showTags'),
+        /* Bookmark that is being edited, used to fill the form, etc. */
+        bookmarkToEdit: Alpine.$persist({}).as('bookmarkToEdit'),
+        bookmarkToEditError: null,
+        bookmarkToEditVisible: false,
 
         /* Loading indicator */
         loading: false,
@@ -26,20 +26,23 @@ document.addEventListener('alpine:init', () => {
         /* Search filter */
         search: '',
         /* Show bookmarks with this tag/these tags */
-        tags_filter: [],
+        tagsFilter: [],
         /* Hide bookmarks with these tags */
-        tags_to_hide: Alpine.$persist([]).as('tags_to_hide'),
+        tagsToHide: Alpine.$persist([]).as('tags_to_hide'),
 
         /* Sort on ~ */
-        sort_title_asc: Alpine.$persist(false).as('sort_title_asc'),
-        sort_title_desc: Alpine.$persist(false).as('sort_title_desc'),
-        sort_created_asc: Alpine.$persist(false).as('sort_created_asc'),
-        sort_created_desc: Alpine.$persist(false).as('sort_created_desc'),
+        sortTitleAsc: Alpine.$persist(false).as('sortTitleAsc'),
+        sortTitleDesc: Alpine.$persist(false).as('sortTitleDesc'),
+        sortCreatedAsc: Alpine.$persist(false).as('sortCreatedAsc'),
+        sortCreatedDesc: Alpine.$persist(false).as('sortCreatedDesc'),
 
         async init() {
             /** Initialise the application after loading */
             document.documentElement.setAttribute('data-theme', this.theme);
             console.log('Set theme', this.theme);
+
+            /* Make sure the edit/add bookmark form has a fresh empty object */
+            this.resetEditBookmark();
             /* Bookmarks are refreshed through the getBookmarks() call in the HTML page */
             /* await this.getBookmarks(); */
             setInterval(() => {
@@ -79,14 +82,14 @@ document.addEventListener('alpine:init', () => {
                 this.cache[this.userKey] = {'bookmarks': [], 'latest_changes': {}};
             }
 
-            let latest_status_response = await fetch('/api/v1/' + this.userKey + '/latest_changes/');
-            let latest_status_result = await latest_status_response.json();
-            let should_fetch = false;
-            let latest_modification_in_cache = this.cache[this.userKey].latest_changes.latest_modification || "0000-00-00";
-            should_fetch = latest_status_result.latest_modification > latest_modification_in_cache;
-            this.cache[this.userKey].latest_changes = latest_status_result;
+            let latestStatusResponse = await fetch('/api/v1/' + this.userKey + '/latest_changes/');
+            let latestStatusResult = await latestStatusResponse.json();
+            let shouldFetch = false;
+            let latestModificationInCache = this.cache[this.userKey].latest_changes.latest_modification || "0000-00-00";
+            shouldFetch = latestStatusResult.latest_modification > latestModificationInCache;
+            this.cache[this.userKey].latest_changes = latestStatusResult;
 
-            if (!should_fetch) {
+            if (!shouldFetch) {
                 console.log('Cache is up-to-date');
                 this.loading = false;
                 return;
@@ -98,8 +101,8 @@ document.addEventListener('alpine:init', () => {
             /* Cache the bookmarks to Local Storage */
             this.cache[this.userKey]['bookmarks'] = await response.json();
 
-            let tags_response = await fetch('/api/v1/' + this.userKey + '/tags/');
-            this.cache[this.userKey]['tags'] = await tags_response.json();
+            let tagsResponse = await fetch('/api/v1/' + this.userKey + '/tags/');
+            this.cache[this.userKey]['tags'] = await tagsResponse.json();
 
             /* Filter bookmarks by (blacklisted) tags */
             await this.filterBookmarksByTags();
@@ -121,14 +124,14 @@ document.addEventListener('alpine:init', () => {
             /* Filter away bookmarks with a tag on the 'blacklist' */
 
             /* First make a shallow copy of all bookmarks */
-            let prefiltered_bookmarks = [...this.cache[this.userKey]['bookmarks']] || [];
-            if (this.tags_to_hide.length > 0) {
+            let prefilteredBookmarks = [...this.cache[this.userKey]['bookmarks']] || [];
+            if (this.tagsToHide.length > 0) {
                 console.log('Filtering away bookmarks containing blacklisted tags');
-                this.bookmarks = prefiltered_bookmarks.filter(
-                    i => !this.hasTag(i.tag_list, this.tags_to_hide)
+                this.bookmarks = prefilteredBookmarks.filter(
+                    i => !this.hasTag(i.tag_list, this.tagsToHide)
                 )
             } else {
-                this.bookmarks = prefiltered_bookmarks;
+                this.bookmarks = prefilteredBookmarks;
             }
             this.sortBookmarks();
         },
@@ -145,6 +148,10 @@ document.addEventListener('alpine:init', () => {
             )
         },
         get filteredTags() {
+            if (this.cache[this.userKey].tags === undefined) {
+                console.log('Tags not yet cached');
+                return [];
+            }
             /* Search in the list of all tags */
             return this.cache[this.userKey].tags.filter(
                 i => i.match(new RegExp(this.search, "i"))
@@ -153,27 +160,27 @@ document.addEventListener('alpine:init', () => {
 
         sortBookmarks() {
             /* Sort the bookmarks according to the setting */
-            if (this.sort_title_asc) {
+            if (this.sortTitleAsc) {
                 this.bookmarks.sort((a, b) => a.title.localeCompare(b.title));
-            } else if (this.sort_title_desc) {
+            } else if (this.sortTitleDesc) {
                 this.bookmarks.sort((a, b) => b.title.localeCompare(a.title));
-            } else if (this.sort_created_asc) {
+            } else if (this.sortCreatedAsc) {
                 this.bookmarks.sort((a, b) => a.created_date.localeCompare(b.created_date));
-            } else if (this.sort_created_desc) {
+            } else if (this.sortCreatedDesc) {
                 this.bookmarks.sort((a, b) => b.created_date.localeCompare(a.created_date));
             }
         },
         async sortAlphabetically(order = 'asc') {
             /* Sort the bookmarks (reverse) alphabetically, based on 'asc' or 'desc' */
             this.loading = true;
-            this.sort_created_asc = false;
-            this.sort_created_desc = false;
-            this.sort_title_asc = false;
-            this.sort_title_desc = false;
+            this.sortCreatedAsc = false;
+            this.sortCreatedDesc = false;
+            this.sortTitleAsc = false;
+            this.sortTitleDesc = false;
             if (order === 'desc') {
-                this.sort_title_desc = true;
+                this.sortTitleDesc = true;
             } else {
-                this.sort_title_asc = true;
+                this.sortTitleAsc = true;
             }
             this.sortBookmarks();
             this.loading = false;
@@ -181,14 +188,14 @@ document.addEventListener('alpine:init', () => {
         async sortCreated(order = 'asc') {
             /* Sort the bookmarks (reverse) chronologically, based on 'asc' or 'desc' */
             this.loading = true;
-            this.sort_created_asc = false;
-            this.sort_created_desc = false;
-            this.sort_title_asc = false;
-            this.sort_title_desc = false;
+            this.sortCreatedAsc = false;
+            this.sortCreatedDesc = false;
+            this.sortTitleAsc = false;
+            this.sortTitleDesc = false;
             if (order === 'desc') {
-                this.sort_created_desc = true;
+                this.sortCreatedDesc = true;
             } else {
-                this.sort_created_asc = true;
+                this.sortCreatedAsc = true;
             }
             this.sortBookmarks();
             this.loading = false;
@@ -196,23 +203,64 @@ document.addEventListener('alpine:init', () => {
 
         async toggleTagPage() {
             /* Show or hide the tag page instead of the bookmarks */
-            console.log('Toggle tag page');
-            this.show_bookmarks = !this.show_bookmarks;
-            this.show_tags = !this.show_bookmarks;
+            this.showBookmarks = !this.showBookmarks;
+            this.showTags = !this.showBookmarks;
         },
         async toggleListOrGrid() {
             /* Toggle between 'list' or 'grid' (cards) view */
-            this.show_bookmarks_list = !this.show_bookmarks_list;
-            this.show_bookmarks_cards = !this.show_bookmarks_list;
+            this.showBookmarksList = !this.showBookmarksList;
+            this.showBookmarksCards = !this.showBookmarksList;
         },
 
+        resetEditBookmark() {
+            this.bookmarkToEdit = {
+                'url': '',
+                'title': '',
+                'note': '',
+                'tags': ''
+            }
+        },
         async startAddingBookmark() {
             /* Open 'add bookmark' page */
             console.log('Start adding bookmark');
-            this.bookmark_to_edit = {
-                'url': ''
+            this.resetEditBookmark();
+            // this.show_bookmark_details = true;
+            const editFormDialog = document.getElementById("editFormDialog");
+            this.bookmarkToEditVisible = true;
+            editFormDialog.showModal();
+        },
+        async bookmarkURLChanged() {
+            console.log('Bookmark URL changed');
+            // let response = await fetch('/api/v1/' + this.userKey + '/autocomplete_bookmark/');
+            try {
+                const response = await fetch('/api/v1/' + this.userKey + '/autocomplete_bookmark/', {
+                    method: 'POST',
+                    headers: {
+                        'Content-Type': 'application/json'
+                    },
+                    body: JSON.stringify({
+                        // Bookmark form data
+                        url: this.bookmarkToEdit.url,
+                        title: this.bookmarkToEdit.title,
+                        note: this.bookmarkToEdit.note,
+                        tags: this.bookmarkToEdit.tags
+                    })
+                });
+                const data = await response.json();
+                // TODO: update form fields if needed (auto-fetched title for example
+                console.log(data);
+                this.bookmarkToEditError = 'lolwut';
+            } catch (error) {
+                // enter your logic for when there is an error (ex. error toast)
+
+                console.log(error)
             }
         },
+        async saveBookmark() {
+            console.log('Saving bookmark');
+            // this.bookmarkToEditVisible = false;
+            // this.show_bookmark_details = false;
+        },
         async addBookmark() {
             /* Post new bookmark to the backend */
             //

src/digimarks/tags_service.py

@@ -0,0 +1,102 @@
+"""Helper functions for tags used with Bookmark models."""
+
+from sqlalchemy import Sequence
+from sqlmodel import select
+
+from digimarks.models import Bookmark, Visibility
+
+
+def i_filter_false(predicate, iterable):
+    """Filter an iterable if predicate returns True.
+
+    i_filter_false(lambda x: x%2, range(10)) --> 0 2 4 6 8
+    """
+    if predicate is None:
+        predicate = bool
+    for x in iterable:
+        if not predicate(x):
+            yield x
+
+
+def unique_ever_seen(iterable, key=None):
+    """List unique elements, preserving order. Remember all elements ever seen.
+
+    unique_ever_seen('AAAABBBCCDAABBB') --> A B C D
+    unique_ever_seen('ABBCcAD', str.lower) --> A B C D
+    """
+    seen = set()
+    seen_add = seen.add
+    if key is None:
+        for element in i_filter_false(seen.__contains__, iterable):
+            seen_add(element)
+            yield element
+    else:
+        for element in iterable:
+            k = key(element)
+            if k not in seen:
+                seen_add(k)
+                yield element
+
+
+def clean_tags(tags_list: list) -> list[str]:
+    """Generate a unique list of the tags.
+
+    :param list tags_list: List with all tags
+    :return: deduplicated list of the tags, without leading or trailing whitespace
+    :rtype: list
+    """
+    tags_res = [x.strip() for x in tags_list]
+    tags_res = list(unique_ever_seen(tags_res))
+    tags_res.sort()
+    if tags_res and tags_res[0] == '':
+        del tags_res[0]
+    return tags_res
+
+
+def list_tags_for_bookmarks(bookmarks: Sequence[Bookmark]) -> list[str]:
+    """Generate a unique list of the tags from the list of bookmarks.
+
+    :param Sequence[Bookmark] bookmarks: List of bookmarks to create the list of tags from
+    """
+    tags = []
+    for bookmark in bookmarks:
+        tags += bookmark.tag_list
+    return clean_tags(tags)
+
+
+def set_tags(bookmark: Bookmark, new_tags: str) -> None:
+    """Set tags from `tags`, strip and sort them.
+
+    :param Bookmark bookmark: Bookmark to modify
+    :param str new_tags: New tags to sort and set.
+    """
+    tags_split = new_tags.split(',')
+    tags_clean = clean_tags(tags_split)
+    bookmark.tags = ','.join(tags_clean)
+
+
+async def list_tags_for_user(
+    session,
+    user_key: str,
+) -> list[str]:
+    """List all tags in use by the user."""
+    result = await session.exec(
+        select(Bookmark).where(Bookmark.user_key == user_key, Bookmark.status != Visibility.DELETED)
+    )
+    bookmarks = result.all()
+    tags = []
+    for bookmark in bookmarks:
+        tags += bookmark.tag_list
+    return clean_tags(tags)
+
+
+async def list_bookmarks_for_tag_for_user(
+    session,
+    user_key: str,
+    tag_key: str,
+) -> list[str]:
+    """List all tags in use by the user."""
+    result = await session.exec(select(Bookmark).where(Bookmark.user_key == user_key))
+    # TODO: filter on tag_key
+    bookmarks = result.all()
+    return list_tags_for_bookmarks(bookmarks)

src/digimarks/templates/user_index.html

@@ -11,9 +11,14 @@
                 <ul>
                     <li><h1>digimarks</h1></li>
                     <li>
-                        <button x-data @click="$store.digimarks.toggleTagPage()"
-                                :class="$store.digimarks.show_tags && 'active'">tags
-                        </button>
+                        <div class="button-group">
+                            <button x-data @click="$store.digimarks.toggleTagPage()"
+                                    :class="!$store.digimarks.showTags && 'active'">bookmarks
+                            </button>
+                            <button x-data @click="$store.digimarks.toggleTagPage()"
+                                    :class="$store.digimarks.showTags && 'active'">tags
+                            </button>
+                        </div>
                     </li>
                     <li>
                         <button @click="$store.digimarks.startAddingBookmark()">add bookmark</button>
@@ -28,29 +33,35 @@
         </header>
 
         <main>
-            <section x-cloak x-show="$store.digimarks.show_bookmarks && !$store.digimarks.bookmark_to_edit"
-                     x-transition.opacity>
+            <section x-cloak x-show="$store.digimarks.showBookmarks" x-transition.opacity>
                 <h1 x-bind:title="$store.digimarks.userKey">Bookmarks</h1>
 
                 <p>
+                <div class="button-group">
                     <button @click="$store.digimarks.sortAlphabetically()"
-                            :class="$store.digimarks.sort_title_asc && 'active'">a-z &darr;
+                            :class="$store.digimarks.sortTitleAsc && 'active'">a-z &darr;
                     </button>
                     <button @click="$store.digimarks.sortAlphabetically('desc')"
-                            :class="$store.digimarks.sort_title_desc && 'active'">z-a &uarr;
+                            :class="$store.digimarks.sortTitleDesc && 'active'">z-a &uarr;
                     </button>
                     <button @click="$store.digimarks.sortCreated()"
-                            :class="$store.digimarks.sort_created_asc && 'active'">date &darr;
+                            :class="$store.digimarks.sortCreatedAsc && 'active'">date &darr;
                     </button>
                     <button @click="$store.digimarks.sortCreated('desc')"
-                            :class="$store.digimarks.sort_created_desc && 'active'">date &uarr;
+                            :class="$store.digimarks.sortCreatedDesc && 'active'">date &uarr;
+                    </button>
+                </div>
+                <div class="button-group">
+                    <button @click="$store.digimarks.toggleListOrGrid()"
+                            :class="$store.digimarks.showBookmarksCards && 'active'">grid
                     </button>
                     <button @click="$store.digimarks.toggleListOrGrid()"
-                            :class="$store.digimarks.show_bookmarks_cards && 'active'">list or grid
+                            :class="!$store.digimarks.showBookmarksCards && 'active'">list
                     </button>
+                </div>
                 </p>
 
-                <table x-cloak x-show="$store.digimarks.show_bookmarks_list">
+                <table x-cloak x-show="$store.digimarks.showBookmarksList">
                     <thead>
                     <tr>
                         <th colspan="2">&nbsp;</th>
@@ -92,22 +103,25 @@
                     </template>
                 </ul>
 #}
-                <section x-cloak x-show="$store.digimarks.show_bookmarks_cards" class="cards">
+                <section x-cloak x-show="$store.digimarks.showBookmarksCards" class="cards">
                     <template x-for="bookmark in $store.digimarks.filteredBookmarks" :key="bookmark.id">
                         <div class="card">
                             <div class="card-body">
                                 <div class="card-thumb" x-show="bookmark.favicon"><img
                                         x-bind:src="'/content/favicons/' + bookmark.favicon"></div>
                                 <div class="statuses">
-                                    <div x-show="bookmark.starred" class="star"><i class="fa-fw fa-solid fa-star"></i>
+                                    <div x-show="bookmark.starred" class="star"><i
+                                            class="fa-fw fa-solid fa-star"></i>
                                     </div>
                                     <div x-show="bookmark.http_status !== 200 && bookmark.http_status !== 304"
                                          class="error"><i
                                             class="fa-fw fa-solid fa-triangle-exclamation"></i>
                                     </div>
-                                    <div x-show="bookmark.note"><i class="fa-fw fa-regular fa-note-sticky"></i></div>
+                                    <div x-show="bookmark.note"><i class="fa-fw fa-regular fa-note-sticky"></i>
+                                    </div>
+                                </div>
+                                <div><a x-text="bookmark.title" x-bind:href="bookmark.url" target="_blank"></a>
                                 </div>
-                                <div><a x-text="bookmark.title" x-bind:href="bookmark.url" target="_blank"></a></div>
                             </div>
                             <div class="card-footer">
                                 <button title="show actions"><i class="fa-solid fa-square-caret-down"></i></button>
@@ -123,8 +137,7 @@
                 </section>
             </section>
 
-            <section x-cloak x-show="$store.digimarks.show_tags && !$store.digimarks.bookmark_to_edit"
-                     x-transition.opacity>
+            <section x-cloak x-show="$store.digimarks.showTags" x-transition.opacity>
                 <h1>Tags</h1>
 
                 <table>
@@ -147,7 +160,13 @@
                 </table>
             </section>
 
-            <section x-cloak x-show="$store.digimarks.bookmark_to_edit" x-transition.opacity>
+            <dialog x-cloak id="editFormDialog"
+                    x-transition:enter="modal-enter"
+                    x-transition:enter-start="modal-enter"
+                    x-transition:enter-end="modal-enter-active"
+                    x-transition:leave="modal-leave-active"
+                    x-transition:leave-start="modal-enter-active"
+                    x-transition:leave-end="modal-enter">
                 <h1>Add/Edit bookmark</h1>
                 {#
                 <div class="card-panel {{ theme.ERRORMESSAGE_BACKGROUND }}">
@@ -166,14 +185,47 @@
             </span>
                 </div>
 #}
-                <form>
-                    <input type="text" name="">
-                    <label>
-                        <input type="checkbox" name="strip" id="strip"/>
-                        <span>Strip parameters from url (like <em>?utm_source=social</em> - can break the link!)</span>
-                    </label>
-                </form>
-            </section>
+                <template x-if="$store.digimarks.bookmarkToEditVisible">
+                    <form method="dialog" id="bookmarkEditForm">
+                        <fieldset class="form-group">
+                            <label for="bookmark_url">URL</label>
+                            <input id="bookmark_url" type="text" name="bookmark_url" placeholder="url"
+                                   x-on:change.debounce="$store.digimarks.bookmarkURLChanged()"
+                                   x-model="$store.digimarks.bookmarkToEdit.url">
+                        </fieldset>
+                        <fieldset class="form-group">
+                            <label for="bookmark_title">Title</label>
+                            <input id="bookmark_title" type="text" name="bookmark_title"
+                                   placeholder="title (leave empty for autofetch)"
+                                   x-model="$store.digimarks.bookmarkToEdit.title">
+                        </fieldset>
+                        <fieldset class="form-group">
+                            <label for="bookmark_note">Note</label>
+                            <textarea id="bookmark_note" type="text" name="bookmark_note"
+                                      x-model="$store.digimarks.bookmarkToEdit.note">
+                        </textarea>
+                        </fieldset>
+                        <fieldset class="form-group">
+                            <label for="bookmark_tags">Tags</label>
+                            <input id="bookmark_tags" type="text" name="bookmark_tags"
+                                   placeholder="tags, divided bij comma's"
+                                   x-model="$store.digimarks.bookmarkToEdit.tags">
+                        </fieldset>
+                        <p x-show="$store.digimarks.bookmarkToEditError"
+                           x-data="$store.digimarks.bookmarkToEditError"></p>
+                        <p>
+                            <label>
+                                <input type="checkbox" name="strip" id="strip"/>
+                                <span>Strip parameters from url (like <em>?utm_source=social</em> - can break the link!)</span>
+                            </label>
+                        </p>
+                        <div>
+                            <button value="cancel">Cancel</button>
+                            <button @click="$store.digimarks.saveBookmark()">Save</button>
+                        </div>
+                    </form>
+                </template>
+            </dialog>
         </main>
 
     </article>

src/digimarks/utils.py

@@ -0,0 +1,15 @@
+"""General utility functions."""
+
+import binascii
+import hashlib
+import os
+
+
+def generate_hash(input_text: str) -> str:
+    """Generate a hash from string `input`, e.g., for a URL."""
+    return hashlib.md5(input_text.encode('utf-8')).hexdigest()
+
+
+def generate_key() -> str:
+    """Generate a key to be used for a user or tag."""
+    return str(binascii.hexlify(os.urandom(24)))