Running pykrete

A deployment has five parts: PostgreSQL, a PKCS#11 token holding the signing keys, the pykrete binary (the backend), the freshfront Deno server, and a TLS reverse proxy in front of the backend. This page goes through them in order. Every flag and key used here is described in Configuration.

Requirements

  • Linux.

  • PostgreSQL.

  • immigrant to apply the database migrations that ship with a release.

  • A PKCS#11 module: SoftHSM2 (softhsm2-util) for a software token, or the module of your HSM or smartcard.

  • Deno 2 to run freshfront.

  • A TLS reverse proxy, for example nginx.

Database

Create a database and a role for pykrete:

sudo -u postgres createuser pykrete
sudo -u postgres createdb -O pykrete pykrete
export DATABASE_URL=postgres://pykrete@%2Frun%2Fpostgresql/pykrete
# or over TCP: postgres://pykrete:<password>@127.0.0.1/pykrete

Applying migrations

The backend does not migrate the database on start. Each release ships its migrations in a migrations/ directory; apply the pending ones before starting a new release (immigrant finds migrations/ from the current directory):

cd /path/to/release
immigrant migrate --dry-run apply   # run everything in a transaction, then roll back
immigrant migrate apply

Applied migrations are recorded in the __immigrant_migrations table. apply stops if an already applied migration differs from the one on disk.

Keystore

The backend refuses to start without a PKCS#11 token. For a software token, initialize SoftHSM2 once and note the slot number it prints; the details and the HSM/smartcard variant are in Keys and secrets.

export SOFTHSM2_CONF=/var/lib/pykrete/softhsm2.conf
mkdir -p /var/lib/pykrete/tokens
echo "directories.tokendir = /var/lib/pykrete/tokens" > "$SOFTHSM2_CONF"
softhsm2-util --init-token --free --label pykrete --pin <user-pin> --so-pin <so-pin>
softhsm2-util --show-slots

Running

Backend

export DATABASE_URL=postgres://pykrete@%2Frun%2Fpostgresql/pykrete
export SOFTHSM2_CONF=/var/lib/pykrete/softhsm2.conf
export PYKRETE_PKCS11_MODULE=/usr/lib/softhsm/libsofthsm2.so
export PYKRETE_PKCS11_SLOT=<slot from softhsm2-util --show-slots>
export PYKRETE_PKCS11_PIN=<user-pin>
export RUST_LOG=info

pykrete --config /etc/pykrete/config.json

The module path depends on your distribution. The backend listens on 127.0.0.1:3000 unless listen is set in the config file. Startup takes a moment because the session key is derived with 600000 PBKDF2 iterations. It exits if the config file is missing or invalid, the PKCS#11 variables are missing or the slot does not exist, or the database is unreachable.

freshfront

cd freshfront
FRESHFRONT_SOCKET=/run/pykrete/freshfront.sock \
AUTH_BACKEND=http://127.0.0.1:3000 \
HOOKS_SECRET=<hooks-secret> \
deno task start

Point the backend at it with "frontend": {"upstream": "unix:/run/pykrete/freshfront.sock", ...} in the config file. HOOKS_SECRET must equal frontend.hooks_secret.

Important
Use FRESHFRONT_SOCKET in production. Without it, freshfront listens on TCP port 8000 on all interfaces; if you run it that way, make sure port 8000 is not reachable from outside, so that all traffic goes through the backend.

Reverse proxy and TLS

The backend speaks plain HTTP and never terminates TLS. Put a TLS terminator in front of it and:

  • Pass every path to 127.0.0.1:3000 (or the configured listen address). The backend decides what it handles and proxies the rest to freshfront; routing pages straight to freshfront bypasses that.

  • Set X-Real-IP to the client's address, replacing any value the client sent. The backend trusts this header as is and shows the address to users approving backchannel (CIBA) requests. Without it the address shows as unknown.

  • Allow WebSocket upgrades. Sign-in, consent and logout run over /control.

  • Allow request bodies of at least 2 MiB plus 64 KiB on /graphql (picture uploads).

  • Set domain in the config file to the public host name (with the port, if it is not the default) and tls_terminated to true. Issuer, endpoint URLs, the WebAuthn origin and the internal clients' redirect URIs are built from these two keys, not from the Host header. freshfront marks its cookies Secure when the issuer is https://.

  • Expose LDAP (if enabled) as its own TCP port. pykrete terminates TLS for LDAP itself, so don't put it behind the HTTP proxy.

nginx:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl;
    server_name auth.infra.delta.rocks;
    ssl_certificate     /etc/ssl/auth.infra.delta.rocks/fullchain.pem;
    ssl_certificate_key /etc/ssl/auth.infra.delta.rocks/privkey.pem;
    client_max_body_size 4m;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 300s;
    }
}

Check it from outside:

curl -s https://auth.infra.delta.rocks/.well-known/openid-configuration | jq .issuer
# "https://auth.infra.delta.rocks/"

What happens on startup

Each start runs these steps, in this order. All of them are idempotent.

  1. Read and parse the config file, derive the session key from session_secret.

  2. Connect to PostgreSQL, open the PKCS#11 token.

  3. Signing keys: delete the public keys in the jwk_keys table that have no authority, generate oidc-token-signing-rs256, oidc-token-signing-es256 and oidc-refresh-encrypt-es256 in the token if they don't exist yet, and store their public JWKs in jwk_keys again (served at /jwks).

  4. Self audience A____________________________SELF: identifier set to the issuer (https://auth.infra.delta.rocks/), name set to the domain. Scopes admin:graphql ("GraphQL admin") and account ("Account").

  5. Role role:admin ("Admin"), which grants admin:graphql on the self audience.

  6. Organization permissions: scopes org:manage, org:members, org:roles, org:clients, org:audiences, org:access and org:api-keys on the self audience; roles role:org-admin (all of them) and role:org-member (none), both assignable inside organizations.

  7. Internal clients, both public with PKCE, grant type authorization_code, marked verified. Their redirect URLs are rewritten on every start, so changing domain or tls_terminated updates them.

    1. C______________________GQL_CLIENT "GraphiQL": redirect https://auth.infra.delta.rocks/graphiql, required scope admin:graphql.

    2. C_________________________ACCOUNT "Account": redirect https://auth.infra.delta.rocks/account/callback, post-logout redirect to the issuer, required scope account, optional scope admin:graphql.

  8. Import pictures stored in legacy columns into the images table (failures are only logged), then start an hourly image garbage collector.

  9. Bind the LDAP listener if ldap is configured, then listen for HTTP on listen (default 127.0.0.1:3000).

Nothing else is created: there are no users, organizations or third-party clients after the first start.

The first administrator

There is no setup wizard and no CLI command for the first account. Users are created through the admin GraphQL API, which needs an admin, so the first one has to be inserted with SQL. Run this after the backend has started once (it creates role:admin):

uid="U$(openssl rand -hex 16)"
psql "$DATABASE_URL" <<SQL
BEGIN;
INSERT INTO userdata_users (user_id) VALUES ('$uid');
INSERT INTO userdata_usernames (user_id, username, normalized_username)
  VALUES ('$uid', 'admin', 'admin');
INSERT INTO userdata_profiles (user_id, name) VALUES ('$uid', 'Administrator');
INSERT INTO oauth_user_granted_roles (user_id, user_role_id) VALUES ('$uid', 'role:admin');
COMMIT;
SQL

User ids are U followed by 32 alphanumerics (a database check enforces the prefix and length). Then:

  1. Open https://auth.infra.delta.rocks/account and sign in as admin. The new account has no credentials yet, so no password is asked for.

  2. Click "Add passkey" in the account header (/add-passkey) right away. If you want a password instead, enter admin mode, open /graphiql and issue yourself a registration code (below); the next sign-in asks for the code and a new password.

  3. Click "Enter admin mode". The dashboard signs you in again with prompt=login and requests admin:graphql. The admin GraphQL console is at /graphiql.

Further users, from /graphiql (needs admin:graphql). Create the user, then issue a registration code for them right away:

mutation {
  userdata {
    createUser(username: "alice", name: "Alice", email: "alice@example.com") { userId }
  }
}
mutation {
  userdata {
    user(userId: "U...") { issueRegistrationCode }
  }
}

Hand the code to the user: on first sign-in they enter it and choose a password, and can add passkeys afterwards. issueRegistrationCode refuses users that already have credentials. To make someone an admin:

mutation {
  userdata {
    user(userId: "U...") { grantRole(userRoleId: "role:admin") { userRoleId name } }
  }
}

More admin tasks are in Administration.