# Development Guide

Contributor setup, build metadata, verification, and implementation notes for CrowdSec Web UI.

- Operator setup and configuration: [README.md](README.md)
- Synthetic performance testing: [LOAD_TESTING.md](LOAD_TESTING.md)
- Application API: [API.md](API.md)

## Prerequisites

| Tool | Version |
| --- | --- |
| Node.js | `24.22.1` (LTS) |
| pnpm | `12.15.0` |
| CrowdSec | A reachable LAPI for normal development |

### Install dependencies

Install the project dependencies and GeoNames snapshot.

```bash
pnpm install
pnpm run geocoder:data
```

### GeoNames data

- `geocoder:data` downloads the `cities5000` and admin-1 extracts used for location labels.
- Delete `geonames/` and rerun the command to refresh the local snapshot.
- Packaged images contain a build-time snapshot and never download it at runtime.
- GeoNames data is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

## Local Configuration

1. Create the local data and secret directories:

   ```bash
   mkdir -p data secrets
   cp config.example.yaml data/config.yaml
   ```

2. In `data/config.yaml`:

   - Set `storage.dataDir` to `./data`.
   - Set the LAPI URL to `http://localhost:8080`.
   - Set `password.file` to `./secrets/crowdsec_password.txt`.

3. Put the registered CrowdSec watcher password in `secrets/crowdsec_password.txt`.

The default local path is `./data/config.yaml`; use `CONFIG_FILE` only to select another existing file. For mTLS, replace password authentication with `type: mtls`, `certFile`, and `keyFile`.

See the [configuration reference](README.md#configuration) for all fields, environment overrides, and secret formats.

## Run and Build

### Development server

Development mode starts the API on port `3000` and Vite on port `5173`.

```bash
pnpm run dev
# or
./run.sh dev
```

### Production build

Build and start a local production bundle.

```bash
pnpm run build
pnpm start
# or build and start through the helper
./run.sh
```

### Project layout

| Path | Purpose |
| --- | --- |
| `client/` | React, Vite, and Tailwind CSS frontend |
| `server/` | Hono API and background services |
| `scripts/` | Data, test, release, and load-test helpers |
| `dist/client/` | Built static assets |
| `dist/server/` | Compiled server output |

## Verification

### Complete verification

```bash
pnpm run verify
```

### Individual checks

```bash
pnpm run test
pnpm run coverage
pnpm run typecheck
pnpm run lint
pnpm run build
```

Type checking uses TypeScript 7 through the `@typescript/native` package alias. The
`typescript` alias provides Microsoft's TypeScript 6 API compatibility package for
ESLint and other tools that still require the compiler API. This follows the
[upstream compatibility guidance](https://devblogs.microsoft.com/typescript/announcing-typescript-7-0/#running-side-by-side-with-typescript-6.0).

The synthetic large-dataset workflow is documented separately in [LOAD_TESTING.md](LOAD_TESTING.md).

## Build and Image Metadata

These values apply when building a local production bundle or container image.

| Variable | Default | Purpose |
| --- | --- | --- |
| `DOCKER_IMAGE_REF` | `theduffman85/crowdsec-web-ui` | Image checked for updates; accepts `owner/repo` or a registry-prefixed reference such as `ghcr.io/owner/repo`. |
| `VITE_VERSION` | `0.0.0` | UI version and update-comparison value. |
| `VITE_BRANCH` | `main` | UI branch label; `dev` enables development-build update comparisons. |
| `VITE_COMMIT_HASH` | Empty | Commit shown in the sidebar and used by update logic. |
| `VITE_BUILD_DATE` | Generated at build time | Build timestamp shown in the UI. |
| `VITE_REPO_URL` | `https://github.com/TheDuffman85/crowdsec-web-ui` | Repository used for release and commit links. |

## Development and Test Environment

| Variable | Default | Purpose |
| --- | --- | --- |
| `BACKEND_URL` | `http://localhost:3000` | Vite proxy target for `/api`. |
| `CROWDSEC_MTLS_IMAGE` | `crowdsecurity/crowdsec:v1.8.1` | Image used by the CrowdSec mTLS smoke test. |
| `CROWDSEC_MTLS_KEEP` | `0` | Set to `1` to retain the disposable test container. |
| `CROWDSEC_MTLS_CONTAINER` | Generated | Override the disposable container name. |

`scripts/ensure-native-deps.mjs` also honors standard Node/npm cache variables: `COREPACK_HOME`, `XDG_CACHE_HOME`, `PREBUILD_INSTALL_CACHE`, `npm_config_cache`, `npm_config_devdir`, and `npm_config_nodedir`.

## CrowdSec mTLS Smoke Test

The smoke test starts a disposable CrowdSec LAPI container, mounts an isolated temporary CrowdSec data directory, generates temporary server and client certificates, enables client-certificate verification, logs in through the Web UI LAPI client, and confirms that CrowdSec registered the TLS machine.

```bash
pnpm run test:mtls:crowdsec
```

### Overrides

```bash
CROWDSEC_MTLS_IMAGE=crowdsecurity/crowdsec:latest pnpm run test:mtls:crowdsec
CROWDSEC_MTLS_KEEP=1 pnpm run test:mtls:crowdsec
CROWDSEC_MTLS_CONTAINER=my-crowdsec-test pnpm run test:mtls:crowdsec
```

`CROWDSEC_MTLS_KEEP=1` keeps both the container and its temporary mounted files and prints their paths for inspection. Remove them manually when finished.

## Quick Filter Architecture

Quick Filters are structured-search editors shared by Dashboard, Alerts, and Decisions.

| Path | Responsibility |
| --- | --- |
| `client/src/components/QuickFilters.tsx` | Accessible side drawer, active counts, visible/hidden/unavailable sections, facet search, pagination, and clear controls |
| `client/src/lib/quickFilters.ts` | Browser-local persistence, normalization, page-field compatibility, and synchronization between stored selections and search ASTs |
| `shared/search.ts` | Parsing, compiling, editing, and serializing structured searches |
| `server/app.ts` | Alert/decision facet queries and Dashboard `q` evaluation |

Keep these invariants when changing filters:

- The canonical browser key is `crowdsec-web-ui:quick-filters`. It stores field inclusions/exclusions, the date range, and simulation mode. Do not add another page-specific persistence mechanism.
- Alerts, Decisions, and Dashboard read and write the same stored object. Synchronization updates fields supported by the current page without deleting selections that are only usable elsewhere.
- A field that exists on another page but cannot be applied on the current page belongs in the drawer's **Unavailable** section. Keep its active selection visible and clearable without requesting an unsupported facet.
- Included facet values are exact comparisons serialized with `=`; excluded values use `<>`. The typed `field:value` form must remain a broader contains search. This distinction prevents a selected target such as `tausend.me` from matching subdomains.
- Quick Filters are enabled only when the current advanced-search AST can be represented without changing its meaning. Safe expressions use exact `=`/`<>` facets joined by `AND`, same-field exact values joined by `OR`, inclusive `date>=`/`date<=` bounds, and simple exact negation. Broad `:`, free text, strict date bounds, unsupported fields, cross-field `OR`, grouped negation, and conflicting same-field clauses must preserve the advanced query and disable every Quick Filter editing entry point with a visible recovery explanation.
- Multiple included values for one field are joined with `OR`; different fields and exclusions are combined with `AND`.
- Dashboard widgets must update the shared state rather than maintaining parallel country, scenario, AS, target, or date filters. Dashboard-supported `q` fields must also be represented in its cached alert statistics record.
- Dashboard requests carry alert-compatible filters in `q` and decision-compatible filters in `decision_q`. Headline filtered totals use the same indexed SQL predicates as the paginated lists; cached records continue to drive charts, maps, and top lists.
- URL search predicates take precedence over persisted selections for the same field. Stored selections fill fields absent from the initial URL.
- Facet counts are disjunctive: the requested facet's own predicate is removed while all other active filters remain.
- Alerts paging caches exact totals and a bounded 20-page internal-ID window for every filter/search combination, then loads full records only for the requested page. Cache validity is bounded by the lookback expiry of the oldest matching row and every data publication invalidates it.
- Default decision requests use the precomputed `is_duplicate` indexes. Boolean combinations of duplicate-group identity fields (`instance`, `ip`, and `sim`, plus `status` in the active-only view) must stay on that indexed path because they cannot split a group.
- If active filters can exclude the global primary, rank only internal IDs and ranking columns, then load full records after the page is bounded. The first request caches a bounded 20-page ID window; consecutive pages must not repeat the window function. Keep filtered totals and promotion semantics in the same ranking pass.
- Dashboard row indexes carry a durable SQLite change epoch and revision. Normal refreshes patch cached Alert and Decision records from `dashboard_record_changes`; bulk rebuild epochs force one safe full rebuild. Keep update triggers limited to Dashboard-relevant columns, prune journal entries only through the oldest retained cache revision, and never discard the previous complete index during normal invalidation. Empty-cache and oversized-journal invalidations reset the epoch so installations that do not use Dashboard cannot accumulate an unbounded journal.
- Post-refresh Dashboard warming is background work. Full-build batches must yield while a foreground Alerts or Decisions list query is active so analytics cache preparation cannot multiply paging latency.

When changing this flow, cover persistence/query synchronization in `client/src/lib/__tests__/quickFilters.test.ts`, page integration in the Alerts, Decisions, and Dashboard tests, structured-search semantics in `server/__tests__/search/search.test.ts`, duplicate-group planning in `server/__tests__/app/search-sql.test.ts`, bounded page-window reuse in `server/__tests__/app/api-responses.test.ts`, and Dashboard/API consistency in `server/__tests__/app/api-dashboard.test.ts`.

## Cache and Synchronization Internals

### Storage layout

- `cache.db` is the main SQLite connection and holds everything rebuilt from LAPI: alerts, decisions, allowlists, search indexes, the Dashboard change journal, CVE lookups, and the `meta` bookkeeping table.
- `state.db` is attached as the `state` schema and holds data that exists nowhere else: users, passkeys, notification channels, rules, history and incidents, saved and recent filters, the alert deletion queue, and the `settings` table. It always uses a rollback journal, so each commit is synced (roughly 10–20 ms on local disks); batch writes there. `storage.walEnabled` only applies to the cache.
- Queued deletions are unfinished user requests that LAPI cannot reconstruct. The sync worker cannot read `state.db`, so queued alert IDs are mirrored into the cache table `alert_deletion_tombstones`, which is rebuilt from the queue on every start.
- `getMeta`/`setMeta` route by key: keys listed in `STATE_SETTING_KEYS` or starting with `auth_` or `branding_` go to `state.settings`, everything else to `meta`. Add every new durable setting there, or it is lost with the cache.
- Only the main process attaches `state.db`. The sync worker opens the cache alone (`statePath: null`), so its long write transactions never lock the state file, and query workers read only `cache.db`.
- On the first start after upgrading, durable tables and settings are copied from a legacy `crowdsec.db` into `state.db`, after which `crowdsec.db` is deleted and the cache is rebuilt by a full synchronization.

### Synchronization

CrowdSec Web UI keeps normalized alert and decision history in SQLite.

- Bootstrap imports the configured lookback period in chunks.
- Regular refreshes import the newest delta and reconcile a bounded number of historical windows.
- Recent windows are checked more often; windows with cached active decisions are prioritized.
- Reconciliation progress persists across restarts and fair scheduling prevents old windows from starving.
- The moving current window shares the normal delta request when due, avoiding a duplicate LAPI call.
- Normal and idle refresh intervals trigger each target cadence.
- Timed-out history requests retry with smaller windows down to `crowdsec.sync.alertSyncMinChunk`.
- Failed bootstrap work retries in the background using `crowdsec.sync.bootstrapRetryDelay`; partially imported caches remain available and are marked partial.

### Refresh recovery

- The scheduler uses a generation per start or idle wake-up. An older in-flight loop must never schedule another timer after its generation is replaced or stopped.
- SQLite reads and sync writes run in persistent subprocesses. Native SQLite statements cannot be interrupted reliably by terminating a JavaScript worker thread. On timeout, kill the owning subprocess and wait for its exit before admitting another writer; retiring read processes still count toward pool capacity until exit.
- Keep transaction ownership and atomic publication intact. Never recover by clearing the refresh promise while its transaction is still running. Reconciliation metadata and data commit together, and failed refreshes retain their previous cursor.
- Publication waits expire after the larger of `storage.queryTimeout` and `storage.analyticsQueryTimeout`, plus 30 seconds. Expired writers are removed from the lock queue before rollback, so they cannot later acquire and leak the publication lock.
- GeoNames initialization and lookups fall back to coordinates after 30 seconds. A late successful initialization remains usable by future requests. MQTT delivery has a deadline covering connect, publish acknowledgement, and forced teardown.
- Recovery tests must interrupt executing SQL as well as drop acknowledgements. Cover subsequent successful refreshes, queued writes, stale callbacks, startup/shutdown, and reader/writer ordering. Do not replace these checks with tests that only assert a timeout error.

### Write and deletion safety

- Only changed alerts and added or deleted decisions are written.
- Unchanged alerts are compared without constructing decision mutations, keeping large blocklist checks inexpensive.
- Missing records are deleted only after every required LAPI query for the window succeeds.
- Relative LAPI ranges are padded and filtered back to exact local boundaries so transport delay, timestamp rounding, or partial scope responses cannot cause destructive reconciliation.
- `decisions.alert_id` is authoritative; alert rows do not duplicate embedded decision objects or ID arrays.
- Unknown CrowdSec extensions and open-ended event metadata remain compact JSON; legacy full-payload columns are cleared during migration.
- Active duplicate winners are refreshed in batches and stored as indexed flags.
- Alerts are indexed and ordered by CrowdSec `start_at` when present, otherwise `created_at`, so replayed alerts retain their original event time.
- Retention follows `crowdsec.sync.lookback` and cleanup is automatic.

To force a full cache reset, call `POST /api/cache/clear`. See [API.md](API.md) for authentication and request details.

## Translations

Translations live in `client/src/locales/`. Keep every locale aligned with `client/src/locales/en.json`. Server-side localization reuses these files, so update notification and synchronization keys together with related UI copy.