Architecture

Single process, single database, zero external dependencies.

Overview

Echelon Analytics is a Deno + Fresh application backed by a single SQLite database. All state — sessions, rate limiter, burst maps, buffered writers — lives in memory. Runs as a single process (no --parallel flag).

Tech Stack

Request Flow

How a pageview goes from browser to database:

1. Script Load

Browser fetches /ea.js, which returns a dynamically-generated tracker script with an embedded WASM blob (rotated every 6 hours) and a per-minute challenge string.

2. Client Initialization

• Generates or loads session ID from sessionStorage
• Reads UTM parameters from the URL
• Starts WASM proof-of-work solver asynchronously
• Attaches interaction listeners (pointer, scroll, click, keydown)
• Waits 800ms minimum before firing beacon

3. Pageview Beacon

On user interaction, the tracker loads /b.gif as an image with query params: path, site ID, session ID, PoW token, screen dimensions, referrer, UTM data.

4. Server Processing

• Rate limit check — drop if exceeded
• Known bot UA filter — drop matching requests
• Visitor identity — HMAC hash (cookieless) or _ev cookie (opt-in)
• Bot scoring — compute 0–100 heuristic score
• PoW verification — valid (+0), missing (+15), or invalid (+25)
• Data enrichment — geo, device type, OS, referrer classification
• Push to in-memory BufferedWriter

5. Buffered Write

The BufferedWriter batches records (up to 50k) and flushes to SQLite every 10–15 seconds in a single transaction. On shutdown, remaining records are flushed before exit.

6. Behavioral Events

Scroll depth, clicks, bounces, session events, and custom events are sent via sendBeacon to POST /e. Same scoring and buffering pipeline, separate writer instance.

Database

SQLite Configuration

Core Tables

Supporting Tables

Adapter Interface

The database layer uses an interface (DbAdapter) with methods for query, queryOne, run, exec, and transaction. Currently backed by node:sqlite. All queries use parameterized ? placeholders.

Data Pipeline

Buffered Writers

Two BufferedWriter<T> instances run in parallel:

Each flush runs in a single transaction. If a flush fails, records are re-added to the buffer.

Daily Rollup

At ~03:00 UTC, the maintenance task:

• Aggregates yesterday's visitor_views into visitor_views_daily
• Filters out bot_score ≥ 50 and excluded visitors
• Groups by site, date, device type, country, and returning status
• Uses INSERT OR REPLACE (idempotent on composite primary key)

🔬 "Did you know raw visitor data is preserved separately from rollups? You can re-analyze with different bot score thresholds at any time by querying the raw tables directly." -🦭

Data Purge

After rollup, expired data is purged:

• visitor_views and semantic_events: older than ECHELON_RETENTION_DAYS (default 90)
• visitor_views_daily: older than 730 days
• perf_metrics: older than retention period
• Followed by PRAGMA incremental_vacuum(100)

Visitor Identity

Cookieless (default)

HMAC-SHA256 of IP + User-Agent + site ID + date, truncated to 16-char hex. Resets daily — no cross-day tracking. Requires no cookie consent.

Cookie (opt-in)

HttpOnly _ev cookie with a 16-char hex value, 30-day TTL. Enabled with data-cookie on the script tag.

Proof-of-Work System

• WASM module generated from a 64-byte random seed every 6 hours
• Implements a SipHash-inspired algorithm with randomized constants
• Challenge string rotates every minute (HMAC-SHA256 of minute bucket)
• Client solves and caches token in sessionStorage (90% reuse, 10% re-solve)
• 150ms solve timeout — if WASM is too slow, beacon fires without token (penalty applied)
• Server verifies against current + previous WASM slots and last N minute buckets

Authentication

Bearer Token

Set ECHELON_SECRET env var. Verified via constant-time comparison to prevent timing attacks.

Username/Password

PBKDF2-SHA256 with 600k iterations, 256-bit key, 128-bit random salt. Uses Web Crypto API only — no external dependencies.

Sessions

In-memory session store. Random UUID tokens, 24-hour TTL, pruned every 30 minutes. echelon_session HttpOnly cookie.

CSRF Protection

Mutating requests (POST/PATCH/DELETE) with cookie auth must include a matching Origin or Referer header.

Admin UI Architecture

Styling

The admin UI uses CSS custom properties (--ea-* prefix) for all colors. The design uses a government-inspired palette with navy and burgundy accents on a clean white surface — built for visual integrity. Tailwind CSS v4 handles utility classes.

Live Dashboard

The DashboardLive Preact island polls /api/stats/dashboard every 10 seconds and renders:

• Now gauge — active visitors in the last 5 minutes
• 60-minute chart — per-minute visitor and event counts (SVG line chart)
• 24-hour chart — per-hour visitor and event counts (SVG line chart)
• Recent visitors — last 20 non-bot visitors with time-ago display
• Recent events — last 20 events with type and data

Navigation

The AdminNav component provides a persistent navigation bar with:

• Site selector dropdown (populated from known sites in the database)
• Date range selector (7/14/30/60/90/180/365 days)
• Live stats bar showing human/bot views, unique visitors, buffer sizes, response time, and uptime
• Both selectors persist via cookies and update the URL on change

Directory Structure

echelon-analytics/
  main.ts                  Entry point, graceful shutdown
  deno.json                Config, import aliases, tasks
  vite.config.ts           Vite build config

  lib/
    tracker.ts             Tracker script generation
    beacon.ts              Pageview beacon handler
    events-endpoint.ts     Behavioral events handler
    bot-score.ts           Bot scoring heuristics
    challenge.ts           PoW challenge management
    challenge-wasm.ts      WASM bytecode builder
    buffered-writer.ts     Generic batch writer
    auth.ts                PBKDF2 password hashing
    session.ts             In-memory session store
    config.ts              Env var centralization
    maintenance.ts         Daily rollup and purge
    stats.ts               Stats query handlers

    utm.ts                 Campaign cache
    rate-limit.ts          Per-IP rate limiter
    format.ts              Date/time formatting
    admin-stats.ts         Live stats queries
    db/
      adapter.ts           Database interface
      sqlite-adapter.ts    SQLite implementation
      database.ts          Init, migrations, singleton
      schema.ts            DDL and indexes

  routes/
    ea.js.ts               GET /ea.js
    b.gif.ts               GET /b.gif
    e.ts                   POST /e
    _middleware.ts          CORS, security headers
    api/                   REST API routes
    admin/                 Admin UI routes

  islands/
    DashboardLive.tsx      Live dashboard (charts, tables)
    RealtimePanel.tsx      Realtime visitor monitor
    TrendChart.tsx         SVG trend charts
    BotActions.tsx         Exclude/include visitor

    CampaignForm.tsx       Campaign CRUD
    ExperimentForm.tsx     Experiment CRUD
    ConsentCssEditor.tsx   Per-site CSS editor

  components/
    AdminNav.tsx           Nav shell with live stats
    Pagination.tsx         Server-rendered pagination

  assets/                  Tailwind CSS, images
  static/                  Favicon, static files

Design Decisions

Single Process

All state is in-memory. Simple to deploy and reason about. Trade-off: horizontal scaling requires moving to external session/state stores.

WAL Mode

Write-Ahead Logging enables concurrent reads while writes proceed. Combined with synchronous = NORMAL, balances durability and throughput.

Buffered Writes

Trades immediate durability for throughput. Beacon returns instantly (~1ms), records batch-flush every 10–15 seconds. Graceful shutdown flushes remaining buffers.

Runtime WASM Generation

Each deployment generates unique WASM blobs. Bot toolkits can't pre-compute solutions. Low barrier for browsers (150ms timeout with fallback).

Raw Data + Rollups

Raw data kept for inspection and re-analysis (90 days). Rollups provide fast long-range queries (2 years). Bot filtering happens at rollup time — raw data preserves everything for forensics.