Statlab BigCommerce Data Sync

Next.js 16 middleware on Vercel syncing BigCommerce orders, invoices & products to Azure Cosmos DB, HawkSearch, and Mailgun. Built for Statlab's B2B e-commerce operations.

BigCommerce / API

Azure Cosmos DB

HawkSearch

Email / Mailgun

Admin Dashboard

Cron / Background

1 Architecture Overview

Ctrl/Cmd + scroll to zoom · Drag to pan · Double-click to fit

BigCommerce

Source of Truth

V2 Orders, V3 Catalog, B2B Edition APIs. Sends webhooks on order/shipment/invoice events.

Cosmos DB

Persistence

Two containers: orders (orders, products, shipments, invoices) and operations (errors, activity, health, sync history).

HawkSearch

Product Search

Schema sync, product indexing, category hierarchy, content rules. Reindexed every 15 minutes via cron.

Mailgun

Email Delivery

Order confirmation/status emails rendered via Handlebars templates and sent through Mailgun API.

2 Data Flow

Entry Point

Webhook Routes

app/webhooks/bigcommerce/route.ts — orders & shipments
app/webhooks/b2b/route.ts — invoices

Handlers

Event Processing

src/webhooks/handlers/orders.handler.ts
src/webhooks/handlers/shipments.handler.ts
src/webhooks/handlers/invoices.handler.ts

Address Matcher

B2B Order Processing

Score-based matching: Street (40pts), City (30pts), State (20pts), ZIP (10pts). Min threshold configurable via ORDER_PROCESSOR_MIN_SCORE.

Every 1 Minute

Order Reprocessing

Fetches up to 10 unprocessed B2B orders, runs address matching, and updates BC status. Protected by CRON_SECRET bearer token.

app/api/cron/reprocess-orders/route.ts

Every 15 Minutes

HawkSearch Reindex

Fetches products modified since last run, transforms to HawkSearch format, and bulk indexes them. Tracks job status in Cosmos DB.

app/api/cron/hawksearch-reindex/route.ts

Vercel Cron Config

{
  "crons": [
    { "path": "/api/cron/reprocess-orders", "schedule": "* * * * *" },
    { "path": "/api/cron/hawksearch-reindex", "schedule": "*/15 * * * *" }
  ]
}

vercel.json

Phase 1

Fetch

Paginated fetch from BigCommerce V2/V3/B2B APIs

→

Phase 2

Transform

Map to Cosmos DB schema, strip PII, validate with Zod

→

Phase 3

Persist

Upsert documents to Cosmos DB containers

→

Phase 4

Track

Record sync job in history with counts and timing

Full Sync Orchestrator

CosmosSyncOrchestrator in src/jobs/cosmos-sync.ts — runs syncOrders() then syncInvoices() with progress tracking. Triggered via POST /api/cosmos/sync/orders/full or POST /api/cosmos/sync/invoices/full.

3 Service Layer

BigCommerce Services

API Clients

sync-client.ts — V2/V3 orders, invoices, products, customers
products-client.ts — V3 Catalog API for HawkSearch
customers-client.ts — V3 Customers API for B2B pricing
webhooks-client.ts — Webhook registration/toggle
retry-fetch.ts — Exponential backoff (3 retries, 1s–10s)

src/services/bigcommerce/

Cosmos DB Services

Persistence Layer

OrdersService — CRUD for orders, products, shipments
InvoicesService — CRUD for B2B invoices
ActivityService — Processing audit trail (7-day TTL)
ErrorsService — Error logging & retrieval
SyncHistoryService — Job tracking
HealthService — Health snapshots

src/services/cosmosdb/

HawkSearch Services

Search Index Management

HawkSearchClient — Core API client with retry
SchemaSyncService — Field catalog sync
ProductIndexingService — Bulk product indexing
HierarchySyncService — Category tree sync
FieldConfigService — Versioned field config
ContentRulesService — Search content rules

src/services/hawksearch/

Email & Processing

Outbound Services

EmailSenderService — Handlebars rendering + Mailgun
OrderStatusMailer — Order confirmation emails
OrderProcessorService — Address matching orchestration
AddressMatcher — Score-based algorithm (0-100)

src/services/email/ & src/services/order-processor/

4 Module Dependencies

Layered architecture — arrows flow from entry points down through services to infrastructure.

Route Layer

Next.js App Router handlers. Entry points for webhooks, cron, admin API, and UI pages.

app/

Handler Layer

Domain-specific webhook event processors. Orchestrate fetching, transforming, and saving.

src/webhooks/

Library Layer

Config, auth, and factory functions. Singletons for Cosmos, factories for BC clients.

src/lib/

Service Layer

Business logic, API clients, data transforms, validation. No cross-dependencies.

src/services/

5 NPM Dependencies

Package	Version	Purpose
next	16.1.6	App framework — API routes, server components, middleware
@azure/cosmos	4.9.1	Cosmos DB client — document CRUD, queries, container management
mailgun.js	12.7.0	Mailgun API — order status email delivery
iron-session	8.0.4	Encrypted session cookies for admin authentication
bcryptjs	3.0.3	Password hashing — admin login verification
zod	3.25.76	Schema validation — config, API responses, transforms
pino	10.3.1	Structured JSON logging — webhook, email, cron events
handlebars	4.7.8	Email template rendering — order confirmation HTML
ulid	3.0.2	Unique ID generation — Cosmos DB document IDs
form-data	4.0.5	Multipart form data for Mailgun API requests

6 Extension Points

Add a New Data Type

Create transform in src/services/cosmosdb/[domain]/
Create service class extending the Cosmos CRUD pattern
Add TypeScript types to src/types/
Add Zod schema in cosmosdb/shared/schemas.ts
Export from src/services/cosmosdb/index.ts
Wire up in the factory at cosmosdb/client/factory.ts

Add a New Webhook Handler

Create handler class at src/webhooks/handlers/[domain].handler.ts
Implement handle[Event](data) methods
Register factory in src/lib/webhook-utils.ts
Add scope routing in app/webhooks/bigcommerce/route.ts
Register webhook scope in bigcommerce/webhook-scopes.ts

Add a New Cron Job

Create route at app/api/cron/[job-name]/route.ts
Verify CRON_SECRET bearer token in the handler
Add schedule entry to vercel.json crons array
Optionally track job runs via SyncHistoryService

Add a New Admin API Endpoint

Create route at app/api/admin/[resource]/route.ts
Middleware auto-protects /api/admin/* routes
Use getCosmosClient() for data access
Follow existing pattern: try/catch with JSON responses

Add a New Admin Dashboard Page

Create page at app/admin/[resource]/page.tsx
Layout inherits from app/admin/layout.tsx (sidebar nav)
Create React components in src/components/admin/
Fetch data via the admin API routes with fetch()

Add a New External Integration

Create service in src/services/[integration]/
Add client config with Zod validation
Add retry logic using the retry-fetch.ts pattern
Create factory function in src/lib/
Add required env vars and document them

7 Config & Auth

BigCommerce

BIGCOMMERCE_STORE_HASH
BIGCOMMERCE_ACCESS_TOKEN
BC_WEBHOOK_SECRET
BC_API_MAX_RETRIES (default: 3)

Cosmos DB

COSMOS_CONNECTION_STRING
COSMOS_DATABASE (statlab-orders)
COSMOS_CONTAINER (orders)
COSMOS_OPS_CONTAINER (operations)

HawkSearch

HAWKSEARCH_API_KEY
HAWKSEARCH_CLIENT_GUID
HAWKSEARCH_DATABASE_ID
HAWKSEARCH_ENGINE_NAME

Email / Mailgun

MAILGUN_API_KEY
MAILGUN_DOMAIN
EMAIL_FROM
WEBSITE_BASE_URL

Admin Auth

SESSION_SECRET — iron-session key
ADMIN_USERNAME
ADMIN_PASSWORD_HASH — bcrypt
Cookie: 30-day TTL, HTTP-only

Order Processor

ORDER_PROCESSOR_MIN_SCORE (75)
ORDER_PROCESSOR_TARGET_STATUS (11)
ORDER_PROCESSOR_ADDRESS_MATCHING
CRON_SECRET — cron auth

Authentication Flow

Step 1

/login

→

Step 2

Verify

bcrypt hash compare

→

Step 3

Session

iron-session cookie

→

Step 4

Middleware

Protect /admin/*

Generated from statlab-bigcommerce-data-sync · 2026-03-26