Pathpipe

Pathpipe started because I was tracking my own internship applications in a spreadsheet. It became a two-person open-source project with Pierre Guéroult: a tracker where you log applications, watch companies you care about, and let a worker tell you when those companies publish something new.

Architecture

A Turborepo monorepo with pnpm workspaces and three deployable apps:

• apps/api, a NestJS service. It is the only process that touches PostgreSQL. Business rules, validation and authorization live here once, not in every client.
• apps/web, a Next.js App Router frontend. Every API call is a Server Action, so tokens and internal URLs never reach the browser.
• apps/worker, a single binary started with an argument (email or ats) that runs one BullMQ consumer per process. Workers scale, deploy and fail independently of the API.

Shared packages carry the contracts: @repo/db (TypeORM entities and DTOs used by both API and web), @repo/queues (queue names, job payload types, default job and worker options), @repo/email (React Email templates), @repo/ui, plus ESLint, Jest, Prettier and TypeScript configs. A DTO change breaks the build on both ends instead of at runtime.

The job-discovery pipeline

Users watch companies. On a cron schedule (and on demand from the admin page), the ATS worker asks the API for all watched companies through an internal, API-key-protected endpoint, then scrapes each careers page:

1. Plain HTTP first. Fetch the page with a 15 s timeout. Look for application/ld+json blocks with @type: JobPosting; structured data gives title, URL, location, salary range and posting date for free.
2. Heuristic extraction if there is no JSON-LD: candidate links are matched against job-URL patterns and filtered by a blocklist of non-job titles (navigation, legal pages, etc.).
3. Headless browser fallback. If HTTP yields nothing, Playwright loads the page, waits for network idle, and runs the same JSON-LD then DOM extraction on the rendered page. Most modern ATS pages are client-rendered, so this step is what makes the coverage acceptable.

Results are posted in batch to the API, which inserts them one by one and treats a PostgreSQL unique-violation (23505) as “already known” rather than an error. When at least one row is new, the worker enqueues a job-alert job; the email worker renders the template and sends it. Queue defaults come from @repo/queues: 3 attempts with exponential backoff starting at 2 s, completed and failed jobs capped, worker concurrency of 5 with a 5 jobs/second limiter.

Applications and companies

Applications live on a kanban board with drag-and-drop whose order is persisted (kanbanOrder column), or in a paginated table with search, sort (including by salary), hideable columns and batch delete. Deletion is soft. Companies have CSV import and export, logos stored as blobs in the database with a Brandfetch lookup as fallback, and a watch/unwatch toggle that feeds the worker. Fuzzy company search uses the pg_trgm extension, enabled through a TypeORM migration.

Authentication and hardening

Passport strategies for local login, JWT, Google and LinkedIn. Email verification and password reset go through the email queue; accounts created with an OAuth provider are blocked from password reset. An email OTP flow is available. API keys are tied to a user and checked by a guard together with roles, which is how the worker authenticates to the internal endpoints. A global NestJS throttler allows 60 requests per minute and is covered by e2e tests in apps/api/test. Health checks use @nestjs/terminus.

Delivery

Pull requests run a reusable build-lint-test workflow plus type-checking and dependency review. Pushing to prod runs the same checks, then builds three Docker images (api, web, worker) in parallel, pushes them to GitHub Container Registry, and calls the Dokploy API to redeploy the Compose stack on my server. Schema changes ship as TypeORM migrations with a CLI; automatic synchronization is disabled in production. Locally, pnpm run dev brings up PostgreSQL and Redis in Docker, MailDev to catch outgoing mail, and a React Email preview server.

What I built

I authored the NestJS API (auth, applications, companies, job postings, API keys, internal endpoints, rate limiting, migrations), the Next.js app (kanban, table view, company management, settings, marketing site), the @repo/db, @repo/queues, @repo/email and @repo/ui packages, the migration of workers to BullMQ, the Dockerfiles and Compose files, and the GitHub Actions pipelines including the Dokploy deployment.

Why these choices

One API owning the database. With a frontend, an admin UI and two workers, letting each talk to PostgreSQL would have meant four copies of the authorization rules. The workers call the API with an API key instead.

Server Actions instead of a public client API. Credentials stay server-side and the browser bundle has no API client at all. The cost is that every interaction is a round trip through the Next.js server, which is fine for a form-heavy app.

BullMQ over an in-process scheduler. Scraping with a headless browser is slow and flaky. Putting it behind a queue gives retries, backoff and rate limiting for free, and keeps a stuck Chromium from taking the API down.

Progressive scraping. JSON-LD is cheap and structured, so try it first. Playwright is expensive, so use it last. The blocklist and URL patterns are deliberately simple heuristics; the unique constraint downstream means false positives cost a duplicate attempt, not a duplicate row.