Real-time User Activity Analytics Backend

This project challenges you to build a sophisticated NestJS backend system that can ingest, process, and broadcast real-time user activity data. You will design a system capable of handling a stream of events, aggregating them into meaningful analytics, and serving these insights instantly to a hypothetical dashboard via WebSockets. This requires a strong understanding of event-driven architecture, real-time communication, and robust data management.

What to Build

YouYou will develop a multi-service NestJS application that simulates a simplified analytics pipeline. The core components include:

1. Event Ingestion Service: An API endpoint to receive raw user activity events.
2. Analytics Processor Service: A background worker that consumes events, processes them, and aggregates data into various metrics.
3. Real-time Dashboard Gateway: A WebSocket server that provides authenticated clients with live updates of aggregated analytics.

Specific Requirements & Acceptance Criteria

• Event Ingestion Endpoint: Create a POST /events API endpoint that accepts raw user activity events (e.g., eventType: 'pageView', userId: 'uuid', timestamp: 'ISOString', payload: { url: '/home' }).

  • Implement robust request validation using NestJS Pipes (e.g., class-validator).
  • Upon successful validation, the event should be pushed to an internal queue (you can simulate this with an in-memory event emitter for simplicity or use a library like NestJS-Queue with BullMQ for a more robust solution, if you wish to tackle an extra challenge).

• Analytics Processor: Develop a service that consumes events from the queue.

  • Process events to derive key metrics. Examples include:
    • Total unique users within the last 5 minutes, 30 minutes, 24 hours.
    • Count of specific event types (e.g., pageView, login, itemAddedToCart) per minute/hour.
    • Simple funnel analysis (e.g., users who viewed a product, then added to cart, then purchased).
  • Store aggregated metrics in a PostgreSQL database using TypeORM or Prisma.
  • Ensure data aggregation is efficient and can handle a moderate stream of events.

• Real-time Dashboard Gateway: Implement a NestJS WebSocket Gateway (@nestjs/platform-socket.io or similar).

  • Clients connecting to /dashboard should receive real-time updates as new analytics data becomes available.
  • Implement JWT-based authentication for WebSocket connections. Only authenticated users with a valid token should be able to connect and receive updates.
  • Periodically (e.g., every 5-10 seconds, or upon new data aggregation) broadcast the latest aggregated metrics to all connected and authenticated WebSocket clients.

• Database Management: Utilize PostgreSQL for storing both raw (if desired for auditing) and aggregated analytics data.

  • Define clear database schemas for events and aggregated metrics.
  • Implement database migrations (if using TypeORM) or schema definitions (if using Prisma).

• Error Handling: Implement comprehensive error handling across all services, including input validation, database operations, and queue interactions.

• Testing: Include basic unit tests for core event processing and aggregation logic, and e2e tests for the event ingestion API endpoint and WebSocket connectivity/authentication.

• Scalability & Performance Considerations: Briefly discuss in your README.md any considerations you made for scaling the event ingestion or processing, even if not fully implemented.

What to Include in Your Submission

• Source Code: The complete NestJS project source code.
• package.json: With all dependencies listed.
• Database Setup: Either SQL migration files or Prisma schema definition, along with instructions on how to set up the database.
• README.md: A detailed README.md file that includes:
  • Project setup instructions.
  • API endpoint documentation (e.g., POST /events structure).
  • WebSocket connection details and expected message formats.
  • Instructions on how to run tests.
  • A brief explanation of your architectural design choices and how you handled key challenges (e.g., real-time updates, event processing, authentication).
• Demonstration: Optionally, include curl commands or a Postman collection to demonstrate submitting events and connecting to the WebSocket endpoint (with an example JWT).