Design a Notification System (HLD)

Requirements: Design a notification system (e.g., for an app like Zomato) that can send push/SMS/email notifications to users for events like order updates, offers, and reminders. Produce a high-level design (HLD).

Design:

• Ingestion: upstream services (order service, marketing service, etc.) publish notification-worthy events to a message queue/event stream (e.g., Kafka) rather than calling the notification service synchronously, decoupling producers from delivery.
• Notification service: consumes events, applies templating (per notification type/locale), user preference and rate-limit checks (don't spam a user), and routes to the right channel (push/SMS/email) based on event type and user settings.
• Channel delivery: per-channel workers/adapters call external providers (APNs/FCM for push, an SMS gateway, an email provider), with retry queues and dead-letter queues for failed deliveries.
• Storage: a notifications table/store for delivery status and history (for support/debugging and in-app notification center), keyed by user id with TTL/archival for old notifications.
• Scale considerations: partition the queue by user id for ordering per user, rate-limit per user and per channel-provider to respect provider throughput limits, and use idempotency keys so retries don't duplicate a delivered notification.
• Reliability: track delivery state per notification, with exponential backoff retries, and prioritize latency-sensitive notifications (e.g., OTPs) over bulk/marketing sends via separate queues or priority levels.