A real time messaging system - Evolving design with Requirements

Continuing in this series of how requirements might evolve for systems, this time I explored how a real time messaging system might evolve over time. I grouped requirements by area, to enable more comprehensive exploration. How does your system design evolve as you address the following requirements?

1) Design a system to enable one producer to send messages to one consumer

2) Extend the system to make sure that messages are never lost

Scaling:

3) Extend the system to support multiple independent consumers

4) Extend the system to support multiple co-operating consumers

5) Extend the system to support multiple independent producers 

6) Extend the system to support multiple co-operating producers 

7) How would you change the system if messages are big?

Hardware:

8) What hardware kinds would you use for this system?

9) How much hardware would be required to build this system ( say 1K producers, 1K consumers, 10GBPS data write at peak, and 24 hours of durability required)

10) How would you reduce hardware cost? Occasional message losses are fine.

11) What can be built in one half of that cost? Can we move some processing to clients to enable cheaper system cost?

12) Could Individual message send to system be too costly. Enable producers and consumer to send and consume in batch

13) How would system change if durability was not needed? Can that help reduce cost?

SLA's:

14) What kind of write SLAs could this system offer to producers? (for individual and batch operations both)

15) What kind of read SLAs could this system offer to consumers?(for individual and batch operations both)

16) What if we needed far faster SLAs in write and read operations?

Multi tenancy:

17) Extend the system to a multi tenant environment. Ensure that each customer stays within their assigned quota.

18) Extend the system so that customers can use other customers quota if its unused.

19) Publish metrics so that consumers can see how much of the system are they using

20) Extend the system so that each customer can only read/write their own topics. We dont support cross customer operations yet.

Ordering:

21) Extend the system so that messages from a given producer are always given to customers in order

22) What if some reordering is okay?( for e.g. messages more than one minute apart are properly ordered)

23) Enable Producers to send priority messages to the consumer on the same queue/topic

Extend system functionality:

24) Extend the system to auto expire messages based on the time to live sent by the producer

25) Extend the system to enable producers to delete message based on the key sent by the producer

26) Extend the system to enable producers to update message based on the key sent by the producer

27) Consumers want to only get those messages which pass their filter. Extend the system to let consumers add filters on any topic

Operational:

28) The system needs to support rolling restarts for weekly patches of OS. Are there any limitations or constraints on recycling fleet fast?

29) Is there any concern with doubling capacity overnight? We will have a new client start hitting us tomorrow.

30) Is this system cloud ready?

31) What metrics would you publish to help customers troubleshoot common issues?

Thanks

Umesh