Design Challenges
Systems and processes will fail. Guaranteed
Systems will fail. The network will fail, the software that engineers write will crash, the disks on machines will fail, and people will make mistakes.
Impedance mismatch. Data ingestion exceeds capacity to consume
There are times when parts of the system will outpace other parts of the chain. If there is a weak link, the entire process can break.
Priorities change
The system must be able to adapt to changing priorities and quickly
System Evolution
The Spec was created to connect a highly distributed system of components and third-party managed dependencies that are loosely or not-at-all designed to work together