The Rules for Data Processing Pipeline Builders
You that data is eating the world. And whenever any reasonable amount of data needs processing, a complicated multi-stage data processing pipeline will be involved.
This all adds up to quite a complex system! And just as with any other engineering system, unless carefully maintained, pipelines tend to turn into a house of cards – failing daily, requiring manual data fixes and constant monitoring. (more…)