It depends on the project. The set of available options has evolved over time with many new players showing up for a while, getting hot, and then cooling off in favor of other things.

Currently, and for the foreseeable future, Streams is still my primary data-in-motion processing platform.

However, there are situations where I consider leveraging the capabilities of structured streaming available in Apache Spark 2.x, Apache Kafka, and/or Redis, sometimes together with Streams.

We tried some PoCs with Apache Spark for some similar objectives for CDR files. We learned toward Streams due to its streaming process, as opposed to Spark's batch processing style. We appreciated the well- designed architecture, service host distributions, high available options, and TELCO TEDA solutions.

We evaluated Tibco.