The issue we encountered was the inability to efficiently extract and evaluate our data from existing databases, causing limitations with Tableau, which struggled to handle datasets exceeding a hundred million records. Consequently, we explored alternative systems and initially attempted to access all data from target systems. In pursuit of faster and more effective data management, we considered Exasol as an alternative OLAP system. Ultimately, we opted for SingleStore.
When I was working at Wag, MySQL had trouble scaling. We were using an M4-10X RDS instance and three replicas. These machines are expensive and with the growth that the company was experiencing, it was clear we would not be able to scale properly. There were heavy writes done against MySQL that should not have been there; an event table for example. This table was close to 900M rows, taking more than 300GB. I decided to migrate this table to MemSQL, freeing up both lots of space and write resources on MySQL. After the event table was migrated to MemSQL, using the MemSQL built-in S3 pipeline, the table was compressed down to 30GB.
The SingleStore platform combines transactional, analytical, and vector data workloads in a real-time, unified, distributed SQL database that is MySQL and MongoDB wire protocol-compatible. SingleStore also combines the capabilities of traditional databases with the scalability of the cloud, making it suitable for a range of data-driven applications.
SingleStore provides a unified platform that supports both transactional and analytical workloads, enabling real-time data processing and...
The issue we encountered was the inability to efficiently extract and evaluate our data from existing databases, causing limitations with Tableau, which struggled to handle datasets exceeding a hundred million records. Consequently, we explored alternative systems and initially attempted to access all data from target systems. In pursuit of faster and more effective data management, we considered Exasol as an alternative OLAP system. Ultimately, we opted for SingleStore.
The solution is primarily used for storing data.
When I was working at Wag, MySQL had trouble scaling. We were using an M4-10X RDS instance and three replicas. These machines are expensive and with the growth that the company was experiencing, it was clear we would not be able to scale properly. There were heavy writes done against MySQL that should not have been there; an event table for example. This table was close to 900M rows, taking more than 300GB. I decided to migrate this table to MemSQL, freeing up both lots of space and write resources on MySQL. After the event table was migrated to MemSQL, using the MemSQL built-in S3 pipeline, the table was compressed down to 30GB.