asked on January 21, 2019
Hello,
We need to handle tens of millions of records in the Pivot table (for example, 50 million records).
What is the best practice for handling such amount of data with the FlexMonster Pivot Table?
Thanks.
Regards,
- Luis
⋅ January 22, 2019
Hello Luis,
Thank you for posting your questions here.
50 million records is a large data set and as the result, it is almost impossible to handle it with simple data sources such as CSV and JSON.
Therefore it is necessary to delegate some operations on the server side. From our experience, the best solutions for such large datasets are SSAS (SQL Server Analysis Services) and Elasticsearch. Please see more information about them here: https://www.flexmonster.com/doc/connecting-to-microsoft-analysis-services/, https://www.flexmonster.com/blog/introducing-flexmonster-pivot-for-elasticsearch/. You can choose the one that fits you better.
Please let us know in case you need more guidance from us.
Regards,
Dmytro
⋅ January 22, 2019
Thanks for the information, Dmytro.
I am evaluating the option of delegating operations to our proprietary analytics server which exposes REST Services to retrieve data, perform aggregations, etc. Does the Pivot Table have a way to "plug in" calls to my own REST services to retrieve data, retrieve dimensions, perform aggregations, etc.?
Thanks.
Regards,
- Luis
⋅ January 22, 2019
Hello Luis,
Thank you for your feedback.
Currently, we do not have such REST API available.
The ability to allow the users using their custom data source based on the REST API is already in our roadmap to version 2.8.
Right now, we are very close to releasing the major version 2.7.
Our dev team has already started working on that task but it is a rather complex thing and we are not ready to provide you with the ETA.
However, we will notify you as soon as we are ready with the beta.
Regards,
Dmytro
⋅ January 28, 2019
Thanks, Dmytro.
Yes, please notify me when you are ready with the beta. Also, please notify me when you have an ETA for the beta and for the Release containing this functionality.
Thanks again.
Regards,
-Luis
⋅ January 29, 2019
Hello Luis,
Thank you for your feedback.
We are glad that you are interested in such an update.
We will return to you as soon as there are some results.
Regards,
Dmytro
