asked 2 days ago
Hi Team ,
I have a large set of data , almost about 13 Million rows that I need to represent with the flexmonster dashboard.
As of now the load takes observably large amount of time to load the entire dataset into the browser and then into flexmonster component.
I wanted to ask if we can stream the data from the source and then do a partial load into the flexmonster component ?
⋅ 2 days ago
And to add , Can we explore data compression as well instead of json data input ?
⋅ 9 hours ago
Hello Pranay,
Thank you for contacting us.
Flexmonster does not currently support partial loading of the dataset. To improve the performance, we recommend two possible approaches: using JSON with a custom prefiltering mechanism or using Flexmonster Data Server.
Prefiltering JSON data
When working with JSON, Flexmonster must first download the entire file and only then begin processing it. Hence, the best way to improve performance is to reduce the size of the loaded dataset. To achieve this, we recommend implementing a custom filter view on your end. Users can select filter conditions, and only after the filter is applied, Flexmonster can start loading data from a server-side script that returns JSON.
One of the possible approaches to this is to initialize the loading after the user applies filters with the setReport API call. The filter condition can be passed as query parameters in the data source:
dataSource: {
type: "json",
filename: "https://example.com/api/data?year=2025",
},
There are other ways to optimize the JSON loading process, for example:
useStreamLoader data source option).These can increase loading speed even without prefiltering, but they will not provide significant improvements with a dataset of this size.
Using Flexmonster Data Server
For handling large data sets, we provide a special tool for delegating data aggregation to the server-side - Flexmonster Data Server. It stores the data in a server's RAM, allowing for quick access and processing. Flexmonster Data Server is based on the custom data source API - our custom communication protocol that lets you retrieve already aggregated data from the server and pass it to the Flexmonster pivot table. It provides a significant performance boost for the pivot table, where the data is grouped and the number of rows is lower compared to the initial dataset. However, for a flat table layout, the performance improvement would not be significant since all the data should still be fetched to the client side.
Additionally, we would like to note that our team is actively working on the upcoming Flexmonster 3.0 major update, which will introduce partial data loading via websockets using Flexmonster Pivot Server. You are welcome to check out our roadmap for more details: https://prerelease.flexmonster.com/roadmap
Looking forward to hearing your feedback.
Best regards,
Maksym