Elasticsearch Paging & terminate_after with Flexmonster

Hello, Cho!

Thank you for reaching out to us.
We are currently working on your questions and will get back to you soon.

Best Regards,
Maksym

Hello, Cho!

Please note that Flexmonster does not support pagination for any data source out-of-box. The component always loads the entirety of the data specified in the data source. Despite that, the pagination can still be implemented by adding UI controls that will change the data source to load a different subset of data.

The additional sample you provided with manual page querying and setting them as JSON can be a solution for implementing the pagination for Elasticsearch. However, this approach would have limitations regarding the volume of data loaded per page. With JSON data sources, all the data is loaded on the client. Hence, the amount of data that can be loaded and the speed of aggregations depend on the RAM/CPU available to the browser. Usually, loading JSON datasets exceeding 100MB would cause performance issues.

If the solution to retrieving pages as JSON by querying your Elasticsearch instance does not work for you, we suggest implementing a custom data source API server. This server would implement our communication protocol between the Flexmonster client and the server. With this approach, a new query is formed on every change in the report, like expanding or filtering data, and data is loaded by parts. On the server, you can transform requests from Flexmonster to Elasticsearch queries. This would allow more customization options, including the ability to process and handle parameters added with the customizeAPIRequest method, including the paging. On the downside, this solution would require much more significant development effort than a direct connection to Elasticsearch and querying JSON data from Elasticsearch.

To evaluate all the data source options available and check which one would be better for you, we recommend reading this article: https://www.flexmonster.com/blog/how-to-choose-the-best-data-source-to-use-with-flexmonster/ 

Please let us know which solution would work better for you.

Best Regards,
Maksym

Hello, Cho!

Hope you are doing well.
We are wondering if you had time to evaluate the available options for using Elasticsers's paging with Flexmonster.
Looking forward to hearing your feedback.

Best Regards,
Maksym

Hello, Cho!

Thank you for your reply.

Please note that Flexmonster does not support working with array field values. To show the data from an array, we recommend flattening the data for it to match one of two supported formats:

• An array of objects, where each object is an unordered set of key-value pairs.
• An array of arrays, where the first array contains field names, while other arrays are an ordered collection of values.

Referring to the given example, this can be done by creating two objects, each of them containing data from the first and second object in the "infoCount" array:

[{ 
"currentTime": "2024-03....", 
"activeUsers": 120, 
"errorPer": 0.05, 
"operationStatus": "Active", 
"type": "AA", "instances": 40, 
"responseTime": 200 
}, 
{ 
"currentTime": "2024-03....", 
"activeUsers": 120, 
"errorPer": 0.05, 
"operationStatus": "Active", 
"type": "AB", 
"instances": 40, 
"responseTime": 200 
}]

However, such duplication of top-level values can cause changes in their resulting aggregations. For example, the sum of "activeUsers" in the dataset above would be 240. 

With this limitation of the JSON approach, we agree that the custom API can be a viable option for you. With the custom API, you will fully control the data aggregations. This allows you to create a custom logic for calculating each aggregation to avoid the impact of duplicated data on the results.

We are looking forward to hearing your feedback.

Best Regards,
Maksym

Hello, Cho!

Thank you for your reply.

Could you please provide us with a current Flexmonster report configuration? Also, we would like to know more about your use case, in particular for complex array-like values. Do you need only to display them, or do you need to aggregate the data from arrays?
Knowing this information would greatly help us guide you through the integration of Flexmonster.

Looking forward to hearing from you.

Best Regards,
Maksym

Hello, Cho!

Thank you for sharing more information with us.

Working with complex data structure

Please note that Flexmonster does not support data structures like arrays out-of-the-box. Working with arrays would require transforming the complex data structure to display its data in a single string label or numeric value. For instance, while working with arrays, you can flatten the data as described in our previous reply or reduce array values to a single value (for example, display the sum of numeric values in an array). For your case, we recommend considering one of the following approaches depending on the data source.

While loading the data from Elsaticsearch as JSON, we suggest performing these aggregations on arrays before passing them to a client for better performance. Thus, the result of the Elasticsearch query should already contain preprocessed array data.

Using the custom API approach would grant more flexibility. As a result, you can receive different aggregations of arrays based on user requests instead of loading predefined summary fields based on array data. With "api" data source, Flexmonster will form a new query on every change in the report, making the custom API more similar to our native integration with Elasticsearch. On the downside, implementing the custom data source API server might be more time-consuming than other data sources. You are welcome to find more information about the custom API server implementation in our docs: https://www.flexmonster.com/doc/implement-custom-data-source-api/ 

Displaying a single array

Speaking of displaying a single array of data, it is possible to access detailed values within arrays, as you described.

Firstly, you should display the initially preprocessed string/value from the array on the grid. Then, to open the detail view popup, you can use cellclick or celldoubleclick event handlers. When a cellclick or celldoubleclick event occurs, the event handler receives the CellDataObject with the information about the members that represent the cell. Having the information from the cell object, you can retrieve the array from your dataset by filtering the copy of data loaded from Elasticsearch or performing a new query to find it.

The data received for the detailed view may not be structurally connected to data from the pivot table. Therefore, we recommend replacing our default detailed view with a custom pop-up that has a separate Flexmonster instance.

Leverage the Elasticsearch mapping with JSON data source

The Elasticsearch mapping is different from the mapping supported for JSON datasource. We recommend transforming the mapping used in Elasticsearch into the appropriate format. Check out our guide for using mapping with JSON data sources: https://www.flexmonster.com/doc/managing-data-presentation-json/ 

Please let us know if the provided information was helpful. Feel free to contact us if any additional questions regarding some part of implementing such functionality would arise.

Best Regards,
Maksym

Hello, Cho!

Hope you are doing well.
We would like to know if the information provided was helpful for your project. Were you able to successfully implement the suggested approaches for mapping and handling complex data structures and arrays in your data source?
Looking forward to hearing your feedback.

Best Regards,
Maksym