asked on September 13, 2024
Hi Flexmonster team,
We have run into a scale issue that has brought us to a halt. Here is our scenario. We have a customer's dataset we are aggregating and displaying in a pivot table that is made up of 30M rows. We are currently using the Flexmonster Custom Data Source API.
The most granular dimensions in our pivot table are contract IDs - there are about 3M unique values for these. Anytime we add this as a row or a report filter, loading the members takes minutes because the client library requests all the members contained in the displayed result set. The thing is, we already have them in the select response when we have it as a row.
We saw this improvement back in 2022 to not need to request members when the data element is set to be not filterable: https://www.flexmonster.com/question/custom-datasource-api-avoid-members-call-on-some-fields/
There is also a filter option to prevent live search and instead require someone to type in a search for a filter and press enter to see the results. Here is Flexmonster's fiddle example: https://jsfiddle.net/flexmonster/w0e1aykt/
Our customers need to be able to search and filter by contract IDs, but they don't need to see all the contract IDs expanded for a non-filtered dataset. What we want to do is make it so ContractID isn't needed by members, but does allow for server side searching, only returning back the members that match the non-live search box to then allow the user to select or deselect. Is there a way to achieve this or make this an improvement to the library?
Thanks,
Bill
⋅ September 16, 2024
Hello, Bill!
Thank you for reaching out to us.
We have tested the described use case on our side and recommend sending an empty members list for the "Id" field to the client.
The idea is that for the field located in the Report filters no members are displayed on the grid. So, you can send an empty array with the response for /members request for the "Id" field.
This way, the label filter for the "Id" field remains working as it uses the client input for filtering, and no members are sent to the client.
The example response should look as follows:
{
members: []
}
Please try the suggested approach on your side and let us know if it works well for you.
Looking forward to hearing from you.
Kind regards,
Solomiia
⋅ September 16, 2024
Solomiia,
I work with Bill and took a look at your recommendation. There are a couple issues with this solution that would prevent us from using it.
First, this will only work when the field is a Report Filter. Once it is moved into a row(or column), the client will not expand any rows for the given field. Previously, in this thread it looks like there was a solution for this by setting `"filters": false` in the mapping config for the field. This fixes the problem when the field is a row, Unfortunately when we do this, the field cannot be used as a report filter. This is the biggest blocker for us since we need to support adding this field as a row at times.
Second, It is somewhat confusing when I open the report filter that it shows me the selection filter with "0 of 0 selected". The user has to have the knowledge to click into the labels to be able to apply a filter. They are still able to use the search bar but it doesn't do anything.
Third, when setting members to [], there appears to be a bug that once I set a filter via the "label" filter I am unable to ever remove the filter. If I clear the label filter, I am unable to click "apply" to close the filter modal with no filters applied.
Is there a solution that would allow us to use the field as either a row or a report filter without needing to make a members request that returns every single possible value for the field? Given its possible to not select all members when using `"filters": false`. It seems like it should be possible to add the ability to add a "labels" only filter for a field without making a members call for all possible values. An example of what this might look like is here(I edit this video to what it might look like, it doesn't work like this today).
Thanks,
Brian
⋅ September 17, 2024
Hello, Brian!
Thank you for your feedback and additional details about the use case.
To maintain the best user experience for this use case we suggest replacing the default filter pop-up with your own custom filter control. This way you can provide the filter inputs you need (for example, have only a labels filter) and use Flexmonster API calls to apply the filter underneath. And it would be easy to add your own search with server-side filtering logic.
We have prepared a JSFiddle sample with predefined members to illustrate the idea: https://jsfiddle.net/flexmonster/9tg68b4w/.
Also, our team is going to research possible ways to avoid loading all the members to the pivot table, to cover the use cases where the whole list of members is not needed immediately. We will be back to you with the research results with ETA October 15th.
Please let us know if you need our further assistance in configuring a custom filter control in Flexmonster.
Looking forward to hearing from you.
Kind regards,
Solomiia
⋅ September 17, 2024
Hi Solomiia,
Thank you for the response. While the custom filter example is interesting, it really doesn't buy us anything because we are currently in a catch-22. If we allow the column with massive amounts of unique data to be filterable, the members API call the library makes under the covers kills our performance. If we make the column as not filterable, then the gear icon disappears and we can even leverage the event override.
Is there a way to tell the library to show the gear icon on a non-filterable column, even if it doesn't trigger the default event? That would allow us to then implement our own custom event handler to make a server side call outside of flexmonster without us having to do crazy hacking on the javascript to try to determine what cells should have a filter icon but dont.
Ultimately preventing the call to members for members we already have in the selected result object would be preferrable, but we could at least work around this for now if we can renable the gear icon for filtering without it making the call to members.
Thanks,
Bill
⋅ September 18, 2024
Hello, Bill and Brian!
Thank you for your swift response.
Please note that with the filters:false set for the field, this field won't be displayed in the Report Filters area.
So, we assume the approach where "Id" field is located only in Rows/Columns and there are no /members request works for you.
Then, it is possible to query the necessary HTML elements to restore the gear icon on the cell header and add an event listener to it: https://jsfiddle.net/flexmonster/cx8gknub/.
Regarding our improvements research, our team will get back to you if there are any updates on the matter within the given ETA.
Feel free to reach out in case of any other questions.
Kind regards,
Solomiia
⋅ 5 hours ago
Hello, Bill and Brian!
Hope you are doing well.
Our team is happy to inform you about our research updates. As a result, a new filters.members property for the custom data source API was added to the mapping.
When filters.members is set to false, the /members request won't be sent for that field. Instead, the members will be read from the /select response to display the data on the grid.
In addition, the filter icons will remain visible and the field can be used in the report filters area. This way you can use the filter icons and replace the default filter pop-up with your custom one.
You can see how the new property works in this JSFiddle: https://jsfiddle.net/flexmonster/k2L90rpb/.
Also, kindly note that in the next minor release, the selection filter won't be available in the default filter pop-up window for fields that have filters.members set to false. Only the labels and values filters will be available.
The feature is included in the 2.9.88 version of Flexmonster: https://www.flexmonster.com/release-notes/version-2-9-88/.
You are welcome to update the component: https://www.flexmonster.com/doc/updating-to-the-latest-version/.
Please let us know if the new property is helpful for your use case.
Kind regards,
Vera
⋅ 2 hours ago
We are going to be busy for the next week on some important projects, but once we get a chance to breath we will implement this change and let you know how it works!
