Inquiry about handling large files in Flexmonster (.NET Server) and client-side optimization

Hello Maksym, thank you for your prompt reply!

Here is the additional context you requested:

1. View used when the issue occurs

• We use both pivot and flat views.

• The problem happens more often in flat view, due to the large volume of data being rendered on the client side.

1. Concrete scenarios (same CSV ~4.1 GB)
   All three cases use the same CSV (~4.1 GB). In every case, the load takes a long time and often results in Out of Memory errors in the browser or server errors.

   a. Flat view with row and column filters

   • Expected result: ~6,000 rows × 29 columns (both row and column filters applied).

   • Symptom: very long load times and OOM errors on 8 GB RAM machines.

   b. Flat view with high row count

   • Expected result: ~3,350,000 rows × 5 columns (only column filters applied).

   • Symptom: very long load times and frequent OOM/server errors on 8 GB RAM machines.

   c. Compact (pivot) view with hierarchies

   • Even in compact view, the same dataset still loads slowly and fails in many attempts.

   Note: On 32 GB RAM PCs these scenarios usually load, while on 8 GB RAM PCs (very common among our customers) they lead to OOM or crashes.

2. Are all rows shown in flat view or are filters applied first?

• In (a) and (b), filters are applied before rendering (always column projection; in (a) also row filters).

• Still, in (b) the resulting dataset is very large (~3.35M rows).

1. Example report configuration (JSON)
   Below is an actual example of a report template that reproduces one of the problematic scenarios:

{
"dataSource": {
"type": "api",
"url": "https://nubimetrics.com/api/cube",
"index": "SAS CSV",
"singleEndpoint": false,
"mapping": {
"Periodo": { "visible": false },
"Periodo.Year": { "caption": "Período.Ano" },
"Periodo.Month": { "caption": "Período.Mês" },
"Periodo.Day": { "caption": "Período.Dia" },
"Period.Year": { "caption": "Período.Ano" },
"Period.Month": { "caption": "Período.Mês" },
"Period.Day": { "caption": "Período.Dia" },
...
},
"withCredentials": false,
"concurrentRequests": false
},
"slice": {
"reportFilters": [
{ "uniqueName": "Categoria_Nivel_3", "sort": "asc" },
{ "uniqueName": "Categoria_Nivel_4", "sort": "asc" },
{ "uniqueName": "Categoria_Nivel_5", "sort": "asc" },
{ "uniqueName": "Categoria_Nivel_6", "sort": "asc" }
],
"rows": [
{ "uniqueName": "Nombre_Tienda_Oficial", "filter": { "members": ["nombre_tienda_oficial.[loja.oficial.15289]"] }, "sort": "asc" },
{ "uniqueName": "Marca", "sort": "asc" },
{ "uniqueName": "Nickname_Vendedor", "sort": "asc" },
{ "uniqueName": "Categoria_Nivel_2", "sort": "asc" },
...
],
"columns": [{ "uniqueName": "[Measures]" }],
"measures": [
{ "uniqueName": "Unidades_Vendidas", "aggregation": "sum", "active": true },
{ "uniqueName": "Monto_Vendido_Moneda_Local", "aggregation": "sum", "active": true, "format": "31spc8p5" },
{ "uniqueName": "Mes.Day", "aggregation": "sum", "active": true },
...
],
"flatOrder": [
"Nombre_Tienda_Oficial",
"Marca",
"Nickname_Vendedor",
"Categoria_Nivel_2",
...
],
"flatSort": [
{ "uniqueName": "Unidades_Vendidas", "sort": "desc" }
]
},
"options": {
"viewType": "grid",
"grid": {
"type": "flat",
"showFilter": true,
"showHeaders": true,
"showTotals": "off",
"showGrandTotals": "off",
"drillThroughMaxRows": 1000
}
},
"formats": [
{
"name": "31c9dq7y",
"thousandsSeparator": ",",
"decimalSeparator": ".",
"decimalPlaces": 0,
"currencySymbol": " USD"
},
{
"name": "31c9i9md",
"thousandsSeparator": ".",
"decimalSeparator": ",",
"decimalPlaces": 2,
"currencySymbol": "$"
},
{
"name": "31spc8p5",
"thousandsSeparator": ".",
"decimalSeparator": ",",
"decimalPlaces": 0
}
],
"localization": "/Locales/pt/pivotTable.json?v.0.1.0",
"version": "2.9.92",
"creationDate": "2025-01-15T11:22:45.052Z"
}


Additionally, I confirm that we are using the Flexmonster Data Server from your official GitHub repository (.NET), and in our workflow we only load CSV files, with some customizations applied server-side (aliases, date formats, data types).

We would highly appreciate your recommendations regarding:

• Possible Data Server adjustments to reduce memory usage in these scenarios (parameters, patterns).

• Suggestions for client-side optimizations to handle memory in Flat view with large datasets (e.g., segmented/paged loading strategies, recommended limits).

If needed, we can also share more report JSONs or detailed logs/error traces.

Thank you again for your support!

Best regards,
Alejandro Gallardo

Hello Maksym,

Thank you for your suggestion and for sharing the example with filter.members = false.

We applied this optimization on our side (disabled member filters for the high-cardinality fields). However, when testing with a ~500 MB CSV, the browser still shows the following error:

“The file is too large and cannot be loaded completely due to the browser limitations.”
(Screenshot attached)

This happened in flat view, even after disabling member filters.

Please let us know if there are any additional optimizations we can try (either server-side or client-side) to prevent this browser limitation, or if you recommend a specific upper bound on rows/columns for flat view to avoid such errors.

Thanks again for your support and guidance.

Best regards