asked on May 27, 2024
Hi,
I GZip the csv data from the server to the FlexMonster component in the browser and so far never had a problem.
Now there is a new issue. If the data is streamed the sum aggregate is wrong. But the same file CSV file if opened in excel or loaded as local CSV file in FlexMonster (even on your website) shows the correct aggregate value.
I am attaching the sample data. Please extract the CSV from this GZip file, load this data in FlexMonster and use the Measure called "WA VALUE". It's total should be (659,233,908.1489869). But if the same file is streamed online all the totals come wrong. It's like a complete disaster. Some string columns also get wrong data.
Also if I try to disable GZip the file loads to 96% or 98% then stops.
The file size is too large to attach so please download it from here, it is gzipped (the gzip file is in an encrypted zip so please send me an email for the password) (but please make sure you stream a gzipped output from your webserver to the flexmonster component):
⋅ May 28, 2024
Experimented a lot more and realized it's working perfectly in FireFox but not in Google Chrome or Edge (guess they are both Chromium).
I think it has something to do with how the browser unzips the gzip file. Must be the latest update.
⋅ May 29, 2024
Hello, Shripal!
Thank you for reporting this issue and sharing the access credentials via email.
We have started investigating this loading behavior. The information and sample data provided would greatly help us. Additionally, could you please tell us which version of Flexmonster the issue occurs on?
Looking forward to hearing from you.
Best Regards,
Maksym
⋅ May 29, 2024
I am using the latest version. May 2024 I think. I upgraded to the latest version before I started investigating.
⋅ May 30, 2024
Hello, Shripal!
We could reproduce these issues with loading and not finishing without GZip and showing incorrect results with GZip. Our developers will investigate these cases and provide a fix in one of the following minor releases, ETA June 24th.
Meanwhile, we recommend using the Flexmonster Data Server—our special server-side tool responsible for fetching data from a data source, processing and aggregating it and returning it to a client in a ready-to-show format. This would not only address the current issue but also offer much better performance when working with such large files.
Please let us know if using FDS is an option for you.
Best Regards,
Maksym
⋅ May 30, 2024
Dear Maksym,
We never had these problems in Chrome before. Maybe their new update has messed up something. FireFox is working perfectly. This is just for your knowledge.
The data server would be of no use to us because it requires pre-defined queries, and we have dynamically run queries that are populated to a CSV and then streamed to the client.
If the data server can allow us to specify the variable values to the stored procedures, we don't mind using the same.
Please guide if this is possible.
⋅ May 31, 2024
Hello, Shripal!
Thank you for your response.
We agree that this issue is most likely caused by some recent Chrome update. Indeed, Flexmonster Data Server does not support dynamic queries because it is designed to maximize performance with static data.
Please stay tuned for updates regarding the fix.
Best Regards,
Maksym
⋅ June 25, 2024
Hello, Shripal!
Hope this message finds you well.
After researching the issue with large file loading, we discovered that after one of the recent updates in Chrome, its XHR loader was limited to only 256 MB of data. Flexmonster relies on XHR requests, which causes incomplete data loading and errors with larger files. Furthermore, the issue cannot be reproduced in Firefox or by loading the file from a local file system, confirming the issue's specificity to Chrome.
As a workaround, we suggest switching from CSV to JSON format and utilizing the useStreamLoader data source parameter. This method bypasses Chrome’s limitations by streaming data, allowing complete loading of large files in chunks. We recommend using an array of arrays format of JSON data because it is similar to a CSV, meaning it can be relatively simple to switch to JSON.
Please let us know if any questions arise.
Best Regards,
Maksym
⋅ July 23, 2024
Hello, Shripal!
Hope you are doing well.
Our team is wondering if you had time to check our previous reply. Does the proposed solution with the useStreamLoader parameter work for you?
We are looking forward to hearing from you.
Best Regards,
Maksym
⋅ July 24, 2024
Dear Maksym,
We are still to do the same, been very busy with other things. Currently all the datasets we have are loaded as CSV so it will be a bit time consuming to do the same.
⋅ July 25, 2024
Hello, Shripal!
Thank you for your response.
Our team will be awaiting your feedback when you get back to the issue with data loading.
Best Regards,
Maksym
