Problem loading 1 million rows

Some additional backgroud:
 

• It crashes after it has loaded the data (Accoding to the on-screen load status), while analysing the data
• It crashes aways while analysing the 300.000 th row
• I am loading json data

Also, it is taking forever to load the data in Microsoft Edge, while in chrome it is very fast.
 
Sorry if the information is very vague, but I am trying to give as much information as possible so you can have a broad view of the scenario. Fell free to ask for any specific information.
 
Kindest regards,
Andre Pinheiro

Hi Tanya,
thank you very much for your prompt response.
Yes, we are aware that it is not flexmonster that is crashing, but rather the browser. However, it is a little bit difficult to understand why it is failling with an amount of data that supposely should not be a problem, as per our understanding.

- How your JSON data is getting to the browser?
we have a web service that executes a SQL query, translate it into json and sends the response to flexmonster
- When the browser crashes while analyzing the 300.000-th row, is it happening only for 1.000.000 records of data?
When the number of records is above 300.000, it crashes
- Do you use Flexmonster Data Compressor? It reduces the size of the response from the server.
No, since based on some previous tests it seemd to be more slow than the approach we are using now. Going to execute a new set of tests using the data compressor approach again, and get back to you with the results
- Could you please consider sharing your sample data? (any sensitive information can be sent via e-mail)
Sure, no problem. Going to export it and attach to this ticket
- Also, I would like to inform you that we have released a beta version of Flexmonster Pivot for Elasticsearch. This data source allows loading just the data necessary for each report. As a result, it is possible to handle much larger data because not all data is loaded into the browser at once.
Thanks. If everything else fails, we will take a look into it.

Hi Tanya,
 
I was not able to attach the file here, since it's size is around 27 mega (zipped). Are there any alternatives for me to send you the file?
 
Thanks,
Andre 

If it works for you, I can publish it on our sharepoint site and send you the link.

Hi Tanya,
 
some more feedback:
– Do you use Flexmonster Data Compressor? It reduces the size of the response from the server.
I've tested it, and it also fails, unfortunatelly.  

Hi Tanya,
 
thank you very much. Here is the link for the file:
 
https://vmetrix.sharepoint.com/:u:/g/EQIGPF4vpmxFgkTwUrAxpmgBZhfIYZw2jkJezA_ZdS_6og
 
Kindest regards,
Andre Pinheiro

Hello,
We also have a crash issue with large csv file (316629 rows, 240MB). Here are the details if they can help:
 - Google Chrome 64 bits on a 32 GB ram computer (so more than enough memory)
 - Chrome crashes while "Loading data..." after having loaded about 220MB
 - If developers tools are open, it pauses before crash with message "Paused before potential out-of-memory crash"
 - Sometimes Chrome succeed loading the data (exactly same data) without crashing, but not often.
 - Firefox 64 bits does not crash (and it loads data faster)

Iryna,
 
Thanks for the reply. 
Unfortunately, the given answer is not acceptable. One of the key selling points of flexmonster is being able to load large amounts of data, including having an example of loading 1 million of records in your website, and also I don't see any size limits described anywhere (maybe I have missed it somewhere?).
Regarding the suggestion:

• I am not very familiar with elastisearch, but the main feature that we need from flexmonster is its pivot table caracteristics. Elastisearch would retrieve the data by demand (please correct me if I am wrong), and for this reason I cannot see how the pivot table feature would work
• Even if this suggestion is implemented, in a situation where some user loads the complete data, flexmonster would crash as well, wouldn't it?
• As stated before, We do not have much knowledge using elasticsearch, and adding an additional component to the project is not a feasible solution at this point, since it would involve additional development and even possible licenses costs.

 
Kind regards,
Andre Pinheiro

Hello Dmytro,
 
thanks for your detailed answer. Please find my comments in bold:
 
We were always interested in increasing the capabilities of our component. Currently, our developers are working on adding some possible optimizations to make your JSON work. 
 
Thanks. However, we are also able to reproduce it using Flexmonster's OCVS format, thus not being a isolated JSON "issue". If you wish I can send the same JSON content in OCVS format.

At the same time, we would like to pay your attention to the fact that there is always the browser’s resources limitation and your dataset is very close to it. In the modern world, it is only a matter of time for the data to keep growing. When you load a huge dataset into the browser’s memory it always has the risk to be crashed. Consistent data growth is a way for a consistent browser’s crashing.
I understand that. However, based in one of the main flexmonster selling points, we grew our espectatives around the 1 million number, even adding a message to the end user if he tries to load more than this amount of data. So far we are only able to load around 300.000 rows with around 10 fields with values, far away from the promissed 1 million...

 
The usage of modern data storages is the way to be prepared for such sort of situations. Our team has certain expertise when it comes to working with massive datasets. There is a number of our clients that have outgrown the plain data sources such as CSV and JSON. At the same time, their requirements for the performance and loading speed remained at the same level. Such sort of challenges leads to the total change of the paradigm. All of that can be summarized to the following idea: “When the browser can no longer handle the whole data set it is time to delegate some calculations on the server side”. That is the reason why we support such things as Mondrian, SSAS and Elasticsearch. All of them are powerful data engines which allow making some pre-calculations on the server side. Such an approach allows Flexmonster to obtain already aggregated data and successfully displaying it. In such case, the component stores fewer data into the browser’s memory, since it is already aggregated, but at the same time, it can show significantly more. 
The main benefit here is that the user experience remains the same. Our key pivoting features such as filtering, expanding, drilling etc… remains the same for the end user. The only difference is that the user keeps working with the massive dataset in real time the same way he worked with a much smaller one.
But if I delegate it to the server, I would take away from the user the hability to set the aggregation by itself, wouldn't I? I would have to have another "tool" or graphical interface for the user to create specific aggregation queries. And it also would not make sure that the memory problem would not occur, since I cannot be sure about the amount of data that the user can load without crashing the browser.

Elasticsearch is a free and open-source solution for structured and unstructured data with a huge community. It has good potential for scaling depending on your data needs. We recommend you Elasticsearch since it is one of the most promising storages from all of our supported data sources. All of the pivoting features are available in the real-time. You are welcome to try it in our live demo: https://www.flexmonster.com/demos/js/connect-elasticsearch/. Also here is the example with the much bigger data source which has almost 500K unique members: http://jsfiddle.net/flexmonster/pq7cu9rs/.
Thanks. But the example have only two fields, pre-aggregated. It agains gives a not accurate feeling that I would be able to load 500.000 rows with more columns. The same occurred with the One Million example. If you take a closer look, it only loads this amount of rows becase not all columns have values.
 
How can I calculate the client machines specs in order to be sure about the amount of data can I load at once, without crashing?
 

Please note that you can always rely on our assistance in case you decide to switch the data source.
Let us know in case of any other question.
Regards,
Dmytro

Hello Dmytro,
 
first of all, happy new year!
 
Thanks for the update. Looks very promissing. Looking forward to this.
 
Regarding elastisearch, I will take a look at it more carefully. Thanks.
 
Kindest regards,
Andre Pinheiro

Hello Dmytro,
Thank you and your team for your effort. We are going to perform some tests and revert with results soon.
 
Kindest regards,
Andre

Hello
We are facing the same issue. But our data have more than 1 million rows with lots of columns (about 60). 
The error message is the same: “oh, No! something went wrong on this webpage...”
The error occurres in almost all user's computers with 8GB/4GB.
When running with the devtools of chrome openned (F12 key pressed), it displays a debugger message (Paused before potencial out-of-memory crash) and when you press play on the debugger, the system is able to open it on theirs computers (8GB/4GB).
We can send to you by email the videos.

Thank you for the answer. But I can't understand. Can you explain why it works by giving play to the debugger? the user must not use the browser as an open developer