asked on May 31, 2022
Hi, I have a huge amount of dataset of about 50 million rows. My Database is SQL Server. First I was using Flex Monster Data server, I created an index and connect to SQL Server by string, when I run that my laptop hangs and I need to restart my laptop. Then I tried to get data from sql server through nodejs but unfortunately that also don't work for me. Can anyone kindly help? what should I do?
Or any other solution for this. How can I load 50 million rows data in Flexmonster??
⋅ May 31, 2022
Hello, Osama!
Thank you for your question.
Please note that Flexmonster Data Server loads the entire dataset into RAM. This is required to perform faster data aggregations on the server side. The described issue with loading 50M rows into Flexmonster Data Server is most likely to happen due to a lack of RAM.
We suggest decreasing the amount of data loaded into the Flexmonster. To do so, you can add filters inside the SQL query:
SELECT TOP x * FROM table_nameSELECT * FROM table_name WHERE conditionAlternatively, you can try loading the data using the machine with more RAM available.
Also, could you please tell us which view type(compact, classic or flat) you are using in the Flexmonster component?
Feel free to contact us in case other questions arise.
Best Regards,
Maksym
⋅ June 1, 2022
Thanks for your response! I can't use filters in my query because I want to load full data.
I am using the default view type.
⋅ June 1, 2022
Hello, Osama!
Thank you for your quick reply.
If you want to load the entire dataset without filtering, a machine with more RAM is needed to run Flexmonster Data Server.
Alternatively, you can change the data source to Elasticsearch or Microsoft Analysis Services. This will allow the processing of the same amount of data without loading it directly into RAM. For more information, please refer to the following tutorials:
With a default view type, the client machine will be capable of displaying aggregated data, as it would be much smaller than the original dataset.
Feel free to contact us in case any questions arise.
Best Regards,
Maksym
⋅ June 8, 2022
Hello, Osama!
Our team is wondering whether you tried running Flexmonster Data Server with more RAM or changing the data source type.
Please let us know which approach is preferable for you.
Best Regards,
Maksym
⋅ June 8, 2022
I've not worked on it yet. When I will do I will surely respond
⋅ June 30, 2022
Hi, Osama!
We would like to know if you had time to check the provided solution.
Looking forward to hearing from you.
Best Regards,
Maksym