Horizontal scaling

Hello Varun,

Thank you for reaching out to us.

Flexmonster Data Server DLL can be horizontally scaled across multiple containers with one instance of the Data Server per container. The requests are independent of the server state, allowing requests from a single pivot to be sent to different servers with the same index loaded.

As for the memory allocation, the Data Server typically requires from 2 to 3 times the amount of RAM compared to the original dataset size. However, this number may vary depending on the data structure, cardinality of values in columns, and volume of cached responses. Also, the memory usage may spike on index reload when the KeepDataOnRefresh option is enabled. We recommend testing the performance with your data to get more specific details on memory allocation.

Please let us know if more questions arise.

Best regards,
Maksym

Hi, Varun,

Thank you for sharing these infrastructure details with us.

Our team prepared some recommendations based on the provided autoscaling setup overview.

Autoscaling rules

Your autoscaling rules (scale up at 90%, down at 25%, min 1 / max 5) are generally reasonable, if executed based on the CPU usage. The RAM usage in FDS is relatively constant once the data is fetched. Scaling up at peak usage can work to improve performance, but please note that initializing a new FDS instance can take a long time, since data for indexes must be loaded into RAM.

RAM measurements

FDS stores indexes in RAM, and each running instance holds its own copy of the data. Therefore, each of your containers must have enough RAM to hold all the indexes it serves. With 10-15 indexes and up to 1M records each, 16 GB may not be sufficient to store all indexes in a single instance. However, this strongly depends on the number of fields, their types, and the size of their contents.

For simpler management, we recommend running a single FDS instance per container with all required indexes loaded. Later, if you identify that some indexes are accessed significantly more frequently, you can scale selectively around those.

Session persistence

FDS uses a stateless protocol, so it is not an issue if user requests are handled by different FDS instances. User sessions will not be lost due to scaling, provided that at least 1 copy of the queried index is always loaded into FDS.

However, there is nuance with caching. The built-in cache used in FDS is local and cannot be shared. If you would like to implement a shared cache, you can disable the built-in cache with the CacheSizeLimit config property and implement a shared request-response cache on your side.

To sum up, your autoscaling logic is sensible for improving performance during peak load. You would need to ensure that at least one instance with a loaded index is always up and running, and validate that the allocated RAM is sufficient for real data.

Please let us know if more questions arise.

Best regards,
Maksym

Hello, Varun,

Thank you for reaching out to us.

An out-of-memory crash can happen at the moment of a new allocation, even if reported usage is around 75%. If FDS requests a larger chunk of memory which exceeds the container limit, it can cause the out-of-memory error before all the memory is used. This could happen when FDS is trying to create a large index. We recommend increasing the total amount of RAM available per container.

Feel free to contact us if more questions arise.

Best regards,
Maksym

Hello, Varun,

Thank you for your questions.

Waiting for the next ECS container due to memory constraints

This question seems to be outside of Flexmonster's scope. We recommend checking AWS/ECS settings or consulting AWS support on this part.

Error message displayed & capturing pre-error state

Given that you reported OOM errors, the error in /select request you shared with us is likely caused by FDS running out of memory before that specific request is sent. However, to identify the exact cause of this issue, our team recommends enabling the Data Server logs and examining them after the error occurs. Please check this article to see how the logs can be enabled and configured with FDS DLL: https://www.flexmonster.com/doc/configure-logging-fds-dll/
Regarding your question about capturing the pre-error state of the grid, this is not supported in Flexmonster. Resolving the root cause of the issue is recommended rather than attempting to handle such errors on the client side.

Memory not being released

There are several main reasons that may cause the memory consumption to grow after the index was initially loaded:

• Data reload: if you configured the data reload by setting the DataRefreshTime and have the KeepDataOnRefresh enabled, FDS keeps the previous copy of the index available while the new one is being loaded. While it improves data availability, it can cause significant memory spikes on reload.
• Response caching: by default, FDS caches the responses, which helps to speed up the data loading in cases of cache hits. The cache grows continuously with each request until it reaches the limit specified by the CacheSizeLimit option.

We recommend experimenting with these settings to see how they affect memory usage and stability in your specific environment. The most conservative approach to minimize memory usage is to disable response caching and data refresh.

Please let us know if more questions arise.

Best regards,
Maksym

Hello, Varun,

Thank you for the further explanation.

When an index is deleted, FDS does not explicitly trigger garbage collection. As a result, the memory previously used by that index may not be released immediately and can remain allocated until the .NET garbage collector reclaims it.

Additionally, upon another look at the previously reported error, we have a theory that it may also occur when attempting to access the index while it is reloading. However, as mentioned previously, reviewing the logs is the only reliable way to identify the exact cause: https://www.flexmonster.com/doc/configure-logging-fds-dll/

Feel free to contact us if more questions arise.

Best regards,
Maksym