Read data from S3 bucket

Hello, Varun!

Thank you for writing to us.

Flexmonster supports connecting CSV and JSON files from cloud storage, such as AWS S3.
Since S3 files often require authentication, we would like to mention that Flexmonster allows setting request headers with credentials using the requestHeders data source object property.
For reference, you can check this example, which loads a CSV file from our CDN: https://jsfiddle.net/flexmonster/u5Lp4k98/

Please let us know if our answer helped you.

Best Regards,
Maksym

Hello, Varun!

Thank you for providing additional context.

Flexmonster Data Server does not support connecting to S3 files. However, there are several potential solutions for working with CSV files on S3:

1. You can implement a server-side script that downloads a CSV file to the server and then proceeds with the default logic for creating a CSV data index.
2. Alternatively, you can create a custom parser that would be responsible for loading and processing the data without the need for an additional step of downloading its copy to the server. You can find a guide for implementing a custom parser in our documentation: https://www.flexmonster.com/doc/implementing-custom-parser/

Please let us know if one of these approaches would work for you. Also, we would like to know if you are asking about working with CSV files on S3 because it would just be convenient to have, or because it is critically important for your task?

Looking forward to hearing from you.

Best Regards,
Maksym

Hello, Varun!

Thank you for your reply.
We are working on an example that demonstrates dynamically creating a custom index by loading a CSV file from a URL. We will get back to you once it is ready.

Best Regards,
Maksym

Hello, Varun!

We have modified the sample .NET project with Flexmonster DLL to include the custom parser, which reads remote CSV files from a URL. The project is available on our GitHub: https://github.com/flexmonster/flexmonster-data-server-dll/tree/csv-url-parser-runtime-index-creation

You can find a quick breakdown of what was added or changed below:

1. Added CustomParser.cs - CSVURLParser parser implementation, based on the existing CSVParser. Its type is set to "csv_url" to make it distinguishable from other index types, and the Parse() method has been updated to fetch CSV content over HTTPS using HttpClient(see line 72). The implementation can be further adapted for S3 or other sources.
2. Added CsvUrlIndexOptins class - this class extends the existing CsvIndexOptions. Our implementation only overrides the type; however, you can add further changes to pass more settings, such as AWS credentials.
3. Integrated runtime index creation into the IndexCreateService.cs - the config for the new index is registered with the CsvUrlIndexOptions.

Please let us know if the provided sample helped you.

Best Regards,
Maksym

Hello, Varun!

Thank you for your question.

These can be implemented within the dedicated service for managing indexes, or by adding methods to the existing IndexCreateService:

using Flexmonster.DataServer.Core.DataStorages;
using Flexmonster.DataServer.Core;
using System.Threading.Tasks;
using Microsoft.Extensions.Options;
using Flexmonster.DataServer.Core.Parsers;
using System.Collections.Generic;


namespace DemoDataServerCore.Controllers
{
    public class IndexManagementService
    {
        private readonly IDataStorage _dataStorage;
        private readonly DatasourceOptions _datasourceOptions;

        public IndexManagementService(IDataStorage dataStorage, IOptionsMonitor<DatasourceOptions> datasourceOptions)
        {
            _dataStorage = dataStorage;
            _datasourceOptions = datasourceOptions.CurrentValue;
        }

        public Dictionary<string, IndexOptions> GetAllIndexes()
        {
           return _datasourceOptions.Indexes;
        }

        public void DeleteIndex(string indexName)
        {
            if (_datasourceOptions.Indexes.ContainsKey(indexName))
            {
                _datasourceOptions.Indexes.Remove(indexName);
                _dataStorage.Remove(indexName);
            }
        }
    }

}

Note that the service must be registered within the Startup.cs:

services.AddScoped<IndexManagementService>();

These service methods provide a clean abstraction, and you can implement and configure controller endpoints using them.

Please let us know if our answer helped you.

Best Regards,
Maskym

Hello, Varun!

Thank you for your questions.

To parse JSON files from S3 storage, you should follow a similar process to adding a CSV parser from URL, as it was described in our previous reply: https://www.flexmonster.com/question/read-data-from-s3-bucket/#answer-73928

• Create a custom parser class based on existing implementations of the JSON parser, which is attached to this reply.
• Implement a custom class for index options:

public class JsonUrlIndexOptions : JsonIndexOptions
{
    public JsonUrlIndexOptions(string path) : base(path)
    {
        Type = "json_url";
    }
}

• Modify the parser class to work with S3 files. Since the parser's source code uses the FileStream class, we recommend downloading the JSON file before streaming it (see comment on line 63).
• Register the created parser in the Startup.cs file:

services.AddTransient<IParser, JSONURLParser>();

We recommend reading our documentation article on implementing a custom data parser for more details: https://www.flexmonster.com/doc/implementing-custom-parser/

Please let us know if our answer helped you.

Best Regards,
Maksym

Hi, Varun!

Thank you for your reply.

The choice between CSV and JSON for indexing large datasets with the Data Server depends on the data structure and how it is processed. Performance and memory usage can vary during the parsing process based on data download speed, formatting complexity, data distribution, and custom parser implementation. However, once the index is created, both formats rely on the same in-memory structures, so there is no difference when the index is created and used.

If parsing performance is a priority, we recommend running benchmarks on your data to determine which format works best for your case.

Please let us know if more questions arise.

Best Regards,
Maksym