Read data from S3 bucket

Hello, Varun!

Thank you for writing to us.

Flexmonster supports connecting CSV and JSON files from cloud storage, such as AWS S3.
Since S3 files often require authentication, we would like to mention that Flexmonster allows setting request headers with credentials using the requestHeders data source object property.
For reference, you can check this example, which loads a CSV file from our CDN: https://jsfiddle.net/flexmonster/u5Lp4k98/

Please let us know if our answer helped you.

Best Regards,
Maksym

Hello, Varun!

Thank you for providing additional context.

Flexmonster Data Server does not support connecting to S3 files. However, there are several potential solutions for working with CSV files on S3:

1. You can implement a server-side script that downloads a CSV file to the server and then proceeds with the default logic for creating a CSV data index.
2. Alternatively, you can create a custom parser that would be responsible for loading and processing the data without the need for an additional step of downloading its copy to the server. You can find a guide for implementing a custom parser in our documentation: https://www.flexmonster.com/doc/implementing-custom-parser/

Please let us know if one of these approaches would work for you. Also, we would like to know if you are asking about working with CSV files on S3 because it would just be convenient to have, or because it is critically important for your task?

Looking forward to hearing from you.

Best Regards,
Maksym

Hello, Varun!

Thank you for your reply.
We are working on an example that demonstrates dynamically creating a custom index by loading a CSV file from a URL. We will get back to you once it is ready.

Best Regards,
Maksym

Hello, Varun!

We have modified the sample .NET project with Flexmonster DLL to include the custom parser, which reads remote CSV files from a URL. The project is available on our GitHub: https://github.com/flexmonster/flexmonster-data-server-dll/tree/csv-url-parser-runtime-index-creation

You can find a quick breakdown of what was added or changed below:

1. Added CustomParser.cs - CSVURLParser parser implementation, based on the existing CSVParser. Its type is set to "csv_url" to make it distinguishable from other index types, and the Parse() method has been updated to fetch CSV content over HTTPS using HttpClient(see line 72). The implementation can be further adapted for S3 or other sources.
2. Added CsvUrlIndexOptins class - this class extends the existing CsvIndexOptions. Our implementation only overrides the type; however, you can add further changes to pass more settings, such as AWS credentials.
3. Integrated runtime index creation into the IndexCreateService.cs - the config for the new index is registered with the CsvUrlIndexOptions.

Please let us know if the provided sample helped you.

Best Regards,
Maksym

Hello, Varun!

Thank you for your question.

These can be implemented within the dedicated service for managing indexes, or by adding methods to the existing IndexCreateService:

using Flexmonster.DataServer.Core.DataStorages;
using Flexmonster.DataServer.Core;
using System.Threading.Tasks;
using Microsoft.Extensions.Options;
using Flexmonster.DataServer.Core.Parsers;
using System.Collections.Generic;


namespace DemoDataServerCore.Controllers
{
    public class IndexManagementService
    {
        private readonly IDataStorage _dataStorage;
        private readonly DatasourceOptions _datasourceOptions;

        public IndexManagementService(IDataStorage dataStorage, IOptionsMonitor<DatasourceOptions> datasourceOptions)
        {
            _dataStorage = dataStorage;
            _datasourceOptions = datasourceOptions.CurrentValue;
        }

        public Dictionary<string, IndexOptions> GetAllIndexes()
        {
           return _datasourceOptions.Indexes;
        }

        public void DeleteIndex(string indexName)
        {
            if (_datasourceOptions.Indexes.ContainsKey(indexName))
            {
                _datasourceOptions.Indexes.Remove(indexName);
                _dataStorage.Remove(indexName);
            }
        }
    }

}

Note that the service must be registered within the Startup.cs:

services.AddScoped<IndexManagementService>();

These service methods provide a clean abstraction, and you can implement and configure controller endpoints using them.

Please let us know if our answer helped you.

Best Regards,
Maskym