Missing Fields in custom data source API

Hello,

Thank you for your suggestion.

You are right that relying on the findOne() for retrieving fields can lead to incomplete results when documents do not consistently contain all fields, and there is a potential for achieving more representative results. That said, our connector was designed with a more generic use case in mind, assuming that all fields would be available in the data and represented by null values or empty strings when they are not present.

If you would like to adapt this to your scenario, you can modify the /fields controller in the connector directly: https://github.com/flexmonster/pivot-mongo/blob/master/src/controller/mongo.ts#L59
You can either add a custom query for fields or hard-code the fields corresponding to indexes there.

Please let us know if this approach would work for you.

Best Regards,
Maksym

Hello,

Thank you for your reply.

You are absolutely right - our earlier response was incomplete. While /fields request directly queries the schema in the controller, other methods that are used in /select and /members rely on the private _getSchema method, which uses findOne().

As a workaround, you can override the parseShemaFromDocument method in the MongoResponseParser class via prototype. This method is used in the _getSchema; changing it would affect all other methods using it to get fields:

import { MongoResponseParser } from 'flexmonster-mongo-connector/build/parsers/MongoResponseParser';

// Add functions with your implementation of schema query and schema parsing
MongoResponseParser.prototype.parseShemaFromDocument = customParseShemaFromDocument;

In the default workflow, this method receives the document returned from findOne(). In your implementation, you must ignore that single document, run a query with a larger sample size, and then process the results accordingly. Please note that this method returns the internal APISchema object of our library: https://github.com/flexmonster/flexmonster-mongodb-connector/blob/master/src/schema/APISchema.ts

Alternatively, you can install the MongoDB connector source code into your project and modify it: https://github.com/flexmonster/flexmonster-mongodb-connector/tree/master 

This approach will allow you to change the schema retrieval process across all methods that rely on _getSchema.

Best regards,
Maksym

Hello,

Thank you for your quick response.
We are glad you resolved the issue with the sample size of fields on your side.
Please let us know if more questions arise.

Best regards,
Maksym