asked on November 5, 2020
Excel is able to parse CSVs with newlines properly.. but when I loaded it into flexmonster, it loads, but has calculations incorrect for sums etc.
So it looks like it is silently loading the wrong thing.
Note that if there are sums over data, they do not appear to calculate correctly when parsed from a CSV with new lines.
Is it possible to have
a) flexmonster load newlines correctly, and/or
b) at least issue some warning when loading a file that breaks it?
Thanks,
Mike
Example of CSV:
A0, B0, "This is the first line of the CSV", D0
A, B, "This Comment has
a new line", D
A1, B1, "Comment without new line", D1
A2, B2, "", D2
⋅ November 6, 2020
Hello, Mike,
Thank you for reaching out to us.
Our team recommends setting the ignoreQuotedLineBreaks property of the Data Source Object to false. It indicates whether the line breaks in quotes will be ignored (true) in CSV files or not (false). When set to true, CSV parsing is faster. Set it to false only if your data source has valuable for you line breaks in quotes. Please note that this might slow down CSV parsing a little bit.
Please let us know if it works for you.
Feel free to reach out in case any further questions arise.
Best regards,
Illia
⋅ November 6, 2020
thank you.
It still seems to be broken even with the option set (which should default to true anyway) I will email your support team at ********************* separately with a link that shows the reproduction.
thanks,
Mike
⋅ November 9, 2020
Hello,
Thank you for your feedback.
We have noticed some inaccuracies in our previous response. The correct approach for parsing CSV data with the line breaks is to set the ignoreQuotedLineBreaks property to false.
Our team also wants to confirm that you are right about the fact that this property is set to true by default.
Please let us know if it helps. Feel free to contact us in case any other questions arise.
Best regards,
Illia
⋅ November 9, 2020
Thanks.
It may cause problems if FM silently accepts CSV without letting the user know that it is parsing the file in a way that will cause wrong aggregation / filtering. At the least, it may cause users to spend time trying to isolate the issue.
A) Perhaps some alert (e.g. in console output), would be preferable to silently letting filtering / aggregation issue the wrong result.
B) perhaps consider an option to set ignoreQuotedLineBreaks in the toolbar when interactively loading CSVs.
C) The naming of ignoreQuotedLineBreaks seems confusing. If we have two files, one with line breaks and the other without, with setting ignoreQuotedLineBreaks = true will treat the files differently
but
ignoreQuotedLineBreaks = false, it will treat the files the same?
That seems to be the opposite of what is intuitive.... not sure what can be done to make it more intuitive but allow for backwards compatibility with previous versions.. maybe making a new name?
Thanks
M
⋅ November 10, 2020
Hello, Mike,
Thank you for sharing your suggestions with us.
Our team wants to provide some additional explanations about the points you have mentioned.
Perhaps some alert (e.g. in console output), would be preferable to silently letting filtering/aggregation issue the wrong result.
We want to explain that tracking line breaks in members requires additional steps and negatively affects data loading performance. Therefore, Flexmonster ignores line breaks by default. It means it is not possible to track if the particular data set contains line breaks without these additional operations. This is why Flexmonster does not notify about possibly incorrect results.
Perhaps consider an option to set ignoreQuotedLineBreaks in the toolbar when interactively loading CSVs.
We want to inform you that it is possible to overwrite dedicated tabs of the Toolbar.
Please refer to our guide dedicated to the Toolbar customization for more details.
For example, it is possible to overwrite handlers for "To local CSV" and "To remote CSV" elements of the "Connect" tab so that the ignoreQuotedLineBreaks is always set to false. It allows ensuring all CSV data sets opened interactively are correctly parsed.
The naming of ignoreQuotedLineBreaks seems confusing. If we have two files, one with line breaks and the other without, with setting ignoreQuotedLineBreaks = true will treat the files differently but ignoreQuotedLineBreaks = false, it will treat the files the same?
If the ignoreQuotedLineBreaks property is set to true, Flexmonster ignores line breaks. It means the file with line breaks may be parsed incorrectly.
Otherwise, Flexmonster considers line breaks. As a result, all files are parsed in the same way.
We recommend setting the ignoreQuotedLineBreaks property to false in all your reports to ensure no side effects appear when loading CSV data sets.
We hope it helps.
Please contact us if other questions arise.
Best regards,
Illia
