asked on January 11, 2016
Hi there,
I'd like to evaluate the CSVCompressor to see if it performns well with our data but I can't get it working. I keep getting "Datasource is empty. Please check the CSV file." popup on the client side though I'm pretty sure the data is written to the output stream. Please see the two code snippets below. They are almost identical, the first one works as intended the seconds one always gives the error popup.
Solution without compression (works like a charm):
context.HttpContext.Response.ContentType = "application/csv";
context.HttpContext.Response.Buffer = false;
context.HttpContext.Response.BufferOutput = false;
using (var resultStream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(resultStream))
{
//Here the data is written to the streamWriter
streamWriter.Flush();
resultStream.Position = 0;
byte[] notcompressedData = resultStream.ToArray();
httpContext.Response.OutputStream.Write(notcompressedData, 0, notcompressedData.Count());
httpContext.Response.OutputStream.Flush();
}
}
Solution with compression (doesn't work)
// Tried both "application/csv" and "application/octet-stream" here
context.HttpContext.Response.ContentType = "application/octet-stream";
context.HttpContext.Response.Buffer = false;
context.HttpContext.Response.BufferOutput = false;
using (var resultStream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(resultStream))
{
//Here the data is written to the streamWriter
streamWriter.Flush();
resultStream.Position = 0;
using (var streamReader = new StreamReader(resultStream))
{
byte[] compressedData = CSVCompressor.Compress(streamReader.ReadToEnd(), ',');
httpContext.Response.OutputStream.Write(compressedData, 0, compressedData.Count());
httpContext.Response.OutputStream.Flush();
}
}
}
⋅ January 11, 2016
Hi Maxim,
Could you please check dataSourceType property you provide to the component when you connect to URL that returns compressed CSV? It should be set explicitly to "ocsv".
For example, if you provide URL to your OCSV data in report XML, dataSourceType should be the following:
<config>
<dataSource type="ocsv">
<filename>URL_to_your_OCSV_data</filename>
</dataSource>
...
</config>
Here is an example when you use connectTo() API call:
flexmonster.connectTo({
dataSourceType: 'ocsv',
filename: 'URL_to_your_OCSV_data'
});
Please let me know if the problem was in dataSourceType.
Kind regards,
Iryna
⋅ January 12, 2016
Hi Iryna,
I changed the dataSource type to ocsv in the config file but it didn't help. Now the report loading gets stuck saying "Loading data... 0K of 0K". Please see the screenshot here
I'm sure the data is written to the stream. You can find the data which is compressed here in case if it helps. I got it by setting a break point at this line from the code snippet above.
byte[] compressedData = CSVCompressor.Compress(streamReader.ReadToEnd(), ',');
⋅ January 13, 2016
Hi Maxim,
We've checked your input CSV file and it seems that it works fine with CSVCompressor.Compress() function.
Could you please navigate in a browser to the URL with the script that produces OCSV file and check the output? Is it empty or not?
Regards,
Iryna
⋅ January 13, 2016
Hi Iryna,
I fetched request and response info with fiddler both for config file and csv file:
Config file request (raw)
Config file response (raw)
Config file (unencoded)
CSV file request (raw)
CSV file response (raw)
Please let me know if you need more info.
⋅ January 15, 2016
Hi Maxim,
We reproduced the issue. It is reproducible in Flash version of the component. We recommend you trying HTML5 version, which works correctly.
The issue in Flash version will be fixed in the minor release 2.205 (ETA - February 1).
Please let me know if the switch to HTML5 version works for you.
Regards,
Iryna
⋅ January 15, 2016
Hi Iryna,
Thank you for the info. I think we would better wait for the fix. Could you please reply here when it's released?
⋅ January 15, 2016
Ok. I will inform you here when the fix is released.
⋅ February 1, 2016
Hi Maxim,
The fix of the Flash version is included into the minor release 2.205. Please feel free to download it from the website.
Please let me know whether the updated version works for you.
Kind regards,
Iryna
⋅ February 2, 2016
Hi Iryna,
Thanks for the update. The grid doesn't get stuck fro me anymore but the compression/decompression logic doesn't seem to work correctly. Here's the test data I used. If you check the second column you will see values for "Hour 01", "Hour 02", "Hour 03", "Hour 04", "Hour 05", etc. However, most of the data is missing when displayed by the grid. Please see the screenshot. Is it also a bug? Please let me know if you need more info.
Ionic.Zlib.CF.dll version - 1.9.1.8
CSVCompressor.dll version - 1.0.0.0
⋅ February 3, 2016
Hi Maxim,
Glad to hear the issue with getting stuck is fixed.
Regarding the compression/decompression logic, it is a known issue with "n+" prefix.
The fix will be available in the next minor release (Feb 15).
As a workaround, you can use "+" prefix instead (bu the sorting will not be numeric):
+Year,+Hour,m+Month,+Day,Volume ...
Hope this helps.
Kind regards,
Iryna
⋅ February 3, 2016
Hi Iryna,
Is there a way to get the list of compression/related known bugs? The thing is that we want to upgrade the flexmonster control and use the compression feature but we don't want to lose any existing functionality (like numeric sorting). So I wonder if there are any other compression related known bugs which are not reproducable when not using the compression feature, just like the one you mentioned above.
⋅ February 4, 2016
Hi Maxim,
There are two known issues in the compression/decompression logic:
They will be fixed in the next minor release 2.206 (Feb 15).
Kind regards,
Iryna
⋅ February 5, 2016
Thanks for the info. Could you please let me know when the fixes are releaed?
⋅ February 5, 2016
You are welcome. Yes, sure. I will let you know.
⋅ February 15, 2016
Hi Maxim,
The updated version of CSV Compressor is available for download within the minor release 2.206. Please feel free to download it from the website. Also, you can find the list of all updates included into 2.206 in the blog post Flexmonster’s Biweekly Report: New Enhancements in 2.206 Minor Release
Please let me know whether the updated Compressor works for you.
Kind regards,
Iryna
⋅ March 2, 2016
Hi Iryna,
I've just tried the latest version and it looks better now but not yet fully fixed. Compare the source data n+Day column and see what is actually displayed in the UI after compression/decompression. It should be March 1 and March 2 according to the data source but for some reason it's March 2 and March 3 in UI.
⋅ March 2, 2016
Hi Maxim,
I've just tried to reproduce the issue you described by converting your CSV data with March 1 and March 2 to OCSV using .NET dll and opening OCSV in Flash version of the component - it works as expected, I see March 1 and March 2 in this case as well.
Could you please tell me what is your version of the pivot table component - click on the grid and press Ctrl+Alt+i? Is it Version 2.207 (build 02/29/2016 - 15:25)?
The latest package with the compressor contains two dll files - 2.206 and 2.207 versions. Please let me know which one you use.
Kind regards,
Iryna
⋅ March 2, 2016
The pivot table version matches the you specified above, see the screenshot.
I downloaded the compressor for .NET today. It contains two dll files:
CSVCompressor.dll
Ionic.Zlib.CF.dll
How do I check their version? When I right click a file, hit Properties and go to the Details page I see the following version
CSVCompressor.dll - 1.0.0.0
Ionic.Zlib.CF.dll - 1.9.1.8
⋅ March 2, 2016
Maxim,
Thank you for the information you provided. The blog post contained the link to the previous version of the compressor. Sorry for the confusion, we already corrected this.
Could you please download the package using the link 'Configured for Relational databases: MySQL, Oracle, etc.' in the email you received after downloading the component - http://screencast.com/t/DH0okU2vxmlC? It contains the newest version of the compressor. Each time a minor release is done, we include its newest version there. Thus, it can be downloaded together with the latest version of the component as well.
Please let me know if the issues are resolved with the compressor from the latest package.
Kind regards,
Iryna
⋅ March 2, 2016
Iryna,
Now when I reference either Flexmonster.Compressor.2.206.dll or Flexmonster.Compressor.2.207.dll in my project and run it I get teh following error:
Could not load file or assembly 'Flexmonster.Compressor.2.207' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Any ideas what I'm doing wrong?
Please note the Target Framework of my project is .NET Framework 4.5.1 and it seems you use .NET Framework 4.5.2 to build the compressor dll. I think it might be the problem.
With older version I just referenced it in my project via Add Reference and it worked fine except for the bug.
One more important note, there's a project in the download package which seems to be a demo for compressor. It's located under "server\.net\FlexmonsterCompressor" folder. I tried to run it as well and got the same error.
⋅ March 3, 2016
Hi Maxim,
Thank you for the feedback.
Please download and try DLL that we've built with lower .NET Framework version - https://s3.amazonaws.com/flexmonster/downloads/FLEXMONSTER-COMPRESSOR-2-207.zip
Also, we will include this fix in all of the future minor releases.
Regards,
Ian
⋅ March 3, 2016
⋅ March 3, 2016
We've compressed your sample file and it seems identical to the raw CSV file - http://screencast.com/t/zkVNEDkAhU.
Does this file work the same for you?
Ian
⋅ March 3, 2016
Which version of the control did you use for the test? Please note we use Flash version. If you read the thread from the beginning you already had a similar bug reproducable in flash version only.
Also, how can I get the ocsv file locally? We just write to the stream the array returned by CSVCompressor.Compress, please see the code samples above.
⋅ March 4, 2016
Hi Maxim,
Flash version of the control was used for the test provided by Ian in the previous message.
Have you tried to open OCSV file in your Flash version of the control? Does it work fine?
If you are interested in getting the OCSV file locally, you can do this the following way using System.IO.File.WriteAllBytes:
string fileName = @"d:/data.csv";
string[] data = System.IO.File.ReadAllLines(fileName);
byte[] compressed = Flexmonster.CSVCompressor.Compress(data);
System.IO.File.WriteAllBytes(fileName.Replace(".csv", ".ocsv"), compressed);
Please let us know if the issue is still reproducible on your side.
Kind regards,
Iryna
⋅ March 7, 2016
No, it doesn't. Here's the raw response fetched with Fiddler and the OCSV file I saved locally. I compared it to the one Ian provided and no differences were found. So it looks like something goes wrong when the control processes the stream. It seems as if it just got the first chunk of data and decompressed it without processing the rest. I wonder if you could debug exactly the same scenario, that is
1. Create mvc application.
2. Add flexmonster grid.
3. Use the same config file.
4. When the control hits the data url, stream the data to the output. Just use the code snippet above.
⋅ March 7, 2016
Just ran a simple test with much smaller dataset and it works fine. I can see both February and March data.
+Year,+Hour,m+Month,n+Day,AudienceVolume
2016,Hour 00,February,1,239798
2016,Hour 03,February,1,221564
2016,Hour 10,February,1,51997
2016,Hour 12,February,1,107123
2016,Hour 17,February,1,242805
2016,Hour 18,February,1,250511
2016,Hour 21,February,1,262876
2016,Hour 22,February,1,265017
2016,Hour 01,February,2,279784
2016,Hour 03,February,2,261356
2016,Hour 10,February,2,63433
2016,Hour 17,February,2,255312
2016,Hour 18,February,2,260558
2016,Hour 20,February,2,260068
2016,Hour 21,February,2,257819
2016,Hour 22,February,2,258427
2016,Hour 01,February,3,281013
2016,Hour 03,February,3,253854
2016,Hour 10,February,3,58109
2016,Hour 17,February,3,247559
2016,Hour 01,March,1,271034
2016,Hour 02,March,1,283793
2016,Hour 20,March,1,249112
2016,Hour 02,March,2,247045
2016,Hour 02,March,3,250230
2016,Hour 06,March,3,81268
2016,Hour 09,March,3,40509
I'm almost sure now that something goes wrong when the grid processes the stream.
⋅ March 7, 2016
Hello Maxim,
Thank you for details, but we still unable to recreate the issue on our side.
We've prepared sample MVC project, could you please point us to the difference with your case?
Please note that we've used Action approach.
Thanks,
Ian
⋅ March 8, 2016
Hi Ian,
Please check the updated project. Your solution doesn't actually stream the data. We have to use streaming due to large datasets as we don't want to upload them fully into memory. I know that with compression we will have to uplaod the whole dataset into memory but I want to make as few changes to the code as possible. Besides we might want to revert quickly if compression feature doesn't perfom ok for us.
You can use the code in the comments to check that streaming solution works fine if we don't compress the data.
⋅ March 9, 2016
Hi Maxim,
Thank you for the updated MVC project. It helped us to reproduce and fix the issue. Please download your MVC project with the updated SWF file:
https://s3.amazonaws.com/flexmonster/adsupply/FLEXMONSTER-2016-ADSUPPLY-MAR09.ZIP
Please let us know if it works fine now.
Kind regards,
Iryna
⋅ March 18, 2016
Hello Maxim,
Could you please let me know whether the fix helped you to resolve the issue?
Thanks,
Ian
⋅ July 11, 2016
Hi Ian,
It worked, thank you