OCSV Hierarchy over tree levels produces incorrect data

We attached an exemple of how data should be shown and the OCSV generated automatically by the compressor
Instead of generate a hierarchy, it is repeating the same value in all hierarchy entries 

___ocsv2___2.7.1/2.213
+empresa,+filial,d+data,+descricao,+departamento,+departamento:categoria,+departamento:categoria:subcategoria,+departamento:categoria:subcategoria:segmento,+departamento:categoria:subcategoria:segmento:subsegmento,+departamento:categoria:subcategoria:segmento:subsegmento:produt,-valor_total_custo,+fornecedor,+comprador,-quantidade_vendida,-preco_custo,-preco_vendido,-preco_margem_zero,-valor_total_vendido,-valor_lucro,-margem_lucro
nova empresa,LJ 10,1545357600,SANDALIA IPANEMA CLASSICA MASC,BAZAR,UTILIDADE DO LAR,SANDALIA,SANDALIA FEMININA,SANDALIA FEMININA,SANDALIA IPANEMA CLASSICA MASC,1379.4,,,2,689.7,15.98,948.05,31.96,10,10
^0,LJ 1,1545357600,^0,^0,^0,^0,^0,^0,^0,689.68,^0,^0,1,689.68,15.98,948.02,15.98,10,10
^0,LJ 2,1545357600,^0,^0,^0,^0,^0,^0,^0,1379.36,^0,^0,2,689.68,15.98,948.01,31.96,10,10

And this is a sample of how we should show the data based on an interface sample

I was thinking, there is anyway to determine how hierarchy should work in code from the front end ? I mean, when hierarchies go five levels deep or more, the database truncate the column name, making it unreadable.
 
Instead of receiving the data from back end directly, there is a way to configure this in flexmonster itself ?

Hello, Vera
 
Well, we've still tried to make it work, but no sucess.
 
About the way flexmonster organizes hierarchy, well, no problem about that. We have pretty deep hierarchies and we just want to make it work, because the column name from the database truncates the string. If you have any sugestion, I'm listening 🙂
About the CSV, we did the folowing: 
 

• We downloaded the oCSV data ( about 120mb ) for you to analize
• We downloaded the JSON data ( about 850mb ) for you to compare
• We've put the data into Google Drive in the folowing link: https://drive.google.com/drive/folders/1moP1lLJbDsrBUay3AIN6UAbw4agzb7G0?usp=sharing

 It is important to have both data to compare, because we suspect that the compressor might be making pointers to values as a way avoiding duplicates, but in doing so, it is discarding important data, wich may be present into the JSON.
 
Hint: The dataset is pretty big, so be sure to disable syntax highlighting in your code editor ( it consumes a lot of processor and ram not doing so )
 
 
Although we are testing large datasets, we do know that we may need ElasticSearch to manage all the data in a fashionable way, but we'll implement it later. For now, we're working with smaller datasets ( about 100.000 rows ) to make it work as intended.
Let me know if you have problems downloading the data. I'll be here
For now, thank you for all your efforts 

Hello, Vera
 
Thank you! We'll be using in JSON for now and wait your return. As we have so much data, we will find a way to make it work as intended
 
Best regards

Hi Vera!

That's great news. Which version is it ? In the one published 4 days ago ? 2.7.4 ?

Hi, Vera, we've tested the 2.7.4 version and the data is still coming wrong.

Do you know if something in the syntax changed to make it work ? We didn't find anything about the subject. 

We are sending into the results of our tests. To further help you understand the problem and give a better picture:
"Bebida" is portuguese for "drink".
"bebida alcoólica" is portuguese for "alcoholic drink".
So, "bebida alcoólica" should be inside "Bebidas". This is how we put this inside the query, but it is not how it showed us.

You know if we're doing something wrong ? If so, can you help us to fix the problem ?
 

Hello, Tanya, good morning
I'm not sure, but we may be doing something wrong. We've updated flexmonster compressor, but the result is the same, sadly
 
We are sending you the two images, One contains the updated version of flexmonster, the other contains the result we received

we hope it helps 🙂
 
 

Hi, Tanya, thank for your response.
We separated the results from our backend and created two files. One with the JSON, using the configuration of data we need to have and another file with the compressed OCSV data we got from the server.
 
We are sending a print of how the results behave when using JSON ( and that's how they should behave when using OCSV )

If there's something more I can do, just ask.
The image files both represent how data should behave inside our trees. They're for reference. The JSON contains the data needed to make it work, the OCSV contains the compressed data we're receiving
 
link with the files and the json -> https://drive.google.com/drive/folders/1c-q54ZPizCDvgcOIOoGuuPMYvc9Jm9SA?usp=sharing
 
Regards,
Allan
 
 

Dmytro, sorry, I think I didn't understand. 

I've came to open this issue because I can't use JSON for my data. My actual json has several lines, wich makes hard for the browser to parse. So we've switched to the compressor.

We had problems with the data compressor, so we contacted you and we've sent examples of what we need.
We've waited to the compressor updates. So far, so good

The update was released, we have tested and we've detected that the problem persists. We contacted you again and told what happened and tried the suggestion you gave. 

The last time we chatted Tanya asked us to send examples of how data should behave and the actual data with it. We have sent it. 

I saw that we forgot to send the header with the correct configuration. But I'm confused, what now ?

You sent me a file with the configuration that is correct. We've used the same configuration and it worked on our end. But we don't want to use JSON. Our data in JSON comes in almost 10mb and the browser can't parse it. 

There's something I need to do now, another step I can take here ? 

Hello, Dmytro
Thank you for your response!
Yes! It was a misunderstanding. We implemented as you sugested and here are our results
 
When using the flexmonster compressor with only one "departamento" entry, it goes well. It is fine tuned and work as intended. 
 
We ran into problems as we added more "departamento"'s into the mix. I'll send you samples of what happened. I think we evolved pretty well and we are in the right direction. We just need to fill the gaps now. We appreciate all your efforts to make it work. If we're doing something wrong on our side, tell us and we'll retry here
 
Now, we'll send what happened to us: 

The first two examples ( bebidas-right and flexmonster-bebidas-right) show that the data, using a single "departamento", is correct. Everything is in their places. 

The other examples ( expected-bebidas and received-bebidas ) show that, when we've put more than one "departamento"'s it confuses the data and spill out "REFR COLA PET" as "departamento" when it should be inside "BEBIDAS"
This happens with any other combinations of data, but not when using those data alone.

We guess that, somewhere along the line, the compressor is confusing the stream and classifying the data incorrectly when the input comes from different parents. 

Is there something we can do here to solve ? Is something that you can solve ? If there's something we can do, just ask 🙂 
 
Also the dataset is the same we've send in last comments. And it still in the google drive 
 

Yes! It seems that we are!
Thank you! 🙂
 
Sure we can send you the data. We have limited that data output to ~6500 entries. They're in json and they have the same pattern we exposed earlier
It's attached to this message 
 
Regards
 

Hi Vera!

Glad to know that! We'll be testing the results and will return soon!

Thank you!