I tried to push a 300+ options category along other smaller categories (2-5 options) (61.4 KB)
, the import works fine BUT the generation of the categoryComboOption initially failed because of lack of heap but since I increase the memory it just “run” (1 cpu @ 100 %) since 4 hours.
Is is a known bug ? is there way to avoid it ?
server info:
|postgres|14|
|Version|2.40.1|
|Build revision|19eb664|
|Java version|11.0.20.1|
|Java vendor|Ubuntu|
|OS name|Linux|
|OS architecture|amd64|
|OS version|5.15.0-88-generic|
|Memory info|Mem Total in JVM: 933 Free in JVM: 585 Max Limit: 3096|
|CPU cores|2|
This likely is an issue with the potential size of the resulting number of category option combinations. For instance, this category combo “ageGroup_claim_detail_status_claim_type_icd_product_sex” has 8 categories. Category option combinations need to be created for the entire crossproduct of all of those dimensions. So, multiply the number of category options and you get a very large number (2.7 million was the back of the envelope calculation). I am not sure this design is going to work. I would consider trying to break this into different dimensions. Age/Sex are logical categories. I am not sure what “PRODUCT” is, but you could use data elements instead of representing it as a category. Payment/Status and Claim status might make more sense as an attribute option combination on the dataset. Our general recommendation is not to exceed three dimensions for category combos, since otherwise the analysis becomes very difficult. You will also likely end up with MANY category option combinations which are never used, which will have a very deleterious effect on performance. Hope this helps.
I will have a look on a different approach using your recommendation, (I came from an event approach but the few millions records took literally month to be pushed )
Do you know if I can send attribute using ADX format ? Should I just add then on the group level using their code as hinted here Data - DHIS2 Documentation (there is no example)