FileResourceCleanUpJob running and loading all datavalueaudit entries

Stephan_Mestach · 3 April 2019 07:16

In our system there’s no data element with FileResource type but it seems that FileResourceCleanUpJob is running and is loading all the datavalueaudit (more than 5.000.000 of them)

The server is often dumping or running low in free memory.

I found a long running query without where clause

select this_.datavalueauditid as datavalu1_134_0_, this_.dataelementid as dataelem2_134_0_, this_.periodid as periodid3_134_0_, this_.organisationunitid as organisa4_134_0_, this_.categoryoptioncomboid as category5_134_0_, this_.attributeoptioncomboid as attribut6_134_0_, this_.value as value7_134_0_, this_.created as created8_134_0_, this_.modifiedby as modified9_134_0_, this_.audittype as auditty10_134_0_ from datavalueaudit this_ order by this_.created desc

I found in the code a condition on the FileResourceRetentionStrategy in the job

github.com

dhis2/dhis2-core/blob/92bb7f20a9364f8be27d05bd35d5fea85402eac7/dhis-2/dhis-services/dhis-service-core/src/main/java/org/hisp/dhis/fileresource/FileResourceCleanUpJob.java#L107


      
          List<Pair<String, String>> deletedAuditFiles = new ArrayList<>();
          
          
fileResourceService.getOrphanedFileResources().stream()
              .filter( fr -> !isFileStored( fr ) )
              .filter( fr -> safeDelete( fr ) )
              .forEach( fr -> deletedOrphans.add( ImmutablePair.of( fr.getName(), fr.getUid() ) ) );
          
          

          
if ( retentionStrategy != FileResourceRetentionStrategy.FOREVER )
          {
              deletedAuditFiles = getExpiredFileResources( retentionStrategy ).stream()
                  .filter( pair -> safeDelete( fileResourceService.getFileResource( pair.getRight() ) ) )
                  .collect( Collectors.toList() );
          }
          
          
if ( !deletedOrphans.isEmpty() )
          {
              log.warn( String.format( "Deleted %d orphaned FileResources: %s", deletedOrphans.size(), prettyPrint( deletedOrphans ) ) );
          }
          
          
if ( !deletedAuditFiles.isEmpty() )

Note that the table systemsetting doesn’t contain ‘keyFileResourceRetentionStrategy’ record

I suspect the retention strategy is null and so != FOREVER and the query on datavalueaudit gets triggered with an empty list of data elements, ending up to the query without where clause.

Is there a way configure this to FOREVER in the UI ?

Thanks

Stéphan

Stephan_Mestach · 3 April 2019 07:43

As a workaround now, I’ve created an unused datalement of File type

Stian · 9 April 2019 15:16

Hi @Stephan_Mestach,

Thanks for making us aware of this issue! I’ve submitted a simple fix that skips the query if no data elements exists. It should be available in new versions of the war file soon after it has been merged.

Stephan_Mestach · 16 April 2019 06:33

Where are we supposed to configure the “keyFileResourceRetentionStrategy” is there a UI or per data element ?

Stian · 10 July 2019 14:32

Hi @Stephan_Mestach,

sorry for the late reply. This is a system setting, and can be set either through the api as other system settings. Currently we seem to have missed adding this setting to the UI, so I created an issue to resolve that ([DHIS2-7248] - Jira)