FileResources CleanUp Task

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

···

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi,

To be 100% clear:

If a standard set of documents are to be uploaded - as is the case here - we simply need to create one text data element for each type of document to store the UID returned with the response field? There is no need for any other data values to reference the stored file?

If uploaded documents are NOT standard (e.g. if you asked users to upload let us say a number of photos), you will still need to create x data elements to store the UID references to whatever is uploaded?

What if you want multiple data value records to reference the same document - e.g. one with the doc UID, another with the date of the upload, and a third a boolean data value indicating how many times the document has been accessed. In that case, what is the exact construct used to link the document to the other fields? (this is just for my own understanding of the design).

Regards

Calle

···

On 25 July 2016 at 11:53, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike

···

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

This should be fine. Are you experiencing that it doesn’t work as expected? (I didn’t consider the tracker use case when I wrote ‘dataset’. My bad…)

Calle: your assumptions are correct. DV to FR is one-to-one. You would need a DE per specific file for your event/dataset, or alternately package multiple files in some archive format and store that instead (you could handle this on the client side).

What if you want multiple data value records to reference the same document […]

A FR data value is just a data value with a weak reference to a file. Grouping data elements is no different than for straight numeric or text values. That is, there is no ‘link’ between the fields, but you could use a construct such as a data set (aggr) or programstageinstance/event (tracker) to group DEs.

···

On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi Halvdan,

We want the user to upload the file and be able to retrieve it but the files are deleted(purged) at 2AM everyday.

  • INFO 2016-07-25 02:00:00,072 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: PERSONAL_CONSIDERATION_SUBMISSION. Samuel Mur 7910025121081.pdf, uid: ko0TqodGcPh (AuditLogUtil.java [taskScheduler-4])

  • INFO 2016-07-25 02:00:00,082 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Craig Anders Personal Consideration.pdf, uid: ShiqhAsXc78 (AuditLogUtil.java [taskScheduler-4])

  • INFO 2016-07-25 02:00:00,096 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Motivation for Comm - Copy.pdf, uid: L1fLHOSqwkM (AuditLogUtil.java [taskScheduler-4])

  • INFO 2016-07-25 02:00:00,102 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: SATA - TOUCH WORLD CUP 2015 - SA MIXED TEAM CONFIRMATION.pdf, uid: EWQXKQsCunv (AuditLogUtil.java [taskScheduler-4])

  • INFO 2016-07-25 02:00:00,108 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: N_Isaacs_WPTA_July2016.pdf, uid: AmoMiCsoSvD (AuditLogUtil.java [taskScheduler-4])

How do we disable this task from running?

Regards,

Mike

···

On Mon, Jul 25, 2016 at 1:27 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This should be fine. Are you experiencing that it doesn’t work as expected? (I didn’t consider the tracker use case when I wrote ‘dataset’. My bad…)

Calle: your assumptions are correct. DV to FR is one-to-one. You would need a DE per specific file for your event/dataset, or alternately package multiple files in some archive format and store that instead (you could handle this on the client side).

What if you want multiple data value records to reference the same document […]

A FR data value is just a data value with a weak reference to a file. Grouping data elements is no different than for straight numeric or text values. That is, there is no ‘link’ between the fields, but you could use a construct such as a data set (aggr) or programstageinstance/event (tracker) to group DEs.

On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Ok.

That task should detect unreferenced fileResource objects and delete them. If they are referenced, however (i.e. there is a data value referencing their UID) they should not be deleted. If the uid in your example is in fact stored in a data value with a FR data element there is a bug, if not it works as intended (to prevent orphaned files).

Could you check if the ‘assigned’ field of the FR is true after creating a new event with a FR data elements? If not, there is a problem in the server side event creation logic.

There is no way to disable the task (also, you really shouldn’t).

···

On Mon, Jul 25, 2016 at 1:37 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

We want the user to upload the file and be able to retrieve it but the files are deleted(purged) at 2AM everyday.

  • INFO 2016-07-25 02:00:00,072 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: PERSONAL_CONSIDERATION_SUBMISSION. Samuel Mur 7910025121081.pdf, uid: ko0TqodGcPh (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,082 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Craig Anders Personal Consideration.pdf, uid: ShiqhAsXc78 (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,096 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Motivation for Comm - Copy.pdf, uid: L1fLHOSqwkM (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,102 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: SATA - TOUCH WORLD CUP 2015 - SA MIXED TEAM CONFIRMATION.pdf, uid: EWQXKQsCunv (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,108 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: N_Isaacs_WPTA_July2016.pdf, uid: AmoMiCsoSvD (AuditLogUtil.java [taskScheduler-4])

How do we disable this task from running?

Regards,

Mike

On Mon, Jul 25, 2016 at 1:27 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This should be fine. Are you experiencing that it doesn’t work as expected? (I didn’t consider the tracker use case when I wrote ‘dataset’. My bad…)

Calle: your assumptions are correct. DV to FR is one-to-one. You would need a DE per specific file for your event/dataset, or alternately package multiple files in some archive format and store that instead (you could handle this on the client side).

What if you want multiple data value records to reference the same document […]

A FR data value is just a data value with a weak reference to a file. Grouping data elements is no different than for straight numeric or text values. That is, there is no ‘link’ between the fields, but you could use a construct such as a data set (aggr) or programstageinstance/event (tracker) to group DEs.

On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

To update everyone: this is definitely a bug with this stuff when using FileResource with Tracker (2.23 -> ). I’m currently working on a fix, but it will take some time.

It hopefully won’t take long, but as it’s a pretty critical issue to have in production you will need to work around it in the mean time. The least intrusive way is to run a cronjob at just before the cleanup task triggers (2:00 server time) and update all fileresource rows in the DB with isassigned=TRUE. Alternately a patch could be applied (comments out a single line to disable the deletion/cleanup task). Let me know if you need any more details.

···

On Mon, Jul 25, 2016 at 2:06 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Ok.

That task should detect unreferenced fileResource objects and delete them. If they are referenced, however (i.e. there is a data value referencing their UID) they should not be deleted. If the uid in your example is in fact stored in a data value with a FR data element there is a bug, if not it works as intended (to prevent orphaned files).

Could you check if the ‘assigned’ field of the FR is true after creating a new event with a FR data elements? If not, there is a problem in the server side event creation logic.

There is no way to disable the task (also, you really shouldn’t).

On Mon, Jul 25, 2016 at 1:37 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

We want the user to upload the file and be able to retrieve it but the files are deleted(purged) at 2AM everyday.

  • INFO 2016-07-25 02:00:00,072 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: PERSONAL_CONSIDERATION_SUBMISSION. Samuel Mur 7910025121081.pdf, uid: ko0TqodGcPh (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,082 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Craig Anders Personal Consideration.pdf, uid: ShiqhAsXc78 (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,096 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Motivation for Comm - Copy.pdf, uid: L1fLHOSqwkM (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,102 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: SATA - TOUCH WORLD CUP 2015 - SA MIXED TEAM CONFIRMATION.pdf, uid: EWQXKQsCunv (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,108 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: N_Isaacs_WPTA_July2016.pdf, uid: AmoMiCsoSvD (AuditLogUtil.java [taskScheduler-4])

How do we disable this task from running?

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 1:27 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This should be fine. Are you experiencing that it doesn’t work as expected? (I didn’t consider the tracker use case when I wrote ‘dataset’. My bad…)

Calle: your assumptions are correct. DV to FR is one-to-one. You would need a DE per specific file for your event/dataset, or alternately package multiple files in some archive format and store that instead (you could handle this on the client side).

What if you want multiple data value records to reference the same document […]

A FR data value is just a data value with a weak reference to a file. Grouping data elements is no different than for straight numeric or text values. That is, there is no ‘link’ between the fields, but you could use a construct such as a data set (aggr) or programstageinstance/event (tracker) to group DEs.

On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

This should now be fixed in latest 2.25, 2.24 and 2.23 (currently building on CI, war-file available within the hour).

Anyone using Tracker with File data values will need to upgrade. Please let me know if you have any more issues.

···

On Mon, Jul 25, 2016 at 5:50 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

To update everyone: this is definitely a bug with this stuff when using FileResource with Tracker (2.23 -> ). I’m currently working on a fix, but it will take some time.

It hopefully won’t take long, but as it’s a pretty critical issue to have in production you will need to work around it in the mean time. The least intrusive way is to run a cronjob at just before the cleanup task triggers (2:00 server time) and update all fileresource rows in the DB with isassigned=TRUE. Alternately a patch could be applied (comments out a single line to disable the deletion/cleanup task). Let me know if you need any more details.

On Mon, Jul 25, 2016 at 2:06 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Ok.

That task should detect unreferenced fileResource objects and delete them. If they are referenced, however (i.e. there is a data value referencing their UID) they should not be deleted. If the uid in your example is in fact stored in a data value with a FR data element there is a bug, if not it works as intended (to prevent orphaned files).

Could you check if the ‘assigned’ field of the FR is true after creating a new event with a FR data elements? If not, there is a problem in the server side event creation logic.

There is no way to disable the task (also, you really shouldn’t).


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 1:37 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

We want the user to upload the file and be able to retrieve it but the files are deleted(purged) at 2AM everyday.

  • INFO 2016-07-25 02:00:00,072 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: PERSONAL_CONSIDERATION_SUBMISSION. Samuel Mur 7910025121081.pdf, uid: ko0TqodGcPh (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,082 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Craig Anders Personal Consideration.pdf, uid: ShiqhAsXc78 (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,096 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: Motivation for Comm - Copy.pdf, uid: L1fLHOSqwkM (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,102 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: SATA - TOUCH WORLD CUP 2015 - SA MIXED TEAM CONFIRMATION.pdf, uid: EWQXKQsCunv (AuditLogUtil.java [taskScheduler-4])
  • INFO 2016-07-25 02:00:00,108 ‘system-process’ delete org.hisp.dhis.fileresource.FileResource, name: N_Isaacs_WPTA_July2016.pdf, uid: AmoMiCsoSvD (AuditLogUtil.java [taskScheduler-4])

How do we disable this task from running?

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 1:27 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This should be fine. Are you experiencing that it doesn’t work as expected? (I didn’t consider the tracker use case when I wrote ‘dataset’. My bad…)

Calle: your assumptions are correct. DV to FR is one-to-one. You would need a DE per specific file for your event/dataset, or alternately package multiple files in some archive format and store that instead (you could handle this on the client side).

What if you want multiple data value records to reference the same document […]

A FR data value is just a data value with a weak reference to a file. Grouping data elements is no different than for straight numeric or text values. That is, there is no ‘link’ between the fields, but you could use a construct such as a data set (aggr) or programstageinstance/event (tracker) to group DEs.

On Mon, Jul 25, 2016 at 1:02 PM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan,

The following are the steps I’m using:

  1. Save the file on the api/fileResource endpoint, get the UID back, then save the uid as of a dataValue on the event. Example:

POST to api/fileResources, get id: “a11111bbbb” back.
2. PUT the event:

  { "event" : "<some-event-uid>", "dataValues" : [ { "value" : "a11111bbbb", "dataElement" : "<some-fileResource-dataelement-uid>" } ] }

Are these steps correct?

Are the datasets required in these case? If yes, how should I go about doing this in the tracker module.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Jul 25, 2016 at 11:53 AM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

Hi Mike,

As is explained in the docs, the FR must be referenced by a data value. If the FR is referenced it will not be deleted.

The FR object is not usable by itself, so the upload to the api/*/fileResources is step 1 of 2 in the process of uploading a file (it needs to be tied to a DV afterwards).

In practice this means that in order to use FR you need to create data elements (and datasets) to contain your files.

Hope that answers your question. If not, please let me know.

On Mon, Jul 25, 2016 at 7:57 AM, Mike Nelushi mikeevolution@gmail.com wrote:

Hi Halvdan/Morten,

There seem to be a cleanup task which is ran at 2AM every 24H which necessary maintenance which removes orphaned files (FileResources) which have been posted to the instance but are marked for deletion as Halvdan explained here. And from File resource constraints explained **here “Any file resources which are in this state and are older than two hours will be marked for deletion and will eventually be purged from the system.” **

Our new application allows users to upload files and would like to prevent this from happening.

How do we stop the system from running this file deletion task or tag the files to not be deleted?

Please advice.

Regards,

Mike


Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org