Bulk Deletion of Data

Support - Assistance technique
1

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

2

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

···

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

3

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

···

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

4

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

···

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

5

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

···

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

6

My interpretation was that they would like a flag on each datavalue to say whether it is still current - which I guess would require an extension of that table. However, we do have something akin to this, namely the audit functionality, which preserves the history of earlier (overwritten) values - perhaps that could be leveraged?

Knut

···

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

7

Just FYI, we have two blueprints related to this in 2.22, [1] is implemented already and [2] will hopefully come (but we are still discussing implementation details for that one)

[1] https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

[2] https://blueprints.launchpad.net/dhis2/+spec/soft-event-delete

···

On Tue, Oct 27, 2015 at 4:01 PM, Knut Staring knutst@gmail.com wrote:

My interpretation was that they would like a flag on each datavalue to say whether it is still current - which I guess would require an extension of that table. However, we do have something akin to this, namely the audit functionality, which preserves the history of earlier (overwritten) values - perhaps that could be leveraged?

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Morten

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

8

Thanks for this Morten. Just to clarify the business need here, the desire is to completely delete a dataset or a program from DHIS 2 which is not possible when associated data value records are in the database.

With blueprint #1

https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

it doesn’t mention the deletion of the data values itself. Would the deletion of events via the web api also cascade delete the underlying data values?

Alternately, how would I delete single tracked entity data values via the web API? I can’t seem to find any API call specifically for these values.

And for datasets, I have tried sending a delete request for a specific data value but I get a 406. Not sure if I am doing something wrong? I’ve pasted my curl and results below for both GET and DELETE.

Thanks,

Lorill

curl “http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC” -X GET -u admin:district -v

  • Trying ::1…

  • Connected to localhost (::1) port 8090 (#0)

  • Server auth using Basic with user ‘admin’

GET /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

Host: localhost:8090

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

< HTTP/1.1 200 OK

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3F51C4A3F10135A648E45BAF42818AB7; Path=/dhis/; HttpOnly

< ETag: “0ab3ca659aa5fc585fa72c6474d6b21df”

< Content-Type: application/json;charset=UTF-8

< Content-Language: en-US

< Content-Length: 10

< Date: Tue, 27 Oct 2015 19:16:29 GMT

<

  • Connection #0 to host localhost left intact

[“396411”]

···

On Tue, Oct 27, 2015 at 1:07 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Just FYI, we have two blueprints related to this in 2.22, [1] is implemented already and [2] will hopefully come (but we are still discussing implementation details for that one)

[1] https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

[2] https://blueprints.launchpad.net/dhis2/+spec/soft-event-delete

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Morten

On Tue, Oct 27, 2015 at 4:01 PM, Knut Staring knutst@gmail.com wrote:

My interpretation was that they would like a flag on each datavalue to say whether it is still current - which I guess would require an extension of that table. However, we do have something akin to this, namely the audit functionality, which preserves the history of earlier (overwritten) values - perhaps that could be leveraged?

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

9

With blueprint #1

https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects
it doesn't mention the deletion of the data values itself. Would the
deletion of events via the web api also cascade delete the underlying data
values?

Yes, it should work that way. What you do is this (for events)
{ "events": [
  { "event": "id1" },
  { "event": "id2" }
] }

Then send with
curl -u user:pass -X POST -d @events.json -H "Content-Type:
application/json" http://server:port/api/events?strategy=DELETE

Alternately, how would I delete single tracked entity data values via the
web API? I can't seem to find any API call specifically for these values.

curl -X DELETE -u user:pass http://server:port/api/events/ID should work

And for datasets, I have tried sending a delete request for a specific
data value but I get a 406. Not sure if I am doing something wrong? I've
pasted my curl and results below for both GET and DELETE.

I'm not sure exactly why we have this requirement, but you need to use -H
"Accept: text/plain" (with curl), I will look into it..

···

--
Morten

Thanks,

Lorill

curl "
http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC&quot;
-X GET -u admin:district -v

* Trying ::1...

* Connected to localhost (::1) port 8090 (#0)

* Server auth using Basic with user 'admin'

> GET /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

> Host: localhost:8090

> Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

> User-Agent: curl/7.43.0

> Accept: */*

>

< HTTP/1.1 200 OK

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3F51C4A3F10135A648E45BAF42818AB7; Path=/dhis/;
HttpOnly

< ETag: "0ab3ca659aa5fc585fa72c6474d6b21df"

< Content-Type: application/json;charset=UTF-8

< Content-Language: en-US

< Content-Length: 10

< Date: Tue, 27 Oct 2015 19:16:29 GMT

<

* Connection #0 to host localhost left intact

["396411"]

-----------

curl "
http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC&quot;
-X DELETE -u admin:district -v

* Trying ::1...

* Connected to localhost (::1) port 8090 (#0)

* Server auth using Basic with user 'admin'

> DELETE /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC
HTTP/1.1

> Host: localhost:8090

> Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

> User-Agent: curl/7.43.0

> Accept: */*

>

< HTTP/1.1 406 Not Acceptable

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=10438B228DF6F9FEF289CA05A3E9CA5F; Path=/dhis/;
HttpOnly

< Content-Type: text/html;charset=utf-8

< Content-Language: en

< Content-Length: 1110

< Date: Tue, 27 Oct 2015 19:16:35 GMT

<

* Connection #0 to host localhost left intact

<!DOCTYPE html><html><head><title>Apache Tomcat/8.0.21 - Error
report</title><style type="text/css">H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}.line {height: 1px;
background-color: #525D76; border: none;}</style> </head><body><h1>HTTP
Status 406 - </h1><div class="line"></div><p><b>type</b> Status
report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The
resource identified by this request is only capable of generating responses
with characteristics not acceptable according to the request "accept"
headers.</u></p><hr class="line"><h3>Apache Tomcat/8.0.21</h3></body></html>

On Tue, Oct 27, 2015 at 1:07 AM, Morten Olav Hansen <mortenoh@gmail.com> > wrote:

Just FYI, we have two blueprints related to this in 2.22, [1] is
implemented already and [2] will hopefully come (but we are still
discussing implementation details for that one)

[1]
https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects
[2] https://blueprints.launchpad.net/dhis2/+spec/soft-event-delete

--
Morten

On Tue, Oct 27, 2015 at 4:01 PM, Knut Staring <knutst@gmail.com> wrote:

My interpretation was that they would like a flag on each datavalue to
say whether it is still current - which I guess would require an extension
of that table. However, we do have something akin to this, namely the audit
functionality, which preserves the history of earlier (overwritten) values
- perhaps that could be leveraged?

Knut

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering < >>> jason.p.pickering@gmail.com> wrote:

Hi Aaron,

There is no "soft delete" function, if that is what you are after, but
it has been discussed.

Although there is no direct way as Lars points out, if you know what
you wish to delete, it is still possible. Namely, if you transmit a ""
value for the value which you wish to delete, it will be deleted. Thus, if
you wish to delete all values for a given dataset/OU/period, you can get
all of the values from the API, then set them all to "", and upload the
data again. I think you can also use the DELETE verb as documented here (
https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html\) to
delete a value.

Not sure if that is the functionality you are after though. Typically,
this would be a DB admin's role though and would be done with SQL, if you
are say, going to delete ALL values for a large part of the hierarchy. The
possibility for a whoopsie-moment could be large here.

Regards,
Jason

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders <agladders@2paths.com> >>>> wrote:

Hi Lars - is there a recommended approach to "deactive" data without
getting into the database itself? If not is it worth setting up a
blueprint for it? We may have a group interested in funding that
development.

Thanks,

Aaron

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland < >>>>> larshelge@gmail.com> wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have
decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees <lcrees@2paths.com> >>>>>> wrote:

Hi,

Just following up on this. Any ideas on the best way to
programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees <lcrees@2paths.com> >>>>>>> wrote:

Hi,

I seem to remember a prior thread on this but I can't seem to find
it. We want to be able to programatically bulk delete data values (both
aggregate and program/program stage data) via the Web API. Is there a way
to achieve this?

We don't want to have to do this on a sql level directly via the
database, nor do we want to have to loop through individual data points in
order to delete them.

Thanks,

Lorill

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar
http://dhis2.org

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

10

Hi Morten,

Thanks for the info regarding events, we’ll give it a try.

As for datasets / data values I tried it with “Accept: text/plain” and it now works, thanks. Please let me know if that header requirement is going to change.

Cheers,

Lorill

···

On Tue, Oct 27, 2015 at 9:30 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

With blueprint #1

https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

it doesn’t mention the deletion of the data values itself. Would the deletion of events via the web api also cascade delete the underlying data values?

Yes, it should work that way. What you do is this (for events)

{ “events”: [

{ “event”: “id1” },

{ “event”: “id2” }

] }

Then send with

curl -u user:pass -X POST -d @events.json -H “Content-Type: application/json” http://server:port/api/events?strategy=DELETE

Alternately, how would I delete single tracked entity data values via the web API? I can’t seem to find any API call specifically for these values.

curl -X DELETE -u user:pass http://server:port/api/events/ID should work

And for datasets, I have tried sending a delete request for a specific data value but I get a 406. Not sure if I am doing something wrong? I’ve pasted my curl and results below for both GET and DELETE.

I’m not sure exactly why we have this requirement, but you need to use -H “Accept: text/plain” (with curl), I will look into it…

–

Morten

Thanks,

Lorill

curl “http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC” -X GET -u admin:district -v

  • Trying ::1…
  • Connected to localhost (::1) port 8090 (#0)
  • Server auth using Basic with user ‘admin’

GET /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

Host: localhost:8090

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

< HTTP/1.1 200 OK

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3F51C4A3F10135A648E45BAF42818AB7; Path=/dhis/; HttpOnly

< ETag: “0ab3ca659aa5fc585fa72c6474d6b21df”

< Content-Type: application/json;charset=UTF-8

< Content-Language: en-US

< Content-Length: 10

< Date: Tue, 27 Oct 2015 19:16:29 GMT

<

  • Connection #0 to host localhost left intact

[“396411”]

curl “http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC” -X DELETE -u admin:district -v

  • Trying ::1…
  • Connected to localhost (::1) port 8090 (#0)
  • Server auth using Basic with user ‘admin’

DELETE /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

Host: localhost:8090

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

< HTTP/1.1 406 Not Acceptable

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=10438B228DF6F9FEF289CA05A3E9CA5F; Path=/dhis/; HttpOnly

< Content-Type: text/html;charset=utf-8

< Content-Language: en

< Content-Length: 1110

< Date: Tue, 27 Oct 2015 19:16:35 GMT

<

  • Connection #0 to host localhost left intact
Apache Tomcat/8.0.21 - Error reportH1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}

HTTP Status 406 -

type Status report

message

description The resource identified by this request is only capable of generating responses with characteristics not acceptable according to the request "accept" headers.

Apache Tomcat/8.0.21

On Tue, Oct 27, 2015 at 1:07 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Just FYI, we have two blueprints related to this in 2.22, [1] is implemented already and [2] will hopefully come (but we are still discussing implementation details for that one)

[1] https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

[2] https://blueprints.launchpad.net/dhis2/+spec/soft-event-delete

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Morten

On Tue, Oct 27, 2015 at 4:01 PM, Knut Staring knutst@gmail.com wrote:

My interpretation was that they would like a flag on each datavalue to say whether it is still current - which I guess would require an extension of that table. However, we do have something akin to this, namely the audit functionality, which preserves the history of earlier (overwritten) values - perhaps that could be leveraged?

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill

11

Hi Morten

We have been testing the deletion with version 2.25 and have had success with programs / program stages.

For data sets we are now able to delete dataValues but cannot delete the dataElements for the following reason:

  • Deleting the dataValues creates dataValueAudits

  • These dataValueAudits reference dataElements

  • Therefore, dataElements cannot be deleted

  • We cannot find a way of deleting dataValueAudits (other than through maintenance but this is only implemented per organisation unit)

Any suggestions would be greatly appreciated.

Many thanks

Alan

···

On Wed, Oct 28, 2015 at 10:30 AM, Lorill Crees lcrees@2paths.com wrote:

Hi Morten,

Thanks for the info regarding events, we’ll give it a try.

As for datasets / data values I tried it with “Accept: text/plain” and it now works, thanks. Please let me know if that header requirement is going to change.

Cheers,

Lorill

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, Oct 27, 2015 at 9:30 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

With blueprint #1

https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

it doesn’t mention the deletion of the data values itself. Would the deletion of events via the web api also cascade delete the underlying data values?

Yes, it should work that way. What you do is this (for events)

{ “events”: [

{ “event”: “id1” },

{ “event”: “id2” }

] }

Then send with

curl -u user:pass -X POST -d @events.json -H “Content-Type: application/json” http://server:port/api/events?strategy=DELETE

Alternately, how would I delete single tracked entity data values via the web API? I can’t seem to find any API call specifically for these values.

curl -X DELETE -u user:pass http://server:port/api/events/ID should work

And for datasets, I have tried sending a delete request for a specific data value but I get a 406. Not sure if I am doing something wrong? I’ve pasted my curl and results below for both GET and DELETE.

I’m not sure exactly why we have this requirement, but you need to use -H “Accept: text/plain” (with curl), I will look into it…

–

Morten

Thanks,

Lorill

curl “http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC” -X GET -u admin:district -v

  • Trying ::1…
  • Connected to localhost (::1) port 8090 (#0)
  • Server auth using Basic with user ‘admin’

GET /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

Host: localhost:8090

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

< HTTP/1.1 200 OK

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3F51C4A3F10135A648E45BAF42818AB7; Path=/dhis/; HttpOnly

< ETag: “0ab3ca659aa5fc585fa72c6474d6b21df”

< Content-Type: application/json;charset=UTF-8

< Content-Language: en-US

< Content-Length: 10

< Date: Tue, 27 Oct 2015 19:16:29 GMT

<

  • Connection #0 to host localhost left intact

[“396411”]

curl “http://localhost:8090/dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC” -X DELETE -u admin:district -v

  • Trying ::1…
  • Connected to localhost (::1) port 8090 (#0)
  • Server auth using Basic with user ‘admin’

DELETE /dhis/api/dataValues?de=YBl3JnHlR5F&pe=2015&ou=qubeTudDNSC HTTP/1.1

Host: localhost:8090

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

< HTTP/1.1 406 Not Acceptable

< Server: Apache-Coyote/1.1

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: DENY

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=10438B228DF6F9FEF289CA05A3E9CA5F; Path=/dhis/; HttpOnly

< Content-Type: text/html;charset=utf-8

< Content-Language: en

< Content-Length: 1110

< Date: Tue, 27 Oct 2015 19:16:35 GMT

<

  • Connection #0 to host localhost left intact
Apache Tomcat/8.0.21 - Error reportH1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}

HTTP Status 406 -

type Status report

message

description The resource identified by this request is only capable of generating responses with characteristics not acceptable according to the request "accept" headers.

Apache Tomcat/8.0.21

On Tue, Oct 27, 2015 at 1:07 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Just FYI, we have two blueprints related to this in 2.22, [1] is implemented already and [2] will hopefully come (but we are still discussing implementation details for that one)

[1] https://blueprints.launchpad.net/dhis2/+spec/webapi-bulk-deletion-of-tracker-objects

[2] https://blueprints.launchpad.net/dhis2/+spec/soft-event-delete

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Morten

On Tue, Oct 27, 2015 at 4:01 PM, Knut Staring knutst@gmail.com wrote:

My interpretation was that they would like a flag on each datavalue to say whether it is still current - which I guess would require an extension of that table. However, we do have something akin to this, namely the audit functionality, which preserves the history of earlier (overwritten) values - perhaps that could be leveraged?

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, Oct 27, 2015 at 5:50 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Aaron,

There is no “soft delete” function, if that is what you are after, but it has been discussed.

Although there is no direct way as Lars points out, if you know what you wish to delete, it is still possible. Namely, if you transmit a “” value for the value which you wish to delete, it will be deleted. Thus, if you wish to delete all values for a given dataset/OU/period, you can get all of the values from the API, then set them all to “”, and upload the data again. I think you can also use the DELETE verb as documented here (https://www.dhis2.org/doc/snapshot/en/developer/html/ch01s12.html) to delete a value.

Not sure if that is the functionality you are after though. Typically, this would be a DB admin’s role though and would be done with SQL, if you are say, going to delete ALL values for a large part of the hierarchy. The possibility for a whoopsie-moment could be large here.

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Mon, Oct 26, 2015 at 10:37 PM, Aaron Gladders agladders@2paths.com wrote:

Hi Lars - is there a recommended approach to “deactive” data without getting into the database itself? If not is it worth setting up a blueprint for it? We may have a group interested in funding that development.

Thanks,

Aaron

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

On Thu, Sep 24, 2015 at 2:39 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Lorill,

sorry but at the moment there is no such option. So far we have decided not to open up for this to prevent mishaps.

regards,

Lars

On Thu, Sep 24, 2015 at 11:37 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

Just following up on this. Any ideas on the best way to programatically bulk delete data values?

Thanks,

Lorill

On Tue, Sep 15, 2015 at 1:04 PM, Lorill Crees lcrees@2paths.com wrote:

Hi,

I seem to remember a prior thread on this but I can’t seem to find it. We want to be able to programatically bulk delete data values (both aggregate and program/program stage data) via the Web API. Is there a way to achieve this?

We don’t want to have to do this on a sql level directly via the database, nor do we want to have to loop through individual data points in order to delete them.

Thanks,

Lorill