Weird cache issue

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.

  2. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.

  3. When I now open the list of data sets, all the changed/new names are showing as expected.

  4. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle

···

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Calle,

perhaps you did not update the data set short names?

Lars

···

On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle

···

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars

On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason

···

On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle

···

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason

On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.

···

2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle

···

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.

2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?

  2. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?

  3. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason

···

On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration → Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Jason,

  1. Yes

  2. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).

  3. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

···

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason

On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?

···

2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Jason,

We have another strange Data Integrity check result - this one is not related to any known change in the database. This one relates to the “Data Elements in data set but not in form or sections” - we are seeing that check returning one data element that definitely exist in both the single section and in the relevant form.

Is there a way of getting a list of those Hibernate-generated sQL queries out of the system, so that we can run any of them manually to see exactly what is going on?

Regards
Calle

···

On 2 September 2015 at 11:40, Calle Hedberg calle.hedberg@gmail.com wrote:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards

···

On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?

2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Sorry forgot the result from the api call (the changes I made to the names and short names were mostly to add “(Monthly)” to the end of a number of data sets):

1

1

31

<dataSet lastUpdated=“2015-07-24T11:33:28.618+0000” created=“2015-07-03T22:03:11.000+0000” name=“Campaign H1N1” id="iv02Urztm60"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iv02Urztm60”/>

<dataSet lastUpdated=“2015-07-24T11:38:16.030+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Municipal Monthly Data” id="Qpm4GX36iew"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Qpm4GX36iew”/>

<dataSet lastUpdated=“2015-07-24T11:38:26.178+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Port Health Monthly Data” id="l0304NG1gwv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/l0304NG1gwv”/>

<dataSet lastUpdated=“2015-07-24T11:38:35.377+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Provincial Monthly Data” id="iM4fznPs94J"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iM4fznPs94J”/>

<dataSet lastUpdated=“2015-07-24T11:38:41.790+0000” created=“2015-07-03T22:03:11.000+0000” name=“Inactive Data” id="UxVE9kD9RoF"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/UxVE9kD9RoF”/>

<dataSet lastUpdated=“2015-08-28T18:48:58.741+0000” created=“2015-08-28T16:40:47.254+0000” name=“Infrastructure” id="tfvYSRdfdpZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/tfvYSRdfdpZ”/>

<dataSet lastUpdated=“2015-07-24T11:39:40.614+0000” created=“2015-07-03T22:03:11.000+0000” name=“NIDS2015_ETR/EDR TB quarterly” id="gHo0s5iOnod"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gHo0s5iOnod”/>

<dataSet lastUpdated=“2015-07-27T18:36:09.782+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Condom Distribution Data (Monthly)” id="BfnkDoLdtcG"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/BfnkDoLdtcG”/>

<dataSet lastUpdated=“2015-08-26T14:00:54.787+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Delivery (Daily)” id="ATjx3YqfzHV"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ATjx3YqfzHV”/>

<dataSet lastUpdated=“2015-08-31T08:02:26.942+0000” created=“2015-04-22T06:45:31.162+0000” name=“PHC Headcount (Daily)” id="xfJZtmus2ol"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/xfJZtmus2ol”/>

<dataSet lastUpdated=“2015-07-24T12:07:07.106+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Other (Monthly)” id="FBVG7vMnDIb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FBVG7vMnDIb”/>

<dataSet lastUpdated=“2015-08-26T14:29:19.976+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Register (Daily)” id="FH9bWrl6hZv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FH9bWrl6hZv”/>

<dataSet lastUpdated=“2015-08-26T14:34:46.108+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Daily)” id="KwRlO3f5reh"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/KwRlO3f5reh”/>

<dataSet lastUpdated=“2015-07-24T13:20:56.500+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Monthly)” id="gbFkAg4lqcA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gbFkAg4lqcA”/>

<dataSet lastUpdated=“2015-08-26T14:38:23.043+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Daily)” id="vcJjlYDf2OB"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/vcJjlYDf2OB”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="jB369YiXDu8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/jB369YiXDu8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="ZMf1BVXplHn"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ZMf1BVXplHn”/>

<dataSet lastUpdated=“2015-07-24T13:26:31.789+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily)” id="kJskR5DrDD1"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/kJskR5DrDD1”/>

<dataSet lastUpdated=“2015-08-26T15:26:09.344+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg1” id="b4cDfJzDLW3"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/b4cDfJzDLW3”/>

<dataSet lastUpdated=“2015-08-26T15:19:11.387+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg2” id="Ir6EXEvs5Yb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Ir6EXEvs5Yb”/>

<dataSet lastUpdated=“2015-07-15T10:52:29.154+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Monthly)” id="Gm0HUWePJO8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gm0HUWePJO8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Daily)” id="TujmX4cORWd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/TujmX4cORWd”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Monthly)” id="V4JQ7wu7463"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/V4JQ7wu7463”/>

<dataSet lastUpdated=“2015-07-24T11:13:38.026+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Daily)” id="e5qzoyswpcI"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/e5qzoyswpcI”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Monthly)” id="HeXgEKw9BQA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/HeXgEKw9BQA”/>

<dataSet lastUpdated=“2015-07-24T11:13:58.169+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Daily)” id="qMafPSFYOpj"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/qMafPSFYOpj”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Monthly)” id="AbDpgIIm4YD"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/AbDpgIIm4YD”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Psychiatric Hospital (Monthly)” id="CssSPUq3dix"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/CssSPUq3dix”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“Population estimates” id="L2hwAPHJyTd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/L2hwAPHJyTd”/>

<dataSet lastUpdated=“2015-07-21T08:49:57.110+0000” created=“2015-07-03T22:03:11.000+0000” name=“Private Hospital Maternal Data (Monthly)” id="huaehwB8tYe"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/huaehwB8tYe”/>

<dataSet lastUpdated=“2015-07-24T13:17:13.240+0000” created=“2015-07-03T22:03:11.000+0000” name=“STI Surveillance” id="Gqa1Ezx3foZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gqa1Ezx3foZ”/>

···

On 2 September 2015 at 13:21, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards

On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi

So just to re-iterate the key question here: api calls, the UI Data Set list and similar were all showing the correct (updated) data set names. Whereas “violations” resulting from running Data Integrity was showing the incorrect (old) data set names.

If it’s possible to get a list of those hibernate-generated SQL queries, it might make it easier to track down the root cause…

Regards

Calle

···

On 2 September 2015 at 13:23, Calle Hedberg calle.hedberg@gmail.com wrote:

Sorry forgot the result from the api call (the changes I made to the names and short names were mostly to add “(Monthly)” to the end of a number of data sets):

1

1

31

<dataSet lastUpdated=“2015-07-24T11:33:28.618+0000” created=“2015-07-03T22:03:11.000+0000” name=“Campaign H1N1” id="iv02Urztm60"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iv02Urztm60”/>

<dataSet lastUpdated=“2015-07-24T11:38:16.030+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Municipal Monthly Data” id="Qpm4GX36iew"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Qpm4GX36iew”/>

<dataSet lastUpdated=“2015-07-24T11:38:26.178+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Port Health Monthly Data” id="l0304NG1gwv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/l0304NG1gwv”/>

<dataSet lastUpdated=“2015-07-24T11:38:35.377+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Provincial Monthly Data” id="iM4fznPs94J"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iM4fznPs94J”/>

<dataSet lastUpdated=“2015-07-24T11:38:41.790+0000” created=“2015-07-03T22:03:11.000+0000” name=“Inactive Data” id="UxVE9kD9RoF"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/UxVE9kD9RoF”/>

<dataSet lastUpdated=“2015-08-28T18:48:58.741+0000” created=“2015-08-28T16:40:47.254+0000” name=“Infrastructure” id="tfvYSRdfdpZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/tfvYSRdfdpZ”/>

<dataSet lastUpdated=“2015-07-24T11:39:40.614+0000” created=“2015-07-03T22:03:11.000+0000” name=“NIDS2015_ETR/EDR TB quarterly” id="gHo0s5iOnod"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gHo0s5iOnod”/>

<dataSet lastUpdated=“2015-07-27T18:36:09.782+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Condom Distribution Data (Monthly)” id="BfnkDoLdtcG"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/BfnkDoLdtcG”/>

<dataSet lastUpdated=“2015-08-26T14:00:54.787+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Delivery (Daily)” id="ATjx3YqfzHV"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ATjx3YqfzHV”/>

<dataSet lastUpdated=“2015-08-31T08:02:26.942+0000” created=“2015-04-22T06:45:31.162+0000” name=“PHC Headcount (Daily)” id="xfJZtmus2ol"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/xfJZtmus2ol”/>

<dataSet lastUpdated=“2015-07-24T12:07:07.106+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Other (Monthly)” id="FBVG7vMnDIb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FBVG7vMnDIb”/>

<dataSet lastUpdated=“2015-08-26T14:29:19.976+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Register (Daily)” id="FH9bWrl6hZv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FH9bWrl6hZv”/>

<dataSet lastUpdated=“2015-08-26T14:34:46.108+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Daily)” id="KwRlO3f5reh"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/KwRlO3f5reh”/>

<dataSet lastUpdated=“2015-07-24T13:20:56.500+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Monthly)” id="gbFkAg4lqcA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gbFkAg4lqcA”/>

<dataSet lastUpdated=“2015-08-26T14:38:23.043+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Daily)” id="vcJjlYDf2OB"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/vcJjlYDf2OB”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="jB369YiXDu8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/jB369YiXDu8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="ZMf1BVXplHn"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ZMf1BVXplHn”/>

<dataSet lastUpdated=“2015-07-24T13:26:31.789+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily)” id="kJskR5DrDD1"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/kJskR5DrDD1”/>

<dataSet lastUpdated=“2015-08-26T15:26:09.344+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg1” id="b4cDfJzDLW3"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/b4cDfJzDLW3”/>

<dataSet lastUpdated=“2015-08-26T15:19:11.387+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg2” id="Ir6EXEvs5Yb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Ir6EXEvs5Yb”/>

<dataSet lastUpdated=“2015-07-15T10:52:29.154+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Monthly)” id="Gm0HUWePJO8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gm0HUWePJO8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Daily)” id="TujmX4cORWd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/TujmX4cORWd”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Monthly)” id="V4JQ7wu7463"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/V4JQ7wu7463”/>

<dataSet lastUpdated=“2015-07-24T11:13:38.026+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Daily)” id="e5qzoyswpcI"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/e5qzoyswpcI”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Monthly)” id="HeXgEKw9BQA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/HeXgEKw9BQA”/>

<dataSet lastUpdated=“2015-07-24T11:13:58.169+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Daily)” id="qMafPSFYOpj"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/qMafPSFYOpj”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Monthly)” id="AbDpgIIm4YD"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/AbDpgIIm4YD”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Psychiatric Hospital (Monthly)” id="CssSPUq3dix"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/CssSPUq3dix”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“Population estimates” id="L2hwAPHJyTd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/L2hwAPHJyTd”/>

<dataSet lastUpdated=“2015-07-21T08:49:57.110+0000” created=“2015-07-03T22:03:11.000+0000” name=“Private Hospital Maternal Data (Monthly)” id="huaehwB8tYe"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/huaehwB8tYe”/>

<dataSet lastUpdated=“2015-07-24T13:17:13.240+0000” created=“2015-07-03T22:03:11.000+0000” name=“STI Surveillance” id="Gqa1Ezx3foZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gqa1Ezx3foZ”/>

On 2 September 2015 at 13:21, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Try and add this to your hibernate.properties file

hibernate.show_sql = true

hibernate.format_sql = true

···

On Wed, Sep 2, 2015 at 5:26 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

So just to re-iterate the key question here: api calls, the UI Data Set list and similar were all showing the correct (updated) data set names. Whereas “violations” resulting from running Data Integrity was showing the incorrect (old) data set names.

If it’s possible to get a list of those hibernate-generated SQL queries, it might make it easier to track down the root cause…

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

On 2 September 2015 at 13:23, Calle Hedberg calle.hedberg@gmail.com wrote:

Sorry forgot the result from the api call (the changes I made to the names and short names were mostly to add “(Monthly)” to the end of a number of data sets):

1

1

31

<dataSet lastUpdated=“2015-07-24T11:33:28.618+0000” created=“2015-07-03T22:03:11.000+0000” name=“Campaign H1N1” id="iv02Urztm60"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iv02Urztm60”/>

<dataSet lastUpdated=“2015-07-24T11:38:16.030+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Municipal Monthly Data” id="Qpm4GX36iew"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Qpm4GX36iew”/>

<dataSet lastUpdated=“2015-07-24T11:38:26.178+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Port Health Monthly Data” id="l0304NG1gwv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/l0304NG1gwv”/>

<dataSet lastUpdated=“2015-07-24T11:38:35.377+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Provincial Monthly Data” id="iM4fznPs94J"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iM4fznPs94J”/>

<dataSet lastUpdated=“2015-07-24T11:38:41.790+0000” created=“2015-07-03T22:03:11.000+0000” name=“Inactive Data” id="UxVE9kD9RoF"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/UxVE9kD9RoF”/>

<dataSet lastUpdated=“2015-08-28T18:48:58.741+0000” created=“2015-08-28T16:40:47.254+0000” name=“Infrastructure” id="tfvYSRdfdpZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/tfvYSRdfdpZ”/>

<dataSet lastUpdated=“2015-07-24T11:39:40.614+0000” created=“2015-07-03T22:03:11.000+0000” name=“NIDS2015_ETR/EDR TB quarterly” id="gHo0s5iOnod"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gHo0s5iOnod”/>

<dataSet lastUpdated=“2015-07-27T18:36:09.782+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Condom Distribution Data (Monthly)” id="BfnkDoLdtcG"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/BfnkDoLdtcG”/>

<dataSet lastUpdated=“2015-08-26T14:00:54.787+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Delivery (Daily)” id="ATjx3YqfzHV"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ATjx3YqfzHV”/>

<dataSet lastUpdated=“2015-08-31T08:02:26.942+0000” created=“2015-04-22T06:45:31.162+0000” name=“PHC Headcount (Daily)” id="xfJZtmus2ol"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/xfJZtmus2ol”/>

<dataSet lastUpdated=“2015-07-24T12:07:07.106+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Other (Monthly)” id="FBVG7vMnDIb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FBVG7vMnDIb”/>

<dataSet lastUpdated=“2015-08-26T14:29:19.976+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Register (Daily)” id="FH9bWrl6hZv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FH9bWrl6hZv”/>

<dataSet lastUpdated=“2015-08-26T14:34:46.108+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Daily)” id="KwRlO3f5reh"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/KwRlO3f5reh”/>

<dataSet lastUpdated=“2015-07-24T13:20:56.500+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Monthly)” id="gbFkAg4lqcA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gbFkAg4lqcA”/>

<dataSet lastUpdated=“2015-08-26T14:38:23.043+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Daily)” id="vcJjlYDf2OB"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/vcJjlYDf2OB”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="jB369YiXDu8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/jB369YiXDu8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="ZMf1BVXplHn"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ZMf1BVXplHn”/>

<dataSet lastUpdated=“2015-07-24T13:26:31.789+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily)” id="kJskR5DrDD1"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/kJskR5DrDD1”/>

<dataSet lastUpdated=“2015-08-26T15:26:09.344+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg1” id="b4cDfJzDLW3"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/b4cDfJzDLW3”/>

<dataSet lastUpdated=“2015-08-26T15:19:11.387+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg2” id="Ir6EXEvs5Yb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Ir6EXEvs5Yb”/>

<dataSet lastUpdated=“2015-07-15T10:52:29.154+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Monthly)” id="Gm0HUWePJO8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gm0HUWePJO8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Daily)” id="TujmX4cORWd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/TujmX4cORWd”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Monthly)” id="V4JQ7wu7463"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/V4JQ7wu7463”/>

<dataSet lastUpdated=“2015-07-24T11:13:38.026+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Daily)” id="e5qzoyswpcI"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/e5qzoyswpcI”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Monthly)” id="HeXgEKw9BQA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/HeXgEKw9BQA”/>

<dataSet lastUpdated=“2015-07-24T11:13:58.169+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Daily)” id="qMafPSFYOpj"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/qMafPSFYOpj”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Monthly)” id="AbDpgIIm4YD"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/AbDpgIIm4YD”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Psychiatric Hospital (Monthly)” id="CssSPUq3dix"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/CssSPUq3dix”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“Population estimates” id="L2hwAPHJyTd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/L2hwAPHJyTd”/>

<dataSet lastUpdated=“2015-07-21T08:49:57.110+0000” created=“2015-07-03T22:03:11.000+0000” name=“Private Hospital Maternal Data (Monthly)” id="huaehwB8tYe"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/huaehwB8tYe”/>

<dataSet lastUpdated=“2015-07-24T13:17:13.240+0000” created=“2015-07-03T22:03:11.000+0000” name=“STI Surveillance” id="Gqa1Ezx3foZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gqa1Ezx3foZ”/>


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:21, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Morten,

Those two properties will result in the queries being listed in the tomcat log? If yes, that’s perfect…

Regards

Calle

···

On 2 September 2015 at 13:27, Morten Olav Hansen mortenoh@gmail.com wrote:

Try and add this to your hibernate.properties file

hibernate.show_sql = true

hibernate.format_sql = true


Morten

On Wed, Sep 2, 2015 at 5:26 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

So just to re-iterate the key question here: api calls, the UI Data Set list and similar were all showing the correct (updated) data set names. Whereas “violations” resulting from running Data Integrity was showing the incorrect (old) data set names.

If it’s possible to get a list of those hibernate-generated SQL queries, it might make it easier to track down the root cause…

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 2 September 2015 at 13:23, Calle Hedberg calle.hedberg@gmail.com wrote:

Sorry forgot the result from the api call (the changes I made to the names and short names were mostly to add “(Monthly)” to the end of a number of data sets):

1

1

31

<dataSet lastUpdated=“2015-07-24T11:33:28.618+0000” created=“2015-07-03T22:03:11.000+0000” name=“Campaign H1N1” id="iv02Urztm60"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iv02Urztm60”/>

<dataSet lastUpdated=“2015-07-24T11:38:16.030+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Municipal Monthly Data” id="Qpm4GX36iew"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Qpm4GX36iew”/>

<dataSet lastUpdated=“2015-07-24T11:38:26.178+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Port Health Monthly Data” id="l0304NG1gwv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/l0304NG1gwv”/>

<dataSet lastUpdated=“2015-07-24T11:38:35.377+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Provincial Monthly Data” id="iM4fznPs94J"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iM4fznPs94J”/>

<dataSet lastUpdated=“2015-07-24T11:38:41.790+0000” created=“2015-07-03T22:03:11.000+0000” name=“Inactive Data” id="UxVE9kD9RoF"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/UxVE9kD9RoF”/>

<dataSet lastUpdated=“2015-08-28T18:48:58.741+0000” created=“2015-08-28T16:40:47.254+0000” name=“Infrastructure” id="tfvYSRdfdpZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/tfvYSRdfdpZ”/>

<dataSet lastUpdated=“2015-07-24T11:39:40.614+0000” created=“2015-07-03T22:03:11.000+0000” name=“NIDS2015_ETR/EDR TB quarterly” id="gHo0s5iOnod"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gHo0s5iOnod”/>

<dataSet lastUpdated=“2015-07-27T18:36:09.782+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Condom Distribution Data (Monthly)” id="BfnkDoLdtcG"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/BfnkDoLdtcG”/>

<dataSet lastUpdated=“2015-08-26T14:00:54.787+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Delivery (Daily)” id="ATjx3YqfzHV"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ATjx3YqfzHV”/>

<dataSet lastUpdated=“2015-08-31T08:02:26.942+0000” created=“2015-04-22T06:45:31.162+0000” name=“PHC Headcount (Daily)” id="xfJZtmus2ol"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/xfJZtmus2ol”/>

<dataSet lastUpdated=“2015-07-24T12:07:07.106+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Other (Monthly)” id="FBVG7vMnDIb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FBVG7vMnDIb”/>

<dataSet lastUpdated=“2015-08-26T14:29:19.976+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Register (Daily)” id="FH9bWrl6hZv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FH9bWrl6hZv”/>

<dataSet lastUpdated=“2015-08-26T14:34:46.108+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Daily)” id="KwRlO3f5reh"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/KwRlO3f5reh”/>

<dataSet lastUpdated=“2015-07-24T13:20:56.500+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Monthly)” id="gbFkAg4lqcA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gbFkAg4lqcA”/>

<dataSet lastUpdated=“2015-08-26T14:38:23.043+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Daily)” id="vcJjlYDf2OB"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/vcJjlYDf2OB”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="jB369YiXDu8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/jB369YiXDu8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="ZMf1BVXplHn"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ZMf1BVXplHn”/>

<dataSet lastUpdated=“2015-07-24T13:26:31.789+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily)” id="kJskR5DrDD1"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/kJskR5DrDD1”/>

<dataSet lastUpdated=“2015-08-26T15:26:09.344+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg1” id="b4cDfJzDLW3"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/b4cDfJzDLW3”/>

<dataSet lastUpdated=“2015-08-26T15:19:11.387+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg2” id="Ir6EXEvs5Yb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Ir6EXEvs5Yb”/>

<dataSet lastUpdated=“2015-07-15T10:52:29.154+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Monthly)” id="Gm0HUWePJO8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gm0HUWePJO8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Daily)” id="TujmX4cORWd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/TujmX4cORWd”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Monthly)” id="V4JQ7wu7463"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/V4JQ7wu7463”/>

<dataSet lastUpdated=“2015-07-24T11:13:38.026+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Daily)” id="e5qzoyswpcI"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/e5qzoyswpcI”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Monthly)” id="HeXgEKw9BQA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/HeXgEKw9BQA”/>

<dataSet lastUpdated=“2015-07-24T11:13:58.169+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Daily)” id="qMafPSFYOpj"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/qMafPSFYOpj”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Monthly)” id="AbDpgIIm4YD"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/AbDpgIIm4YD”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Psychiatric Hospital (Monthly)” id="CssSPUq3dix"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/CssSPUq3dix”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“Population estimates” id="L2hwAPHJyTd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/L2hwAPHJyTd”/>

<dataSet lastUpdated=“2015-07-21T08:49:57.110+0000” created=“2015-07-03T22:03:11.000+0000” name=“Private Hospital Maternal Data (Monthly)” id="huaehwB8tYe"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/huaehwB8tYe”/>

<dataSet lastUpdated=“2015-07-24T13:17:13.240+0000” created=“2015-07-03T22:03:11.000+0000” name=“STI Surveillance” id="Gqa1Ezx3foZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gqa1Ezx3foZ”/>


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:21, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Yes, it just spits out all the queries hibernate is doing

···

On Wed, Sep 2, 2015 at 5:29 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Morten,

Those two properties will result in the queries being listed in the tomcat log? If yes, that’s perfect…

Regards

Calle


Morten

On 2 September 2015 at 13:27, Morten Olav Hansen mortenoh@gmail.com wrote:

Try and add this to your hibernate.properties file

hibernate.show_sql = true

hibernate.format_sql = true


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Morten

On Wed, Sep 2, 2015 at 5:26 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

So just to re-iterate the key question here: api calls, the UI Data Set list and similar were all showing the correct (updated) data set names. Whereas “violations” resulting from running Data Integrity was showing the incorrect (old) data set names.

If it’s possible to get a list of those hibernate-generated SQL queries, it might make it easier to track down the root cause…

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 2 September 2015 at 13:23, Calle Hedberg calle.hedberg@gmail.com wrote:

Sorry forgot the result from the api call (the changes I made to the names and short names were mostly to add “(Monthly)” to the end of a number of data sets):

1

1

31

<dataSet lastUpdated=“2015-07-24T11:33:28.618+0000” created=“2015-07-03T22:03:11.000+0000” name=“Campaign H1N1” id="iv02Urztm60"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iv02Urztm60”/>

<dataSet lastUpdated=“2015-07-24T11:38:16.030+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Municipal Monthly Data” id="Qpm4GX36iew"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Qpm4GX36iew”/>

<dataSet lastUpdated=“2015-07-24T11:38:26.178+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Port Health Monthly Data” id="l0304NG1gwv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/l0304NG1gwv”/>

<dataSet lastUpdated=“2015-07-24T11:38:35.377+0000” created=“2015-07-03T22:03:11.000+0000” name=“EHS Provincial Monthly Data” id="iM4fznPs94J"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/iM4fznPs94J”/>

<dataSet lastUpdated=“2015-07-24T11:38:41.790+0000” created=“2015-07-03T22:03:11.000+0000” name=“Inactive Data” id="UxVE9kD9RoF"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/UxVE9kD9RoF”/>

<dataSet lastUpdated=“2015-08-28T18:48:58.741+0000” created=“2015-08-28T16:40:47.254+0000” name=“Infrastructure” id="tfvYSRdfdpZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/tfvYSRdfdpZ”/>

<dataSet lastUpdated=“2015-07-24T11:39:40.614+0000” created=“2015-07-03T22:03:11.000+0000” name=“NIDS2015_ETR/EDR TB quarterly” id="gHo0s5iOnod"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gHo0s5iOnod”/>

<dataSet lastUpdated=“2015-07-27T18:36:09.782+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Condom Distribution Data (Monthly)” id="BfnkDoLdtcG"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/BfnkDoLdtcG”/>

<dataSet lastUpdated=“2015-08-26T14:00:54.787+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Delivery (Daily)” id="ATjx3YqfzHV"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ATjx3YqfzHV”/>

<dataSet lastUpdated=“2015-08-31T08:02:26.942+0000” created=“2015-04-22T06:45:31.162+0000” name=“PHC Headcount (Daily)” id="xfJZtmus2ol"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/xfJZtmus2ol”/>

<dataSet lastUpdated=“2015-07-24T12:07:07.106+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Other (Monthly)” id="FBVG7vMnDIb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FBVG7vMnDIb”/>

<dataSet lastUpdated=“2015-08-26T14:29:19.976+0000” created=“2015-07-03T22:03:11.000+0000” name=“PHC Register (Daily)” id="FH9bWrl6hZv"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/FH9bWrl6hZv”/>

<dataSet lastUpdated=“2015-08-26T14:34:46.108+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Daily)” id="KwRlO3f5reh"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/KwRlO3f5reh”/>

<dataSet lastUpdated=“2015-07-24T13:20:56.500+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_District Hospital (Monthly)” id="gbFkAg4lqcA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/gbFkAg4lqcA”/>

<dataSet lastUpdated=“2015-08-26T14:38:23.043+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Daily)” id="vcJjlYDf2OB"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/vcJjlYDf2OB”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="jB369YiXDu8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/jB369YiXDu8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_EMS (Monthly)” id="ZMf1BVXplHn"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/ZMf1BVXplHn”/>

<dataSet lastUpdated=“2015-07-24T13:26:31.789+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily)” id="kJskR5DrDD1"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/kJskR5DrDD1”/>

<dataSet lastUpdated=“2015-08-26T15:26:09.344+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg1” id="b4cDfJzDLW3"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/b4cDfJzDLW3”/>

<dataSet lastUpdated=“2015-08-26T15:19:11.387+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Daily) pg2” id="Ir6EXEvs5Yb"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Ir6EXEvs5Yb”/>

<dataSet lastUpdated=“2015-07-15T10:52:29.154+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Hospital (Monthly)” id="Gm0HUWePJO8"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gm0HUWePJO8”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Daily)” id="TujmX4cORWd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/TujmX4cORWd”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Mobile Services (Monthly)” id="V4JQ7wu7463"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/V4JQ7wu7463”/>

<dataSet lastUpdated=“2015-07-24T11:13:38.026+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Daily)” id="e5qzoyswpcI"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/e5qzoyswpcI”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility with MOU (Monthly)” id="HeXgEKw9BQA"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/HeXgEKw9BQA”/>

<dataSet lastUpdated=“2015-07-24T11:13:58.169+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Daily)” id="qMafPSFYOpj"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/qMafPSFYOpj”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_PHC facility without MOU (Monthly)” id="AbDpgIIm4YD"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/AbDpgIIm4YD”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“PIDS2013_Psychiatric Hospital (Monthly)” id="CssSPUq3dix"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/CssSPUq3dix”/>

<dataSet lastUpdated=“2015-07-03T22:03:11.000+0000” created=“2015-07-03T22:03:11.000+0000” name=“Population estimates” id="L2hwAPHJyTd"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/L2hwAPHJyTd”/>

<dataSet lastUpdated=“2015-07-21T08:49:57.110+0000” created=“2015-07-03T22:03:11.000+0000” name=“Private Hospital Maternal Data (Monthly)” id="huaehwB8tYe"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/huaehwB8tYe”/>

<dataSet lastUpdated=“2015-07-24T13:17:13.240+0000” created=“2015-07-03T22:03:11.000+0000” name=“STI Surveillance” id="Gqa1Ezx3foZ"href=“http://fs.dhis.dhmis.org/staging/api/dataSets/Gqa1Ezx3foZ”/>


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:21, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Ok. I’m sorry the api query was lacking a parameter (should be api/dataSet.json&fields=name,shortName,displayName) to show all the fields.
DisplayNames are generated (i.e. they’re either the translation if one exists or they’re simply the name). If you’re not using translations that’s obviously a dead end though.

···

2015-09-02 12:21 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards

On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Halvdan,

Noted, thanks for clarification - and yes, this issue is not related to that (I’m still 98% sure it’s somehow related to caching, I just don’t know how - my reason for pursuing this is obviously that since we are regularly updating databases directly or via the web api, we want to be sure that all cache refreshing is working and thus ensure that such updates are pushed to users immediately.

Regards

Calle

···

On 2 September 2015 at 13:35, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok. I’m sorry the api query was lacking a parameter (should be api/dataSet.json&fields=name,shortName,displayName) to show all the fields.
DisplayNames are generated (i.e. they’re either the translation if one exists or they’re simply the name). If you’re not using translations that’s obviously a dead end though.

2015-09-02 12:21 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards

On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Probably a stupid question, but did you try incognito mode or another browser?

···

On Wed, Sep 2, 2015 at 12:44 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Noted, thanks for clarification - and yes, this issue is not related to that (I’m still 98% sure it’s somehow related to caching, I just don’t know how - my reason for pursuing this is obviously that since we are regularly updating databases directly or via the web api, we want to be sure that all cache refreshing is working and thus ensure that such updates are pushed to users immediately.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 2 September 2015 at 13:35, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok. I’m sorry the api query was lacking a parameter (should be api/dataSet.json&fields=name,shortName,displayName) to show all the fields.
DisplayNames are generated (i.e. they’re either the translation if one exists or they’re simply the name). If you’re not using translations that’s obviously a dead end though.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 12:21 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

We do not use translations for any South African instance - all English (i.e. the translation table has 0 rows)

We are using either 2.19 or 2.20 (mostly 2.19) - actually a slightly modified version of 2.19 in most cases. That version contains additional code related to DHIS 1.4->DHIS2 conversions etc, but none of that code is in any way linked to this problem.

/api/dataSets gives the result below, or in other words it only show names - and all those names are the correct/updated ones.

I’m not sure where “displayNames” would be coming from - I am not aware that data sets HAVE a field called “displayName” or similar. In which table can I find that?

Regards

On 2 September 2015 at 13:05, Halvdan Grelland halvdanhg@gmail.com wrote:

Could you go to /api/dataSets and confirm that the names, shortNames and displayNames are correct? The names displayed should be the displayNames; which reminds me: do you have translations for the given dataSets/current locale? If you do they will be displayed, and they will not have been updated.

Q:

Is the problem true for all objects in the integrity checks, or just DataSets? What about DataElements?

Q:

Also, what version are you on?

Try running …
select * from translation where objectclass like ‘%DataSet%’;

…and inspect the results (might be a large query if you have a lot of dataSets).

Also find the uid of one of the affected DataSets and try to run:
select * from translation where objectuid like ‘’;
… any hints there?


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-02 10:40 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason,

  1. Yes
  1. Yes, I changed both name and shortname (identical) - but I do not know if the Data Integrity checks display name or short name (settings = short name for that instance).
  1. We are not using any caching proxy

We did restart Tomcat - which should clear the server cache (?) - and I did clear all my local caches.

BUT we had some other strange behaviour with regard to dashboard privileges yesterday, and it was seemingly fixed by clearing the DHIS2 cache MULTIPLE times (not sure how many, but I only cleared the cache one time…)

Regards

Calle

On 2 September 2015 at 10:10, Jason Pickering jason.p.pickering@gmail.com wrote:

So, coming back to Lars’s point.

  1. Did you check the “dataset” table to confirm that your intended changes were persisted?
  1. Did you alter the dataset “shortname” and is that the same which you are seeing in your integrity checks?
  1. You indicated you cleared the DHIS 2 server cache. Can you confirm that you have no other upstream caching proxy which might be caching some response?

The data integrity checks are a series of SQL (actually generated by Hibernate) queries, so as is usually the case, if you make direct database manipulations (and do not clear the server cache and possibly your local browser cache) you may see inconsistent results. For this type of operation, use of the UI is probably better, or alternative with the WebAPI.

Regards,
Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:57 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

Ah, OK, and not - I was not using any queries, I simply updated the Data Set names directly in the table.

As I’ve said - the weird thing is that all those new names immediately showed up in the UI data set list, but the old names are still appearing wherever the Data Integrity process throws up integrity violations

Regards
Calle


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 18:01, Halvdan Grelland halvdanhg@gmail.com wrote:

Calle, I’m pretty sure Jason was referring to whatever you were doing to update the records (your point #1 in original email) which was not, as far as I can understand, using the UI or the web-api. I am not familiar with “pgremote” but am assuming it’s some kind of postgres connector allowing you to run (sql) queries. We’d need to see those queries in order to figure out what could have gone wrong.


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


2015-09-01 14:50 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Jason

I was not executing any custom sql - I’m running the Data Administration -> Data Integrity process checks.

My reference to the sql queries being run relates to my ASSUMPTION that the Data Integrity checks are a series of SQL queries - but maybe not, maybe the integrity checks are a series of web api calls followed by some kind of front-end processing? I just don’t know, and I’m not a Java developer and I don’t have the code base on my laptop so I cannot find out myself…

Seems my initial post was not clear enough on this - apologies for that.

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 1 September 2015 at 14:31, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,
It is very difficult to say much without knowing what SQL it was you executed. In general, use of the API or UI is usually much safer.

Do you have any sort of cache in front of your DHIS2 instance?

Can you confirm from the database that the names were actually changed?

Regards,

Jason


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 12:49 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Lars,

Nope - all data set names and short names are the same.

It’s just weird, and I don’t like inexplicable things happening to a database…

When you run data integrity - WHERE are those queries fetching data from? Are the integrity queries running against the database itself, or some kind of cache repository?

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On 1 September 2015 at 13:00, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Calle,

perhaps you did not update the data set short names?

Lars


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Tue, Sep 1, 2015 at 11:50 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

I’ve got a baffling problem (presumably related to some kind of caching):

  1. yesterday I updated various data records using pgremote, including data set names. We restarted tomcat after the updates.
  1. I’ve also cleared my browser cache, the DHIS2 cache, updated resource tables, etc.
  1. When I now open the list of data sets, all the changed/new names are showing as expected.
  1. BUT when I run Data Integrity, a few violations related to data sets are coming up AND THE DATA SET NAMES ARE THE OLD ONES.

Any ideas what could cause this weird behaviour?

Regards
Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org