Import of DHIS metdata taking long time

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.

···

Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks

Are you sure the process was actually running? No errors in the Tomcat log?

···

On Feb 13, 2017 10:43 AM, “Nalinikanth Meesala” nalinim@thoughtworks.com wrote:

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.


Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Yes, we could see that process was still running, in logs we could see that there are no errors and it is creating, updating dataSets, orgUnits and orgGroups,… etc.

Thanks,

Anil Kumar K

···

On Mon, Feb 13, 2017 at 4:20 PM, Knut Staring knutst@gmail.com wrote:

Are you sure the process was actually running? No errors in the Tomcat log?


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Feb 13, 2017 10:43 AM, “Nalinikanth Meesala” nalinim@thoughtworks.com wrote:

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.


Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi,

Importing a 165mb meta-data export file into 2.24 would be awfully slow, in particular if you have a lot of orgunits. I’ve regularly been exporting/importing ~10,000 OU records - even if I actually only wanted to update a few of them, but DHIS2 do not have any export filters for e.g. lastupdated - and it would typically take several hours. The main culprit is that these OrgUnits are being imported one by one (with a catalina entry for each of them), and I think it is the same for many other objects.

The new export/import functionality is 2.25 and 2.26 is immensely much faster, though - presumably because they are using more efficient sql code (table comparisons instead of one by one). What took 3 hours in 2.24 takes 30 seconds or something in 2.25 and 2.26.

Bottom line: consider moving at least to 2.25, and import there. Otherwise you just have to wait for it to finish…

Regards

Calle

···

On 13 February 2017 at 13:10, Anilkumar Kodi anilkumk@thoughtworks.com wrote:

Yes, we could see that process was still running, in logs we could see that there are no errors and it is creating, updating dataSets, orgUnits and orgGroups,… etc.

Thanks,

Anil Kumar K


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Mon, Feb 13, 2017 at 4:20 PM, Knut Staring knutst@gmail.com wrote:

Are you sure the process was actually running? No errors in the Tomcat log?


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Feb 13, 2017 10:43 AM, “Nalinikanth Meesala” nalinim@thoughtworks.com wrote:

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.


Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Please note that the new importer is available in 2.24 as well, but not through the UI, you will need to use cURL (or a similar http client) and send the payload to /api/24/metadata

curl -X POST -u username:password -d @filename.json -H “Content-Type: application/json” http://server/api/24/metadata

curl -X POST -u username:password -d @filename.xml-H “Content-Type: application/xml” http://server/api/24/metadata

There are fixes in 225/226 that has not been backported… but if your metadata is not too complex, this might work for you.

Also, are you loading this into an empty database? or is it already prefilled with metadata? if you are updating metadata, you might want to add preheatCache=true, which could improve things (depending on your version of 224, this might be default already)

···

On Mon, Feb 13, 2017 at 6:51 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Importing a 165mb meta-data export file into 2.24 would be awfully slow, in particular if you have a lot of orgunits. I’ve regularly been exporting/importing ~10,000 OU records - even if I actually only wanted to update a few of them, but DHIS2 do not have any export filters for e.g. lastupdated - and it would typically take several hours. The main culprit is that these OrgUnits are being imported one by one (with a catalina entry for each of them), and I think it is the same for many other objects.

The new export/import functionality is 2.25 and 2.26 is immensely much faster, though - presumably because they are using more efficient sql code (table comparisons instead of one by one). What took 3 hours in 2.24 takes 30 seconds or something in 2.25 and 2.26.

Bottom line: consider moving at least to 2.25, and import there. Otherwise you just have to wait for it to finish…

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On 13 February 2017 at 13:10, Anilkumar Kodi anilkumk@thoughtworks.com wrote:

Yes, we could see that process was still running, in logs we could see that there are no errors and it is creating, updating dataSets, orgUnits and orgGroups,… etc.

Thanks,

Anil Kumar K


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Mon, Feb 13, 2017 at 4:20 PM, Knut Staring knutst@gmail.com wrote:

Are you sure the process was actually running? No errors in the Tomcat log?


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Feb 13, 2017 10:43 AM, “Nalinikanth Meesala” nalinim@thoughtworks.com wrote:

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.


Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi,

In GENERAL you can import data exported from 2.24 into 2.26 - BUT there seems to be unresolved compatibility issues with the export files of SOME objects. You might be able to identify them by scrutinizing the relevant documentation (which, regrettably, isn’t always 100% up-to-date), but I would rather opt for a segmented export import process (assuming you are importing into an empty database):

  • start with the known “primary” building blocks (those not dependent on anything else), like option sets, attributes, users

  • continue with the section related to orgunits, ougroups, group sets, etc

  • then data element section, then indicators

  • then tracker & events

  • last you export/import everything to related to analysis, reports, charts, maps, dashboards etc.

Do a quick pg_dump after each section. In case of a problem, you can then roll back and try again with a different segment configuration.

Most important of all: document whatever issues you find with incompatible export/import formats, and write a JIRA bug report (as should be obvious, at least upward imports from previous versions must be made to work).

Alternatively, try to import the data into 2.24 using the api/24/metadata as Morten suggests - then upgrade to 2.25 and then 2.26.

My 2c worth

Regards

Calle

···

On 13 February 2017 at 15:08, Karrtik Iyer karrtik@thoughtworks.com wrote:

Hi Calle, thanks, one quick question, can we import files (meta data) exported from 2.24 version into 2.26? Because when we tried that last the import just kept running without showing any progress. Only then we downgraded our version to 2.24.
Thanks,

Karrtik

On Mon, Feb 13, 2017, 5:21 PM Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Importing a 165mb meta-data export file into 2.24 would be awfully slow, in particular if you have a lot of orgunits. I’ve regularly been exporting/importing ~10,000 OU records - even if I actually only wanted to update a few of them, but DHIS2 do not have any export filters for e.g. lastupdated - and it would typically take several hours. The main culprit is that these OrgUnits are being imported one by one (with a catalina entry for each of them), and I think it is the same for many other objects.

The new export/import functionality is 2.25 and 2.26 is immensely much faster, though - presumably because they are using more efficient sql code (table comparisons instead of one by one). What took 3 hours in 2.24 takes 30 seconds or something in 2.25 and 2.26.

Bottom line: consider moving at least to 2.25, and import there. Otherwise you just have to wait for it to finish…

Regards

Calle

On 13 February 2017 at 13:10, Anilkumar Kodi anilkumk@thoughtworks.com wrote:

Yes, we could see that process was still running, in logs we could see that there are no errors and it is creating, updating dataSets, orgUnits and orgGroups,… etc.

Thanks,

Anil Kumar K


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Mon, Feb 13, 2017 at 4:20 PM, Knut Staring knutst@gmail.com wrote:

Are you sure the process was actually running? No errors in the Tomcat log?


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Feb 13, 2017 10:43 AM, “Nalinikanth Meesala” nalinim@thoughtworks.com wrote:

Hi,

We are trying to import Meta data of dhis2 exported from (2.23) to our local instance DHIS 2(2.24) [as import of the exported metadata is not working in DHIS 2.23 ].

The size of meta data is 164 mb can you please help in providing inputs to make the import faster. previously when we tried it took more than 2 days and it was still running.

Ca you please let us know any work around so that we can make it faster.


Thanks & Regards,

Nalinikanth M

Quality Analyst

Email
nalinim@thoughtworks.com
Telephone
+91 9052234588
ThoughtWorks


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg