[Bug 1536817] [NEW] Metadata Import slower in 2.21

Public bug reported:

A metadata import of 1090 data elements (with user group sharing
information) that took 946 seconds in v2.20, took 2,932 seconds in
v2.21.

I can supply the import to developers.

** Affects: dhis2
     Importance: Undecided
         Status: New

···

--
You received this bug notification because you are a member of DHIS 2
developers, which is subscribed to DHIS.
https://bugs.launchpad.net/bugs/1536817

Title:
  Metadata Import slower in 2.21

Status in DHIS:
  New

Bug description:
  A metadata import of 1090 data elements (with user group sharing
  information) that took 946 seconds in v2.20, took 2,932 seconds in
  v2.21.

  I can supply the import to developers.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions

Thanks for reporting this one Jim, I’m seeing the same thing, about a 3 to 4x increase in metaData importation time with 2.21 over 2.20.

image

···

On Fri, Jan 22, 2016 at 7:37 AM, Jim Grace 1536817@bugs.launchpad.net wrote:

Public bug reported:

A metadata import of 1090 data elements (with user group sharing

information) that took 946 seconds in v2.20, took 2,932 seconds in

v2.21.

I can supply the import to developers.

** Affects: dhis2

 Importance: Undecided

     Status: New

You received this bug notification because you are a member of DHIS 2

developers, which is subscribed to DHIS.

https://bugs.launchpad.net/bugs/1536817

Title:

Metadata Import slower in 2.21

Status in DHIS:

New

Bug description:

A metadata import of 1090 data elements (with user group sharing

information) that took 946 seconds in v2.20, took 2,932 seconds in

v2.21.

I can supply the import to developers.

To manage notifications about this bug go to:

https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

** Changed in: dhis2
    Milestone: None => 2.23

** Changed in: dhis2
     Assignee: (unassigned) => Morten Olav Hansen (mortenoh)

** Changed in: dhis2
   Importance: Undecided => Medium

** Changed in: dhis2
   Importance: Medium => High

···

--
You received this bug notification because you are a member of DHIS 2
developers, which is subscribed to DHIS.
https://bugs.launchpad.net/bugs/1536817

Title:
  Metadata Import slower in 2.21

Status in DHIS:
  New

Bug description:
  A metadata import of 1090 data elements (with user group sharing
  information) that took 946 seconds in v2.20, took 2,932 seconds in
  v2.21.

  I can supply the import to developers.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions

Hi Timothy

Was that with or without preheating? the default is now off. We are rewriting the importer for 2.23, so expect a more robust (and hopefully a lot quicker) importer there.

Would it be possible to test without the sharing objects also? just to see if the results are the same.

Please also know that 2.21, 2.22 release both add more validations… so a bit extra time is to be expected (but shouldn’t be a 3-4x increase)

image

···

On Wed, Jan 27, 2016 at 2:54 PM, Timothy Harding tharding@baosystems.com wrote:

Thanks for reporting this one Jim, I’m seeing the same thing, about a 3 to 4x increase in metaData importation time with 2.21 over 2.20.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

On Fri, Jan 22, 2016 at 7:37 AM, Jim Grace 1536817@bugs.launchpad.net wrote:

Public bug reported:

A metadata import of 1090 data elements (with user group sharing

information) that took 946 seconds in v2.20, took 2,932 seconds in

v2.21.

I can supply the import to developers.

** Affects: dhis2

 Importance: Undecided

     Status: New

You received this bug notification because you are a member of DHIS 2

developers, which is subscribed to DHIS.

https://bugs.launchpad.net/bugs/1536817

Title:

Metadata Import slower in 2.21

Status in DHIS:

New

Bug description:

A metadata import of 1090 data elements (with user group sharing

information) that took 946 seconds in v2.20, took 2,932 seconds in

v2.21.

I can supply the import to developers.

To manage notifications about this bug go to:

https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

We had unintentionally set cache preheat to off for meta-data importer.
We have set this back to on in latest 2.22 and trunk. Could you please
try again now?

···

--
You received this bug notification because you are a member of DHIS 2
developers, which is subscribed to DHIS.
https://bugs.launchpad.net/bugs/1536817

Title:
  Metadata Import slower in 2.21

Status in DHIS:
  New

Bug description:
  A metadata import of 1090 data elements (with user group sharing
  information) that took 946 seconds in v2.20, took 2,932 seconds in
  v2.21.

  I can supply the import to developers.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions

A few things caused this, preheat being false on 2.21, attribute values
being validated even there was nothing to validate etc.

Please note that performance on some imports will be slower going
forward as we are doing a lot more validations than before.

Also note that I am rewriting the importer completely, so hopefully
there will be performance upgrades related to that also.

For now, update to latest 2.21, 2.22, and 2.23 for what I could fix.

** Changed in: dhis2
       Status: New => Fix Released

···

--
You received this bug notification because you are a member of DHIS 2
developers, which is subscribed to DHIS.
https://bugs.launchpad.net/bugs/1536817

Title:
  Metadata Import slower in 2.21

Status in DHIS:
  Fix Released

Bug description:
  A metadata import of 1090 data elements (with user group sharing
  information) that took 946 seconds in v2.20, took 2,932 seconds in
  v2.21.

  I can supply the import to developers.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dhis2/+bug/1536817/+subscriptions