Meta-data import - dry-run works, no-dry-run hangs

Hi,

I'm trying to transfer two relatively small data sets with dependent
objects from one DHIS2 instance to another. I've tried to use both XML
and JSON formats.

When importing as a dry run, it works as expected - takes a few seconds.

When importing the same with dry run = No, the progress display
indicates that OrgUnits, categories, dataelements etc have been
imported, but it hangs displaying the "progress bar" at the Datasets 1
..... Nothing is actually imported.

Any idea what could cause this type of behaviour? (NOTE: the
destination instance have many data elements, data sets, orgunits etc
- but zero categories and categorycombooptions - before the import).

Regards from Cape town
Calle

···

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

Hi Calle,

This is quite common behaviour, but unfortunately, the error is only
available in the logs. You will need to get a copy of the logs while
performing the import to see exactly what is causing the process to
fail.

Best regards,
Jason

···

On Sun, Jan 25, 2015 at 1:13 PM, Calle Hedberg <calle.hedberg@gmail.com> wrote:

Hi,

I'm trying to transfer two relatively small data sets with dependent
objects from one DHIS2 instance to another. I've tried to use both XML
and JSON formats.

When importing as a dry run, it works as expected - takes a few seconds.

When importing the same with dry run = No, the progress display
indicates that OrgUnits, categories, dataelements etc have been
imported, but it hangs displaying the "progress bar" at the Datasets 1
..... Nothing is actually imported.

Any idea what could cause this type of behaviour? (NOTE: the
destination instance have many data elements, data sets, orgunits etc
- but zero categories and categorycombooptions - before the import).

Regards from Cape town
Calle

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Jason,

Thanks for the rapid (Sunday :slight_smile: ) feedback!

Sigh - yeah, I actually got hold of Jason and we checked the Tomcat
log since the post, but the logs didn't really clear up exactly why
this is happening:

WARN 2015-01-25 12:37:10,122 HHH000437: Attempting to save one or
more entities that have a non-nullable association with an unsaved
transient entity. The unsaved transient entity must be saved in an
operation prior to saving these dependent entities.
        Unsaved transient entity: ([org.hisp.dhis.period.QuarterlyPeriodType#0])
        Dependent entities: ([[org.hisp.dhis.dataset.DataSet#18246]])
        Non-nullable association(s):
([org.hisp.dhis.dataset.DataSet.periodType])
(UnresolvedEntityInsertActions.java [taskScheduler-9])

I don't really know what "unsaved transient entity" mean here - things
seems to point towards the quarterly dataperiodtype missing, but that
don't make any sense since that period is a built-in standard period
(?)

And it also strange that the dry run would work perfectly while the
real thing crash - that seems to indicate that some checks are missing
from the dry-run routine.

I've been able to work around it by importing all the data elements in
the data set, and then define the data set afresh in the destination
instance and then allocate the new data elements to it. Very
cumbersome, though, so I would have liked to know the root cause of
this.

Regards
Calle

···

On 25/01/2015, Jason Pickering <jason.p.pickering@gmail.com> wrote:

Hi Calle,

This is quite common behaviour, but unfortunately, the error is only
available in the logs. You will need to get a copy of the logs while
performing the import to see exactly what is causing the process to
fail.

Best regards,
Jason

On Sun, Jan 25, 2015 at 1:13 PM, Calle Hedberg <calle.hedberg@gmail.com> > wrote:

Hi,

I'm trying to transfer two relatively small data sets with dependent
objects from one DHIS2 instance to another. I've tried to use both XML
and JSON formats.

When importing as a dry run, it works as expected - takes a few seconds.

When importing the same with dry run = No, the progress display
indicates that OrgUnits, categories, dataelements etc have been
imported, but it hangs displaying the "progress bar" at the Datasets 1
..... Nothing is actually imported.

Any idea what could cause this type of behaviour? (NOTE: the
destination instance have many data elements, data sets, orgunits etc
- but zero categories and categorycombooptions - before the import).

Regards from Cape town
Calle

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

Hi

Can you try with the latest version? we have updated both 2.18 and trunk for a similar bug

···

On Sun, Jan 25, 2015 at 7:47 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Jason,

Thanks for the rapid (Sunday :slight_smile: ) feedback!

Sigh - yeah, I actually got hold of Jason and we checked the Tomcat

log since the post, but the logs didn’t really clear up exactly why

this is happening:

WARN 2015-01-25 12:37:10,122 HHH000437: Attempting to save one or

more entities that have a non-nullable association with an unsaved

transient entity. The unsaved transient entity must be saved in an

operation prior to saving these dependent entities.

    Unsaved transient entity: ([org.hisp.dhis.period.QuarterlyPeriodType#0])

    Dependent entities: ([[org.hisp.dhis.dataset.DataSet#18246]])

    Non-nullable association(s):

([org.hisp.dhis.dataset.DataSet.periodType])

(UnresolvedEntityInsertActions.java [taskScheduler-9])

I don’t really know what “unsaved transient entity” mean here - things

seems to point towards the quarterly dataperiodtype missing, but that

don’t make any sense since that period is a built-in standard period

(?)

And it also strange that the dry run would work perfectly while the

real thing crash - that seems to indicate that some checks are missing

from the dry-run routine.

I’ve been able to work around it by importing all the data elements in

the data set, and then define the data set afresh in the destination

instance and then allocate the new data elements to it. Very

cumbersome, though, so I would have liked to know the root cause of

this.

Regards

Calle

On 25/01/2015, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Calle,

This is quite common behaviour, but unfortunately, the error is only

available in the logs. You will need to get a copy of the logs while

performing the import to see exactly what is causing the process to

fail.

Best regards,

Jason

On Sun, Jan 25, 2015 at 1:13 PM, Calle Hedberg calle.hedberg@gmail.com

wrote:

Hi,

I’m trying to transfer two relatively small data sets with dependent

objects from one DHIS2 instance to another. I’ve tried to use both XML

and JSON formats.

When importing as a dry run, it works as expected - takes a few seconds.

When importing the same with dry run = No, the progress display

indicates that OrgUnits, categories, dataelements etc have been

imported, but it hangs displaying the “progress bar” at the Datasets 1

… Nothing is actually imported.

Any idea what could cause this type of behaviour? (NOTE: the

destination instance have many data elements, data sets, orgunits etc

  • but zero categories and categorycombooptions - before the import).

Regards from Cape town

Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

Hi,

OK, will try tomorrow - busy with other work right now.

Regards
calle

···

On 25/01/2015, Morten Olav Hansen <mortenoh@gmail.com> wrote:

Hi

Can you try with the latest version? we have updated both 2.18 and trunk
for a similar bug

--
Morten

On Sun, Jan 25, 2015 at 7:47 PM, Calle Hedberg <calle.hedberg@gmail.com> > wrote:

Jason,

Thanks for the rapid (Sunday :slight_smile: ) feedback!

Sigh - yeah, I actually got hold of Jason and we checked the Tomcat
log since the post, but the logs didn't really clear up exactly why
this is happening:

WARN 2015-01-25 12:37:10,122 HHH000437: Attempting to save one or
more entities that have a non-nullable association with an unsaved
transient entity. The unsaved transient entity must be saved in an
operation prior to saving these dependent entities.
        Unsaved transient entity:
([org.hisp.dhis.period.QuarterlyPeriodType#0])
        Dependent entities: ([[org.hisp.dhis.dataset.DataSet#18246]])
        Non-nullable association(s):
([org.hisp.dhis.dataset.DataSet.periodType])
(UnresolvedEntityInsertActions.java [taskScheduler-9])

I don't really know what "unsaved transient entity" mean here - things
seems to point towards the quarterly dataperiodtype missing, but that
don't make any sense since that period is a built-in standard period
(?)

And it also strange that the dry run would work perfectly while the
real thing crash - that seems to indicate that some checks are missing
from the dry-run routine.

I've been able to work around it by importing all the data elements in
the data set, and then define the data set afresh in the destination
instance and then allocate the new data elements to it. Very
cumbersome, though, so I would have liked to know the root cause of
this.

Regards
Calle

On 25/01/2015, Jason Pickering <jason.p.pickering@gmail.com> wrote:
> Hi Calle,
>
> This is quite common behaviour, but unfortunately, the error is only
> available in the logs. You will need to get a copy of the logs while
> performing the import to see exactly what is causing the process to
> fail.
>
> Best regards,
> Jason
>
>
> On Sun, Jan 25, 2015 at 1:13 PM, Calle Hedberg >> > <calle.hedberg@gmail.com> >> > wrote:
>> Hi,
>>
>> I'm trying to transfer two relatively small data sets with dependent
>> objects from one DHIS2 instance to another. I've tried to use both XML
>> and JSON formats.
>>
>> When importing as a dry run, it works as expected - takes a few
>> seconds.
>>
>> When importing the same with dry run = No, the progress display
>> indicates that OrgUnits, categories, dataelements etc have been
>> imported, but it hangs displaying the "progress bar" at the Datasets 1
>> ..... Nothing is actually imported.
>>
>> Any idea what could cause this type of behaviour? (NOTE: the
>> destination instance have many data elements, data sets, orgunits etc
>> - but zero categories and categorycombooptions - before the import).
>>
>> Regards from Cape town
>> Calle
>>
>>
>>
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@gmail.com
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>> _______________________________________________
>> Mailing list: DHIS 2 developers in Launchpad
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : DHIS 2 developers in Launchpad
>> More help : ListHelp - Launchpad Help
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@gmail.com
> tel:+46764147049
>

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

Morten

I've just tried it with the very latest stable 2.18 - no difference.
The dry run works fine, but the real import chokes on the data set.

The work-around - importing data elements (or data element groups) -
works fine - I just have to re-create the data set, re-allocate data
elements to the data set, and allocate the data set to orgunits.

Interestingly: I've also just tried to import the same dataset xml
export file into the Sierra Leone demo database, and that works. Which
logically means that there is probably something with my destination
instance that causes this block. Since the instance is a conversion
from a 1.4 data file, there might be an unknown issue related to the
conversion that does not get picked up by integrity checks etc.

Will continue checking tomorrow.

Regards
Calle

···

On 25/01/2015, Calle Hedberg <calle.hedberg@gmail.com> wrote:

Hi,

OK, will try tomorrow - busy with other work right now.

Regards
calle

On 25/01/2015, Morten Olav Hansen <mortenoh@gmail.com> wrote:

Hi

Can you try with the latest version? we have updated both 2.18 and trunk
for a similar bug

--
Morten

On Sun, Jan 25, 2015 at 7:47 PM, Calle Hedberg <calle.hedberg@gmail.com> >> wrote:

Jason,

Thanks for the rapid (Sunday :slight_smile: ) feedback!

Sigh - yeah, I actually got hold of Jason and we checked the Tomcat
log since the post, but the logs didn't really clear up exactly why
this is happening:

WARN 2015-01-25 12:37:10,122 HHH000437: Attempting to save one or
more entities that have a non-nullable association with an unsaved
transient entity. The unsaved transient entity must be saved in an
operation prior to saving these dependent entities.
        Unsaved transient entity:
([org.hisp.dhis.period.QuarterlyPeriodType#0])
        Dependent entities: ([[org.hisp.dhis.dataset.DataSet#18246]])
        Non-nullable association(s):
([org.hisp.dhis.dataset.DataSet.periodType])
(UnresolvedEntityInsertActions.java [taskScheduler-9])

I don't really know what "unsaved transient entity" mean here - things
seems to point towards the quarterly dataperiodtype missing, but that
don't make any sense since that period is a built-in standard period
(?)

And it also strange that the dry run would work perfectly while the
real thing crash - that seems to indicate that some checks are missing
from the dry-run routine.

I've been able to work around it by importing all the data elements in
the data set, and then define the data set afresh in the destination
instance and then allocate the new data elements to it. Very
cumbersome, though, so I would have liked to know the root cause of
this.

Regards
Calle

On 25/01/2015, Jason Pickering <jason.p.pickering@gmail.com> wrote:
> Hi Calle,
>
> This is quite common behaviour, but unfortunately, the error is only
> available in the logs. You will need to get a copy of the logs while
> performing the import to see exactly what is causing the process to
> fail.
>
> Best regards,
> Jason
>
>
> On Sun, Jan 25, 2015 at 1:13 PM, Calle Hedberg >>> > <calle.hedberg@gmail.com> >>> > wrote:
>> Hi,
>>
>> I'm trying to transfer two relatively small data sets with dependent
>> objects from one DHIS2 instance to another. I've tried to use both
>> XML
>> and JSON formats.
>>
>> When importing as a dry run, it works as expected - takes a few
>> seconds.
>>
>> When importing the same with dry run = No, the progress display
>> indicates that OrgUnits, categories, dataelements etc have been
>> imported, but it hangs displaying the "progress bar" at the Datasets
>> 1
>> ..... Nothing is actually imported.
>>
>> Any idea what could cause this type of behaviour? (NOTE: the
>> destination instance have many data elements, data sets, orgunits etc
>> - but zero categories and categorycombooptions - before the import).
>>
>> Regards from Cape town
>> Calle
>>
>>
>>
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@gmail.com
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>> _______________________________________________
>> Mailing list: DHIS 2 developers in Launchpad
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : DHIS 2 developers in Launchpad
>> More help : ListHelp - Launchpad Help
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@gmail.com
> tel:+46764147049
>

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************

--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

*******************************************