[Branch ~dhis2-devs-core/dhis2/trunk] Rev 4838: Put code uniqueness check back in import module

Sounds good. I remember at some point in time, the code was required

from the UI (but not the DB). As long as NULL values are acceptable,

but a constraint is placed on the uniqueness, I think this should be

fine.

The codes are only really required when integrating between systems.

Then we match on the code (rather than the name). Unless you are

specifically configuring such a scenario, or unless there are

authoritative codes to use such as from a MFL, your average user could

and should be happy to leave them blank. Of course the problem arises

where a user is presented with a blank field and feels naturally

compelled to enter something. We’ll have to deal with that but

perhaps not urgently.

I guess I am not really understanding in the MFL scenario why you should need to match on a code instead of a name. Is it just a matter of convenience? Why not UUID, which I suppose the code could be as well, since it is just a text field anyway?

Lars, I suspect some of the 1.4 import code needs to be looked at, as

if I remember correctly, this could have been one of the reasons (at

my insistence) that we relaxed the uniqueness constraint some time

back, as it made imports from 1.4 databases impossible where there

were uniqueness violations.

Should we just throw away these codes on import?

I suspect so if it is coming from 1.4 but why should we not do this for all DXF imports? There is no guarantee the codes would be unique anyway so seems better to import what we can and ditch the rest as outlined in the previous mail.

Regards,

Jason

···

On Oct 5, 2011 4:33 PM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

2011/10/5 Jason Pickering jason.p.pickering@gmail.com:

Regards,

Jason

2011/10/5 Bob Jolliffe bobjolliffe@gmail.com:

2011/10/5 Jason Pickering jason.p.pickering@gmail.com:

Is there going to be a way for the system to autogenerate a code if 1)

there is a collision during import 2) if the user does not enter one

(I suppose it will be enforced through the UI), but does not know what

the code should be?

My understanding is that the code must be unique but can also be null.

So if there is no code then there is no code.

When importing data in Kenya with datasets from over 8000 orgunits the

probabliity of mismatches and duplicate orgunit codes was quite high.

In this case, where a duplicate is encountered both values are

deleted (and logged) as there is really no way to determine which one

was using the correct code.

Regarding what to do at the UI - if there is no assigned code (eg from

a master facility list) then the user should simply not enter one.

On Wed, Oct 5, 2011 at 1:51 PM, noreply@launchpad.net wrote:


revno: 4838

committer: Lars Helge Overland larshelge@gmail.com

branch nick: dhis2

timestamp: Wed 2011-10-05 13:50:05 +0200

message:

Put code uniqueness check back in import module

modified:

dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java

dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java

dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java

dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java’

— dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java 2011-04-24 15:50:02 +0000

+++ dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java 2011-10-05 11:50:05 +0000

@@ -99,6 +99,10 @@

    {
        match = dataElementService.getDataElementByShortName( object.getShortName() );
    }
  •    if ( match == null )
    
  •    {
    
  •        match = dataElementService.getDataElementByCode( object.getCode() );
    
  •    }
    
    return match;
}

=== modified file ‘dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java’

— dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java 2011-09-24 11:35:11 +0000

+++ dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java 2011-10-05 11:50:05 +0000

@@ -100,6 +100,10 @@

    {
        match = indicatorService.getIndicatorByShortName( object.getShortName() );
    }
  •    if ( match == null )
    
  •    {
    
  •        match = indicatorService.getIndicatorByCode( object.getCode() );
    
  •    }
    
    return match;
}

=== modified file ‘dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java’

— dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java 2011-04-22 21:04:14 +0000

+++ dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java 2011-10-05 11:50:05 +0000

@@ -92,7 +92,14 @@

@Override
protected OrganisationUnit getMatching( OrganisationUnit object )
{
  •    return organisationUnitService.getOrganisationUnitByName( object.getName() );
    
  •    OrganisationUnit match = organisationUnitService.getOrganisationUnitByName( object.getName() );
    
  •    if ( match == null )
    
  •    {
    
  •        match = organisationUnitService.getOrganisationUnitByCode( object.getCode() );
    
  •    }
    
  •    return match;
    
}
@Override

=== modified file ‘dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java’

— dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java 2011-09-23 18:54:49 +0000

+++ dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java 2011-10-05 11:50:05 +0000

@@ -63,6 +63,7 @@

import org.junit.Ignore;

import org.junit.Test;

+@Ignore //TODO fix

public class DataValueSetServiceTest

extends DhisTest

{


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

> Sounds good. I remember at some point in time, the code was required
> from the UI (but not the DB). As long as NULL values are acceptable,
> but a constraint is placed on the uniqueness, I think this should be
> fine.

The codes are only really required when integrating between systems.
Then we match on the code (rather than the name). Unless you are
specifically configuring such a scenario, or unless there are
authoritative codes to use such as from a MFL, your average user could
and should be happy to leave them blank. Of course the problem arises
where a user is presented with a blank field and feels naturally
compelled to enter something. We'll have to deal with that but
perhaps not urgently.

I guess I am not really understanding in the MFL scenario why you should
need to match on a code instead of a name. Is it just a matter of
convenience? Why not UUID, which I suppose the code could be as well, since
it is just a text field anyway?

Yes a code is just a code so it could well be a uuid but might be
something else. The point being I guess that we often need to match
against something which someone else is providing. So code is
adaptable.

>
> Lars, I suspect some of the 1.4 import code needs to be looked at, as
> if I remember correctly, this could have been one of the reasons (at
> my insistence) that we relaxed the uniqueness constraint some time
> back, as it made imports from 1.4 databases impossible where there
> were uniqueness violations.

Should we just throw away these codes on import?

I suspect so if it is coming from 1.4 but why should we not do this for all
DXF imports? There is no guarantee the codes would be unique anyway so seems
better to import what we can and ditch the rest as outlined in the previous
mail.

I was talking of throwing away all 1.4 codes. But we certainly do
want to be able to import codes in general.

Bob

···

2011/10/5 Jason Pickering <jason.p.pickering@gmail.com>:

On Oct 5, 2011 4:33 PM, "Bob Jolliffe" <bobjolliffe@gmail.com> wrote:

2011/10/5 Jason Pickering <jason.p.pickering@gmail.com>:

Regards,
Jason

>
> Regards,
> Jason
>
>
> 2011/10/5 Bob Jolliffe <bobjolliffe@gmail.com>:
>> 2011/10/5 Jason Pickering <jason.p.pickering@gmail.com>:
>>> Is there going to be a way for the system to autogenerate a code if 1)
>>> there is a collision during import 2) if the user does not enter one
>>> (I suppose it will be enforced through the UI), but does not know what
>>> the code should be?
>>
>> My understanding is that the code must be unique but can also be null.
>> So if there is no code then there is no code.
>>
>> When importing data in Kenya with datasets from over 8000 orgunits the
>> probabliity of mismatches and duplicate orgunit codes was quite high.
>> In this case, where a duplicate is encountered *both* values are
>> deleted (and logged) as there is really no way to determine which one
>> was using the correct code.
>>
>> Regarding what to do at the UI - if there is no assigned code (eg from
>> a master facility list) then the user should simply not enter one.
>>
>>>
>>>
>>>
>>> On Wed, Oct 5, 2011 at 1:51 PM, <noreply@launchpad.net> wrote:
>>>> ------------------------------------------------------------
>>>> revno: 4838
>>>> committer: Lars Helge Overland <larshelge@gmail.com>
>>>> branch nick: dhis2
>>>> timestamp: Wed 2011-10-05 13:50:05 +0200
>>>> message:
>>>> Put code uniqueness check back in import module
>>>> modified:
>>>>
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java
>>>>
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java
>>>>
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java
>>>>
>>>> dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java
>>>>
>>>>
>>>> --
>>>> lp:dhis2
>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>
>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>> To unsubscribe from this branch go to
>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>
>>>> === modified file
>>>> 'dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java'
>>>> ---
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java
>>>> 2011-04-24 15:50:02 +0000
>>>> +++
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/DataElementImporter.java
>>>> 2011-10-05 11:50:05 +0000
>>>> @@ -99,6 +99,10 @@
>>>> {
>>>> match = dataElementService.getDataElementByShortName(
>>>> object.getShortName() );
>>>> }
>>>> + if ( match == null )
>>>> + {
>>>> + match = dataElementService.getDataElementByCode(
>>>> object.getCode() );
>>>> + }
>>>>
>>>> return match;
>>>> }
>>>>
>>>> === modified file
>>>> 'dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java'
>>>> ---
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java
>>>> 2011-09-24 11:35:11 +0000
>>>> +++
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/IndicatorImporter.java
>>>> 2011-10-05 11:50:05 +0000
>>>> @@ -100,6 +100,10 @@
>>>> {
>>>> match = indicatorService.getIndicatorByShortName(
>>>> object.getShortName() );
>>>> }
>>>> + if ( match == null )
>>>> + {
>>>> + match = indicatorService.getIndicatorByCode(
>>>> object.getCode() );
>>>> + }
>>>>
>>>> return match;
>>>> }
>>>>
>>>> === modified file
>>>> 'dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java'
>>>> ---
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java
>>>> 2011-04-22 21:04:14 +0000
>>>> +++
>>>> dhis-2/dhis-services/dhis-service-importexport/src/main/java/org/hisp/dhis/importexport/importer/OrganisationUnitImporter.java
>>>> 2011-10-05 11:50:05 +0000
>>>> @@ -92,7 +92,14 @@
>>>> @Override
>>>> protected OrganisationUnit getMatching( OrganisationUnit object )
>>>> {
>>>> - return organisationUnitService.getOrganisationUnitByName(
>>>> object.getName() );
>>>> + OrganisationUnit match =
>>>> organisationUnitService.getOrganisationUnitByName( object.getName() );
>>>> +
>>>> + if ( match == null )
>>>> + {
>>>> + match =
>>>> organisationUnitService.getOrganisationUnitByCode( object.getCode() );
>>>> + }
>>>> +
>>>> + return match;
>>>> }
>>>>
>>>> @Override
>>>>
>>>> === modified file
>>>> 'dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java'
>>>> ---
>>>> dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java
>>>> 2011-09-23 18:54:49 +0000
>>>> +++
>>>> dhis-2/dhis-services/dhis-service-importexport/src/test/java/org/hisp/dhis/importexport/dxf2/service/DataValueSetServiceTest.java
>>>> 2011-10-05 11:50:05 +0000
>>>> @@ -63,6 +63,7 @@
>>>> import org.junit.Ignore;
>>>> import org.junit.Test;
>>>>
>>>> +@Ignore //TODO fix
>>>> public class DataValueSetServiceTest
>>>> extends DhisTest
>>>> {
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to : dhis2-devs@lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help : https://help.launchpad.net/ListHelp
>>>
>>
>