Changed CSV-Format for OrgUnits

Dear all,

according to a previous post on the users list there have obviously been changes
in the CSV-format for importing OrgUnits. Some have been documented, others not
(or might be bugs):

- feature format seems to be mandatory now. I don't know whether that makes
sense, if it does, please indicate that in the docs, too.
- If you decide to change the docs on that, please also add a column with the
technical names to be included in the header line of the csv - the names in the
example differ from the names in the format specification. (or indicate, whether
the header line is optional)
- the import summary records a funny error message. Don't know whether it's new,
but it seems to be wrong: Validation Violations:
[ValidationViolation{property='id', message='Allowed range for length [11, 11],
length is 7', value=NAIROBI}]

I could have posted this as bug-report, but I wasn't sure, whether those are
bugs or features ...

Regards,

Uwe

update inline

Uwe Wahser <uwe@wahser.de> hat am 17. Dezember 2015 um 06:23 geschrieben:

Dear all,

according to a previous post on the users list there have obviously been
changes
in the CSV-format for importing OrgUnits. Some have been documented, others
not
(or might be bugs):

- feature format seems to be mandatory now. I don't know whether that makes
sense, if it does, please indicate that in the docs, too.
- If you decide to change the docs on that, please also add a column with the
technical names to be included in the header line of the csv - the names in
the
example differ from the names in the format specification. (or indicate,
whether
the header line is optional)
- the import summary records a funny error message. Don't know whether it's
new,
but it seems to be wrong: Validation Violations:
[ValidationViolation{property='id', message='Allowed range for length [11,
11],
length is 7', value=NAIROBI}]

UPDATE: only uid's >= 11char were imported - must be a bug

···

I could have posted this as bug-report, but I wasn't sure, whether those are
bugs or features ...

Regards,

Uwe

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

I think header is completely optional (so docs should reflect that).

···

On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23 geschrieben:

Dear all,

according to a previous post on the users list there have obviously been

changes

in the CSV-format for importing OrgUnits. Some have been documented, others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column with the

technical names to be included in the header line of the csv - the names in

the

example differ from the names in the format specification. (or indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length [11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Hi,

The header row is optional - it will automatically be discarded if it’s there. CSV import follows a pre-defined list of columns - you can drop anything at the end, but must include extra commas if you have blank columns in between relevant columns.

  1. All UIDs must be 11 characters, and starting with a letter (NOT a digit).

Regards

Calle

···

On 17 December 2015 at 08:06, Knut Staring knutst@gmail.com wrote:

I think header is completely optional (so docs should reflect that).


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23 geschrieben:

Dear all,

according to a previous post on the users list there have obviously been

changes

in the CSV-format for importing OrgUnits. Some have been documented, others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column with the

technical names to be included in the header line of the csv - the names in

the

example differ from the names in the format specification. (or indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length [11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Uwe,
Thanks for the feedback. We will take a look at it.

From the docs

“DHIS 2 supports import of metadata in the CSV format. Columns which are not required can be omitted in the CSV file, but the order will be affected. If you would like to specify columns which appear late in the order but not specify columns which appear early in the order you can include empty columns (”") for them. "

The language here is not as clear as it could be. I will edit it a bit to reflect the fact that as Calle says, the header is optional and ignored. The order of the columns however is what is important. The required columns are mandatory, other columns are optional as indicated in the docs, however, if you are using columns which appear later in the order, you must include empty columns, so that the order of the columns is correct.
FeatureType should not be required, but it should be required if you choose to include coordinates. We will check on this.

It might be helpful if you could share a snippet of your CSV file with us, which might make diagnosing things a bit easier.

Regards,
Jason

···

On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

The header row is optional - it will automatically be discarded if it’s there. CSV import follows a pre-defined list of columns - you can drop anything at the end, but must include extra commas if you have blank columns in between relevant columns.

  1. All UIDs must be 11 characters, and starting with a letter (NOT a digit).

Regards

Calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 17 December 2015 at 08:06, Knut Staring knutst@gmail.com wrote:

I think header is completely optional (so docs should reflect that).


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23 geschrieben:

Dear all,

according to a previous post on the users list there have obviously been

changes

in the CSV-format for importing OrgUnits. Some have been documented, others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column with the

technical names to be included in the header line of the csv - the names in

the

example differ from the names in the format specification. (or indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length [11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Thanks for quick answers, Jason, Knut and Calle.

Unfortunately the original csv is no longer available, as I modified it a lot in
order to get my data in. I can list my steps and observations, though, if that
helps. Some things will be fine with updated documentations (header line, to
some extend the UID-problem).

Here's my log of today:

The initial file looked like the demo from the documentation, except that the
uid was filled with the county-name and the description-column was filled. This
file produced the following error in catalina.out (no error in gui, just rest of
import parameters):

* ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled task.
(TaskUtils.java [taskScheduler-5])
java.lang.NullPointerException: Name is null
        at java.lang.Enum.valueOf(Enum.java:236)
        at
org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)
        at
org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)
        at
org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)
        at
org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)
[...]

I then specified all columns as empty values and inserted values for the feature
type. DHIS2 would do the import, but reject all counties < 11char with the
posted error message.

I then filled up the field uid with spaces to complete to 11char. This worked -
all counties were imported and visible in the left hand hierarchy in the
OrgUnit-Editor and in the pivot table.

However, when I imported the next hierarchy level in the same way, DHIS2 refused
to show the newly imported OrgUnits under their respective parents (not even
under root) - in pivot tables they showed up correctly. The counties would show
a plus-sign to indicate that there is something underneath, but the wouldn't
open up when clicking on them.

I then replaced all spaces with underscore (_) in uid, code, name etc fields of
the tables organisationunit and _orgunitstructure. As an effect this let me open
all county-nodes to see the underlying orgunits - except for one: this one had a
(') in uid, code, name etc (eg. ROCK'NROLL).

I replaced this as well. Now the hierarchies seem to show up properly everywhere
(editor, pivot tables) and I can even export all orgunits via Metadata-Export
(detailed and simple). The only problem remaining is the api-export: here DHIS2
sorts the OrgUnits nicely by name and then stops the export after 49 entries
(the json-file is properly terminated with all teh brackets) - however, that's
the most important part for me as I need to match the orgunit-codes first, when
importing data into datasets via api.

Anyway, I'm giving up for today - tomorrow I might decide to use the api's to
import the orgHierarchy in the ETL process. I'll need that anyway for nightly
updates, I was just hoping to get some quick gains by doing this manually.

Lot's of words, hope someone can get clues out of this, maybe the reason lies in
the (') although all contents were encapsulated by ("") ...

Regards, Uwe

···

Jason Pickering <jason.p.pickering@gmail.com> hat am 17. Dezember 2015 um
08:01 geschrieben:

Hi Uwe,
Thanks for the feedback. We will take a look at it.

From the docs

"DHIS 2 supports import of metadata in the CSV format. Columns which are
not required can be omitted in the CSV file, but the order will be
affected. If you would like to specify columns which appear late in the
order but not specify columns which appear early in the order you can
include empty columns ("") for them. "

The language here is not as clear as it could be. I will edit it a bit to
reflect the fact that as Calle says, the header is optional and ignored.
The order of the columns however is what is important. The required columns
are mandatory, other columns are optional as indicated in the docs,
however, if you are using columns which appear later in the order, you must
include empty columns, so that the order of the columns is correct.
FeatureType should not be required, but it should be required if you choose
to include coordinates. We will check on this.

It might be helpful if you could share a snippet of your CSV file with us,
which might make diagnosing things a bit easier.

Regards,
Jason

On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg <calle.hedberg@gmail.com> > wrote:

> Hi,
>
> 1.
> The header row is optional - it will automatically be discarded if it's
> there. CSV import follows a pre-defined list of columns - you can drop
> anything at the end, but must include extra commas if you have blank
> columns in between relevant columns.
>
> 2. All UIDs must be 11 characters, and starting with a letter (NOT a
> digit).
>
> Regards
> Calle
>
> On 17 December 2015 at 08:06, Knut Staring <knutst@gmail.com> wrote:
>
>> I think header is completely optional (so docs should reflect that).
>>
>> On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser <uwe@wahser.de> wrote:
>>
>>> update inline
>>> > Uwe Wahser <uwe@wahser.de> hat am 17. Dezember 2015 um 06:23
>>> geschrieben:
>>> >
>>> >
>>> > Dear all,
>>> >
>>> > according to a previous post on the users list there have obviously
>>> been
>>> > changes
>>> > in the CSV-format for importing OrgUnits. Some have been documented,
>>> others
>>> > not
>>> > (or might be bugs):
>>> >
>>> > - feature format seems to be mandatory now. I don't know whether that
>>> makes
>>> > sense, if it does, please indicate that in the docs, too.
>>> > - If you decide to change the docs on that, please also add a column
>>> with the
>>> > technical names to be included in the header line of the csv - the
>>> names in
>>> > the
>>> > example differ from the names in the format specification. (or
>>> indicate,
>>> > whether
>>> > the header line is optional)
>>> > - the import summary records a funny error message. Don't know whether
>>> it's
>>> > new,
>>> > but it seems to be wrong: Validation Violations:
>>> > [ValidationViolation{property='id', message='Allowed range for length
>>> [11,
>>> > 11],
>>> > length is 7', value=NAIROBI}]
>>> UPDATE: only uid's >= 11char were imported - must be a bug
>>> >
>>> > I could have posted this as bug-report, but I wasn't sure, whether
>>> those are
>>> > bugs or features ...
>>> >
>>> > Regards,
>>> >
>>> > Uwe
>>> >
>>> > _______________________________________________
>>> > Mailing list: https://launchpad.net/~dhis2-users
>>> > Post to : dhis2-users@lists.launchpad.net
>>> > Unsubscribe : https://launchpad.net/~dhis2-users
>>> > More help : https://help.launchpad.net/ListHelp
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-users
>>> Post to : dhis2-users@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>> More help : https://help.launchpad.net/ListHelp
>>>
>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> Norway: +4791880522
>> Skype: knutstar
>> http://dhis2.org
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedberg@gmail.com
>
> Skype: calle_hedberg
>
> *******************************************
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Dear Uwe,

See the documentation on github.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/dhis2_developer_manual_full.html#d4934e800

Alex

···

On Thu, Dec 17, 2015 at 8:23 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

according to a previous post on the users list there have obviously been changes

in the CSV-format for importing OrgUnits. Some have been documented, others not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column with the

technical names to be included in the header line of the csv - the names in the

example differ from the names in the format specification. (or indicate, whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether it’s new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length [11, 11],

length is 7’, value=NAIROBI}]

I could have posted this as bug-report, but I wasn’t sure, whether those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda

IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya

IT Specialist (Servers, Networks and Security, Health Information Systems - DHIS2 ) & Solar Consultant

+256 774149 775, + 256 759 800161

"I don’t want to be anything other than what I have been - one tree hill "

Hi Uwe,

unfortunately the CSV metadata import docs lagged behind a bit when we forked the 2.21 docs.

If you go to the “latest build documentation” on the docs page you will the correct one. Direct link:

http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964

The reason for the “java.lang.NullPointerException: Name is null” crash is that we changed to using enums , so that several the “predefined options” changed:

Feature type
No
NONE | MULTI_POLYGON | POLYGON | POINT | SYMBOL
Geospatial feature type.

In general, uids must be 11 alphanumerics, letter first. Please have a look at that link, try again and let us know how it goes.

regards,

Lars

···

On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser uwe@wahser.de wrote:

Thanks for quick answers, Jason, Knut and Calle.

Unfortunately the original csv is no longer available, as I modified it a lot in

order to get my data in. I can list my steps and observations, though, if that

helps. Some things will be fine with updated documentations (header line, to

some extend the UID-problem).

Here’s my log of today:

The initial file looked like the demo from the documentation, except that the

uid was filled with the county-name and the description-column was filled. This

file produced the following error in catalina.out (no error in gui, just rest of

import parameters):

  • ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled task.

(TaskUtils.java [taskScheduler-5])

java.lang.NullPointerException: Name is null

    at java.lang.Enum.valueOf(Enum.java:236)

    at

org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)

    at

org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)

[…]

I then specified all columns as empty values and inserted values for the feature

type. DHIS2 would do the import, but reject all counties < 11char with the

posted error message.

I then filled up the field uid with spaces to complete to 11char. This worked -

all counties were imported and visible in the left hand hierarchy in the

OrgUnit-Editor and in the pivot table.

However, when I imported the next hierarchy level in the same way, DHIS2 refused

to show the newly imported OrgUnits under their respective parents (not even

under root) - in pivot tables they showed up correctly. The counties would show

a plus-sign to indicate that there is something underneath, but the wouldn’t

open up when clicking on them.

I then replaced all spaces with underscore (_) in uid, code, name etc fields of

the tables organisationunit and _orgunitstructure. As an effect this let me open

all county-nodes to see the underlying orgunits - except for one: this one had a

(’) in uid, code, name etc (eg. ROCK’NROLL).

I replaced this as well. Now the hierarchies seem to show up properly everywhere

(editor, pivot tables) and I can even export all orgunits via Metadata-Export

(detailed and simple). The only problem remaining is the api-export: here DHIS2

sorts the OrgUnits nicely by name and then stops the export after 49 entries

(the json-file is properly terminated with all teh brackets) - however, that’s

the most important part for me as I need to match the orgunit-codes first, when

importing data into datasets via api.

Anyway, I’m giving up for today - tomorrow I might decide to use the api’s to

import the orgHierarchy in the ETL process. I’ll need that anyway for nightly

updates, I was just hoping to get some quick gains by doing this manually.

Lot’s of words, hope someone can get clues out of this, maybe the reason lies in

the (’) although all contents were encapsulated by ("") …

Regards, Uwe

Jason Pickering jason.p.pickering@gmail.com hat am 17. Dezember 2015 um

08:01 geschrieben:

Hi Uwe,

Thanks for the feedback. We will take a look at it.

From the docs

"DHIS 2 supports import of metadata in the CSV format. Columns which are

not required can be omitted in the CSV file, but the order will be

affected. If you would like to specify columns which appear late in the

order but not specify columns which appear early in the order you can

include empty columns ("") for them. "

The language here is not as clear as it could be. I will edit it a bit to

reflect the fact that as Calle says, the header is optional and ignored.

The order of the columns however is what is important. The required columns

are mandatory, other columns are optional as indicated in the docs,

however, if you are using columns which appear later in the order, you must

include empty columns, so that the order of the columns is correct.

FeatureType should not be required, but it should be required if you choose

to include coordinates. We will check on this.

It might be helpful if you could share a snippet of your CSV file with us,

which might make diagnosing things a bit easier.

Regards,

Jason

On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg calle.hedberg@gmail.com

wrote:

Hi,

The header row is optional - it will automatically be discarded if it’s

there. CSV import follows a pre-defined list of columns - you can drop

anything at the end, but must include extra commas if you have blank

columns in between relevant columns.

  1. All UIDs must be 11 characters, and starting with a letter (NOT a

digit).

Regards

Calle

On 17 December 2015 at 08:06, Knut Staring knutst@gmail.com wrote:

I think header is completely optional (so docs should reflect that).

On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23

geschrieben:

Dear all,

according to a previous post on the users list there have obviously

been

changes

in the CSV-format for importing OrgUnits. Some have been documented,

others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that

makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column

with the

technical names to be included in the header line of the csv - the

names in

the

example differ from the names in the format specification. (or

indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether

it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length

[11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether

those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

1 Like

These changes have been backported to 2.21 docs now.

https://www.dhis2.org/documentation

···

On Fri, Dec 18, 2015 at 10:37 AM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Uwe,

unfortunately the CSV metadata import docs lagged behind a bit when we forked the 2.21 docs.

If you go to the “latest build documentation” on the docs page you will the correct one. Direct link:


http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964

The reason for the “java.lang.NullPointerException: Name is null” crash is that we changed to using enums , so that several the “predefined options” changed:

Feature type
No
NONE | MULTI_POLYGON | POLYGON | POINT | SYMBOL
Geospatial feature type.

In general, uids must be 11 alphanumerics, letter first. Please have a look at that link, try again and let us know how it goes.

regards,

Lars

On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser uwe@wahser.de wrote:

Thanks for quick answers, Jason, Knut and Calle.

Unfortunately the original csv is no longer available, as I modified it a lot in

order to get my data in. I can list my steps and observations, though, if that

helps. Some things will be fine with updated documentations (header line, to

some extend the UID-problem).

Here’s my log of today:

The initial file looked like the demo from the documentation, except that the

uid was filled with the county-name and the description-column was filled. This

file produced the following error in catalina.out (no error in gui, just rest of

import parameters):

  • ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled task.

(TaskUtils.java [taskScheduler-5])

java.lang.NullPointerException: Name is null

    at java.lang.Enum.valueOf(Enum.java:236)

    at

org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)

    at

org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)

[…]

I then specified all columns as empty values and inserted values for the feature

type. DHIS2 would do the import, but reject all counties < 11char with the

posted error message.

I then filled up the field uid with spaces to complete to 11char. This worked -

all counties were imported and visible in the left hand hierarchy in the

OrgUnit-Editor and in the pivot table.

However, when I imported the next hierarchy level in the same way, DHIS2 refused

to show the newly imported OrgUnits under their respective parents (not even

under root) - in pivot tables they showed up correctly. The counties would show

a plus-sign to indicate that there is something underneath, but the wouldn’t

open up when clicking on them.

I then replaced all spaces with underscore (_) in uid, code, name etc fields of

the tables organisationunit and _orgunitstructure. As an effect this let me open

all county-nodes to see the underlying orgunits - except for one: this one had a

(’) in uid, code, name etc (eg. ROCK’NROLL).

I replaced this as well. Now the hierarchies seem to show up properly everywhere

(editor, pivot tables) and I can even export all orgunits via Metadata-Export

(detailed and simple). The only problem remaining is the api-export: here DHIS2

sorts the OrgUnits nicely by name and then stops the export after 49 entries

(the json-file is properly terminated with all teh brackets) - however, that’s

the most important part for me as I need to match the orgunit-codes first, when

importing data into datasets via api.

Anyway, I’m giving up for today - tomorrow I might decide to use the api’s to

import the orgHierarchy in the ETL process. I’ll need that anyway for nightly

updates, I was just hoping to get some quick gains by doing this manually.

Lot’s of words, hope someone can get clues out of this, maybe the reason lies in

the (’) although all contents were encapsulated by ("") …

Regards, Uwe

Jason Pickering jason.p.pickering@gmail.com hat am 17. Dezember 2015 um

08:01 geschrieben:

Hi Uwe,

Thanks for the feedback. We will take a look at it.

From the docs

"DHIS 2 supports import of metadata in the CSV format. Columns which are

not required can be omitted in the CSV file, but the order will be

affected. If you would like to specify columns which appear late in the

order but not specify columns which appear early in the order you can

include empty columns ("") for them. "

The language here is not as clear as it could be. I will edit it a bit to

reflect the fact that as Calle says, the header is optional and ignored.

The order of the columns however is what is important. The required columns

are mandatory, other columns are optional as indicated in the docs,

however, if you are using columns which appear later in the order, you must

include empty columns, so that the order of the columns is correct.

FeatureType should not be required, but it should be required if you choose

to include coordinates. We will check on this.

It might be helpful if you could share a snippet of your CSV file with us,

which might make diagnosing things a bit easier.

Regards,

Jason

On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg calle.hedberg@gmail.com

wrote:

Hi,

The header row is optional - it will automatically be discarded if it’s

there. CSV import follows a pre-defined list of columns - you can drop

anything at the end, but must include extra commas if you have blank

columns in between relevant columns.

  1. All UIDs must be 11 characters, and starting with a letter (NOT a

digit).

Regards

Calle

On 17 December 2015 at 08:06, Knut Staring knutst@gmail.com wrote:

I think header is completely optional (so docs should reflect that).

On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23

geschrieben:

Dear all,

according to a previous post on the users list there have obviously

been

changes

in the CSV-format for importing OrgUnits. Some have been documented,

others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether that

makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a column

with the

technical names to be included in the header line of the csv - the

names in

the

example differ from the names in the format specification. (or

indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know whether

it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for length

[11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether

those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

1 Like

Hi Lars,

I did some more testing on this:

1) featuretype: I managed to upload the minimized demo from the documentation to
the dhis2-demo server. On my server I got the same error message as before, so
I'll just try to upload a new build of 2.21 and hopefully I'll be fine.

2) apostrophe in uid: I loaded the attached test-file into the demo
installation: Test-province and Toast-province with one child respectively.
Test-province received an uid with apostrophe: ROCKN'ROLL. The effect is, that
you can open Toast in the editor, but not Test although the plus sign indicates
children (see screenshot). Equally the api only returns a few organizations -
alphabetically sorted - up to Baoma MCHP
(https://play.dhis2.org/demo/api/organisationUnits.json).

I didn't do another test for spaces, but I assume that the uid can only digest
[A..Za..z0..9] without causing hick-ups elsewhere, although other characters get
imported without an error message. I propose to specify this in the manual and
to do some checks on valid characters when importing.

Sorry for not updating the docu myself, but I still feel shy :wink:

Regards,

Uwe

sampleunits_demo_apostroph.csv (231 Bytes)

image

···

Lars Helge Øverland <larshelge@gmail.com> hat am 18. Dezember 2015 um 10:37
geschrieben:

Hi Uwe,

unfortunately the CSV metadata import docs lagged behind a bit when we
forked the 2.21 docs.

If you go to the "latest build documentation" on the docs page you will the
correct one. Direct link:

<goog_851896030>
http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964

The reason for the "java.lang.NullPointerException: Name is null" crash is
that we changed to using enums
<https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html> , so that
several the "predefined options" changed:

Feature typeNoNONE | MULTI_POLYGON | POLYGON | POINT | SYMBOLGeospatial
feature type.

In general, uids must be 11 alphanumerics, letter first. Please have a look
at that link, try again and let us know how it goes.

regards,

Lars

On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser <uwe@wahser.de> wrote:

> Thanks for quick answers, Jason, Knut and Calle.
>
> Unfortunately the original csv is no longer available, as I modified it a
> lot in
> order to get my data in. I can list my steps and observations, though, if
> that
> helps. Some things will be fine with updated documentations (header line,
> to
> some extend the UID-problem).
>
> Here's my log of today:
>
> The initial file looked like the demo from the documentation, except that
> the
> uid was filled with the county-name and the description-column was filled.
> This
> file produced the following error in catalina.out (no error in gui, just
> rest of
> import parameters):
>
> * ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled
> task.
> (TaskUtils.java [taskScheduler-5])
> java.lang.NullPointerException: Name is null
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)
> at
>
> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)
> at
>
> org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)
> at
>
> org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)
> [...]
>
> I then specified all columns as empty values and inserted values for the
> feature
> type. DHIS2 would do the import, but reject all counties < 11char with the
> posted error message.
>
> I then filled up the field uid with spaces to complete to 11char. This
> worked -
> all counties were imported and visible in the left hand hierarchy in the
> OrgUnit-Editor and in the pivot table.
>
> However, when I imported the next hierarchy level in the same way, DHIS2
> refused
> to show the newly imported OrgUnits under their respective parents (not
> even
> under root) - in pivot tables they showed up correctly. The counties would
> show
> a plus-sign to indicate that there is something underneath, but the
> wouldn't
> open up when clicking on them.
>
> I then replaced all spaces with underscore (_) in uid, code, name etc
> fields of
> the tables organisationunit and _orgunitstructure. As an effect this let
> me open
> all county-nodes to see the underlying orgunits - except for one: this one
> had a
> (') in uid, code, name etc (eg. ROCK'NROLL).
>
> I replaced this as well. Now the hierarchies seem to show up properly
> everywhere
> (editor, pivot tables) and I can even export all orgunits via
> Metadata-Export
> (detailed and simple). The only problem remaining is the api-export: here
> DHIS2
> sorts the OrgUnits nicely by name and then stops the export after 49
> entries
> (the json-file is properly terminated with all teh brackets) - however,
> that's
> the most important part for me as I need to match the orgunit-codes first,
> when
> importing data into datasets via api.
>
> Anyway, I'm giving up for today - tomorrow I might decide to use the api's
> to
> import the orgHierarchy in the ETL process. I'll need that anyway for
> nightly
> updates, I was just hoping to get some quick gains by doing this manually.
>
> Lot's of words, hope someone can get clues out of this, maybe the reason
> lies in
> the (') although all contents were encapsulated by ("") ...
>
> Regards, Uwe
>
> > Jason Pickering <jason.p.pickering@gmail.com> hat am 17. Dezember 2015
> um
> > 08:01 geschrieben:
> >
> >
> > Hi Uwe,
> > Thanks for the feedback. We will take a look at it.
> >
> > From the docs
> >
> > "DHIS 2 supports import of metadata in the CSV format. Columns which are
> > not required can be omitted in the CSV file, but the order will be
> > affected. If you would like to specify columns which appear late in the
> > order but not specify columns which appear early in the order you can
> > include empty columns ("") for them. "
> >
> > The language here is not as clear as it could be. I will edit it a bit to
> > reflect the fact that as Calle says, the header is optional and ignored.
> > The order of the columns however is what is important. The required
> columns
> > are mandatory, other columns are optional as indicated in the docs,
> > however, if you are using columns which appear later in the order, you
> must
> > include empty columns, so that the order of the columns is correct.
> > FeatureType should not be required, but it should be required if you
> choose
> > to include coordinates. We will check on this.
> >
> > It might be helpful if you could share a snippet of your CSV file with
> us,
> > which might make diagnosing things a bit easier.
> >
> > Regards,
> > Jason
> >
> >
> > On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg <calle.hedberg@gmail.com> > > > wrote:
> >
> > > Hi,
> > >
> > > 1.
> > > The header row is optional - it will automatically be discarded if it's
> > > there. CSV import follows a pre-defined list of columns - you can drop
> > > anything at the end, but must include extra commas if you have blank
> > > columns in between relevant columns.
> > >
> > > 2. All UIDs must be 11 characters, and starting with a letter (NOT a
> > > digit).
> > >
> > > Regards
> > > Calle
> > >
> > > On 17 December 2015 at 08:06, Knut Staring <knutst@gmail.com> wrote:
> > >
> > >> I think header is completely optional (so docs should reflect that).
> > >>
> > >> On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser <uwe@wahser.de> wrote:
> > >>
> > >>> update inline
> > >>> > Uwe Wahser <uwe@wahser.de> hat am 17. Dezember 2015 um 06:23
> > >>> geschrieben:
> > >>> >
> > >>> >
> > >>> > Dear all,
> > >>> >
> > >>> > according to a previous post on the users list there have obviously
> > >>> been
> > >>> > changes
> > >>> > in the CSV-format for importing OrgUnits. Some have been
> documented,
> > >>> others
> > >>> > not
> > >>> > (or might be bugs):
> > >>> >
> > >>> > - feature format seems to be mandatory now. I don't know whether
> that
> > >>> makes
> > >>> > sense, if it does, please indicate that in the docs, too.
> > >>> > - If you decide to change the docs on that, please also add a
> column
> > >>> with the
> > >>> > technical names to be included in the header line of the csv - the
> > >>> names in
> > >>> > the
> > >>> > example differ from the names in the format specification. (or
> > >>> indicate,
> > >>> > whether
> > >>> > the header line is optional)
> > >>> > - the import summary records a funny error message. Don't know
> whether
> > >>> it's
> > >>> > new,
> > >>> > but it seems to be wrong: Validation Violations:
> > >>> > [ValidationViolation{property='id', message='Allowed range for
> length
> > >>> [11,
> > >>> > 11],
> > >>> > length is 7', value=NAIROBI}]
> > >>> UPDATE: only uid's >= 11char were imported - must be a bug
> > >>> >
> > >>> > I could have posted this as bug-report, but I wasn't sure, whether
> > >>> those are
> > >>> > bugs or features ...
> > >>> >
> > >>> > Regards,
> > >>> >
> > >>> > Uwe
> > >>> >
> > >>> > _______________________________________________
> > >>> > Mailing list: https://launchpad.net/~dhis2-users
> > >>> > Post to : dhis2-users@lists.launchpad.net
> > >>> > Unsubscribe : https://launchpad.net/~dhis2-users
> > >>> > More help : https://help.launchpad.net/ListHelp
> > >>>
> > >>> _______________________________________________
> > >>> Mailing list: https://launchpad.net/~dhis2-users
> > >>> Post to : dhis2-users@lists.launchpad.net
> > >>> Unsubscribe : https://launchpad.net/~dhis2-users
> > >>> More help : https://help.launchpad.net/ListHelp
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Knut Staring
> > >> Dept. of Informatics, University of Oslo
> > >> Norway: +4791880522
> > >> Skype: knutstar
> > >> http://dhis2.org
> > >>
> > >> _______________________________________________
> > >> Mailing list: https://launchpad.net/~dhis2-devs
> > >> Post to : dhis2-devs@lists.launchpad.net
> > >> Unsubscribe : https://launchpad.net/~dhis2-devs
> > >> More help : https://help.launchpad.net/ListHelp
> > >>
> > >>
> > >
> > >
> > > --
> > >
> > > *******************************************
> > >
> > > Calle Hedberg
> > >
> > > 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
> > >
> > > Tel/fax (home): +27-21-685-6472
> > >
> > > Cell: +27-82-853-5352
> > >
> > > Iridium SatPhone: +8816-315-19119
> > >
> > > Email: calle.hedberg@gmail.com
> > >
> > > Skype: calle_hedberg
> > >
> > > *******************************************
> > >
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~dhis2-devs
> > > Post to : dhis2-devs@lists.launchpad.net
> > > Unsubscribe : https://launchpad.net/~dhis2-devs
> > > More help : https://help.launchpad.net/ListHelp
> > >
> > >
> >
> >
> > --
> > Jason P. Pickering
> > email: jason.p.pickering@gmail.com
> > tel:+46764147049
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help : https://help.launchpad.net/ListHelp
>

--
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org <https://www.dhis2.org>

1 Like

Okay. In any case, UIDs cannot have apostrophes as its [0-9a-zA-Z] so I wouldn’t spend too much time on that.

Yes we did a fix at some point for the enum stuff so try again with latest 2.21.

Lars

···

On Fri, Dec 18, 2015 at 12:42 PM, Uwe Wahser uwe@wahser.de wrote:

Hi Lars,

I did some more testing on this:

  1. featuretype: I managed to upload the minimized demo from the documentation to

the dhis2-demo server. On my server I got the same error message as before, so

I’ll just try to upload a new build of 2.21 and hopefully I’ll be fine.

  1. apostrophe in uid: I loaded the attached test-file into the demo

installation: Test-province and Toast-province with one child respectively.

Test-province received an uid with apostrophe: ROCKN’ROLL. The effect is, that

you can open Toast in the editor, but not Test although the plus sign indicates

children (see screenshot). Equally the api only returns a few organizations -

alphabetically sorted - up to Baoma MCHP

(https://play.dhis2.org/demo/api/organisationUnits.json).

I didn’t do another test for spaces, but I assume that the uid can only digest

[A…Za…z0…9] without causing hick-ups elsewhere, although other characters get

imported without an error message. I propose to specify this in the manual and

to do some checks on valid characters when importing.

Sorry for not updating the docu myself, but I still feel shy :wink:

Regards,

Uwe

Lars Helge Øverland larshelge@gmail.com hat am 18. Dezember 2015 um 10:37

geschrieben:

Hi Uwe,

unfortunately the CSV metadata import docs lagged behind a bit when we

forked the 2.21 docs.

If you go to the “latest build documentation” on the docs page you will the

correct one. Direct link:

<goog_851896030>

http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s10.html#d5e964

The reason for the “java.lang.NullPointerException: Name is null” crash is

that we changed to using enums

<https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html> , so that
several the “predefined options” changed:

Feature typeNoNONE | MULTI_POLYGON | POLYGON | POINT | SYMBOLGeospatial

feature type.

In general, uids must be 11 alphanumerics, letter first. Please have a look

at that link, try again and let us know how it goes.

regards,

Lars

On Thu, Dec 17, 2015 at 2:59 PM, Uwe Wahser uwe@wahser.de wrote:

Thanks for quick answers, Jason, Knut and Calle.

Unfortunately the original csv is no longer available, as I modified it a

lot in

order to get my data in. I can list my steps and observations, though, if

that

helps. Some things will be fine with updated documentations (header line,

to

some extend the UID-problem).

Here’s my log of today:

The initial file looked like the demo from the documentation, except that

the

uid was filled with the county-name and the description-column was filled.

This

file produced the following error in catalina.out (no error in gui, just

rest of

import parameters):

  • ERROR 2015-12-17 09:12:04,879 Unexpected error occurred in scheduled

task.

(TaskUtils.java [taskScheduler-5])

java.lang.NullPointerException: Name is null

    at java.lang.Enum.valueOf(Enum.java:236)
    at

org.hisp.dhis.organisationunit.FeatureType.valueOf(FeatureType.java:37)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.organisationUnitsFromCsv(DefaultCsvImportService.java:324)

    at

org.hisp.dhis.dxf2.csv.DefaultCsvImportService.fromCsv(DefaultCsvImportService.java:116)

    at

org.hisp.dhis.importexport.action.util.ImportMetaDataCsvTask.call(ImportMetaDataCsvTask.java:92)

[…]

I then specified all columns as empty values and inserted values for the

feature

type. DHIS2 would do the import, but reject all counties < 11char with the

posted error message.

I then filled up the field uid with spaces to complete to 11char. This

worked -

all counties were imported and visible in the left hand hierarchy in the

OrgUnit-Editor and in the pivot table.

However, when I imported the next hierarchy level in the same way, DHIS2

refused

to show the newly imported OrgUnits under their respective parents (not

even

under root) - in pivot tables they showed up correctly. The counties would

show

a plus-sign to indicate that there is something underneath, but the

wouldn’t

open up when clicking on them.

I then replaced all spaces with underscore (_) in uid, code, name etc

fields of

the tables organisationunit and _orgunitstructure. As an effect this let

me open

all county-nodes to see the underlying orgunits - except for one: this one

had a

(’) in uid, code, name etc (eg. ROCK’NROLL).

I replaced this as well. Now the hierarchies seem to show up properly

everywhere

(editor, pivot tables) and I can even export all orgunits via

Metadata-Export

(detailed and simple). The only problem remaining is the api-export: here

DHIS2

sorts the OrgUnits nicely by name and then stops the export after 49

entries

(the json-file is properly terminated with all teh brackets) - however,

that’s

the most important part for me as I need to match the orgunit-codes first,

when

importing data into datasets via api.

Anyway, I’m giving up for today - tomorrow I might decide to use the api’s

to

import the orgHierarchy in the ETL process. I’ll need that anyway for

nightly

updates, I was just hoping to get some quick gains by doing this manually.

Lot’s of words, hope someone can get clues out of this, maybe the reason

lies in

the (’) although all contents were encapsulated by ("") …

Regards, Uwe

Jason Pickering jason.p.pickering@gmail.com hat am 17. Dezember 2015

um

08:01 geschrieben:

Hi Uwe,

Thanks for the feedback. We will take a look at it.

From the docs

"DHIS 2 supports import of metadata in the CSV format. Columns which are

not required can be omitted in the CSV file, but the order will be

affected. If you would like to specify columns which appear late in the

order but not specify columns which appear early in the order you can

include empty columns ("") for them. "

The language here is not as clear as it could be. I will edit it a bit to

reflect the fact that as Calle says, the header is optional and ignored.

The order of the columns however is what is important. The required

columns

are mandatory, other columns are optional as indicated in the docs,

however, if you are using columns which appear later in the order, you

must

include empty columns, so that the order of the columns is correct.

FeatureType should not be required, but it should be required if you

choose

to include coordinates. We will check on this.

It might be helpful if you could share a snippet of your CSV file with

us,

which might make diagnosing things a bit easier.

Regards,

Jason

On Thu, Dec 17, 2015 at 7:22 AM, Calle Hedberg calle.hedberg@gmail.com

wrote:

Hi,

The header row is optional - it will automatically be discarded if it’s

there. CSV import follows a pre-defined list of columns - you can drop

anything at the end, but must include extra commas if you have blank

columns in between relevant columns.

  1. All UIDs must be 11 characters, and starting with a letter (NOT a

digit).

Regards

Calle

On 17 December 2015 at 08:06, Knut Staring knutst@gmail.com wrote:

I think header is completely optional (so docs should reflect that).

On Thu, Dec 17, 2015 at 6:28 AM, Uwe Wahser uwe@wahser.de wrote:

update inline

Uwe Wahser uwe@wahser.de hat am 17. Dezember 2015 um 06:23

geschrieben:

Dear all,

according to a previous post on the users list there have obviously

been

changes

in the CSV-format for importing OrgUnits. Some have been

documented,

others

not

(or might be bugs):

  • feature format seems to be mandatory now. I don’t know whether

that

makes

sense, if it does, please indicate that in the docs, too.

  • If you decide to change the docs on that, please also add a

column

with the

technical names to be included in the header line of the csv - the

names in

the

example differ from the names in the format specification. (or

indicate,

whether

the header line is optional)

  • the import summary records a funny error message. Don’t know

whether

it’s

new,

but it seems to be wrong: Validation Violations:

[ValidationViolation{property=‘id’, message='Allowed range for

length

[11,

11],

length is 7’, value=NAIROBI}]

UPDATE: only uid’s >= 11char were imported - must be a bug

I could have posted this as bug-report, but I wasn’t sure, whether

those are

bugs or features …

Regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org <https://www.dhis2.org>

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org

1 Like