Interoperability question

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  2.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio
Technical Manager
Skype: paulojrgracio

Critical Software Mozambique
Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company
CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45
Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.

DataValues.xls (17.5 KB)

ExcelConverterBean.java (2.44 KB)

XLSDataIn.java (2.6 KB)

Hi Paulo,

Unfortunately, I think the experts on this particular feature are on holiday right at this moment, but hopefully you will get some help soon.

I just wanted to ask if this is a one time import or something that will happen regularly. The reason I’m asking is that there is work well under way right now to develop a module (using the new App framework) for importing data from Excel. It is still a bit rough (and thus not released yet), but it could possibly be of interest to you to experiment a bit with it.

Knut

···

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Knut Staring

Dept. of Informatics, University of Oslo

+4791880522

http://dhis2.org

Hi Knut,

What we are trying to achieve with this is a solution for manually import of data collected using legacy systems that only export data to excel. We can implement an adapter that translates data from excel to SDMX-HD or DXF2, but this is one more step in the process and another software to give maintenance/support.

I was also digging around the new App Framework, suggestion from Pascal Brandt that attended your training session in Uganda. If possible , we would like to do a test drive of this new App and see if it can fit our requirements.

Kind regards,

Paulo Grácio

···

From: Knut Staring [mailto:knutst@gmail.com]
Sent: quarta-feira, 31 de Julho de 2013 19:09
To: Paulo Grácio; Namrata Nehete
Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

Unfortunately, I think the experts on this particular feature are on holiday right at this moment, but hopefully you will get some help soon.

I just wanted to ask if this is a one time import or something that will happen regularly. The reason I’m asking is that there is work well under way right now to develop a module (using the new App framework) for importing data from Excel. It is still a bit rough (and thus not released yet), but it could possibly be of interest to you to experiment a bit with it.

Knut

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  2.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio
Technical Manager
Skype: paulojrgracio

Critical Software Mozambique
Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company
CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45
Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

+4791880522

http://dhis2.org

Hi Paulo,
I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

···

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

···

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]
Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio
Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  2.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio
Technical Manager
Skype: paulojrgracio

Critical Software Mozambique
Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company
CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45
Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine
    The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API? Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

Thanks for your help.

Regards,

Pascal

···

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
E-mail: pascal@jembi.org*

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

···

From: Pascal Brandt [mailto:pascal@jembi.org]
Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan
Crichton
Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is
working on.

What we're looking for is a way to automatically import data into the DHIS.
It's not a once-off import, but it's also not transactional (DHIS2 forms
aren't being completed on a regular basis). For a once-off, we could use CSV
or DXF or one of the other options via the web interface and for
transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see
the following options:

1. Sending data values using SDMX-HD
<http://www.dhis2.org/doc/snapshot/en/user/html/ch25s08.html&gt;
2. Sending large bulks of data values
<http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html&gt;
3. Integration Engine
<http://www.dhis2.org/doc/snapshot/en/user/html/ch26.html&gt;

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it
automatically loaded into DHIS2 the best solution might be to implement an
adapter that translates data to SDMX-HD before send it to DHIS2. Using this
approach you also isolate your application specific data format from other
systems, communication is always done using standard format.

The main point is that needs to be automatic (no user interaction with the
UI) and we need to be able to initiate the process on an ad-hoc bases. Is
there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of
DHIS2 User Manual.

<DHIS2 App Hub;
DHIS2 App Hub

Also, I'm having some trouble finding the documentation for the DXF format,
is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try
and provide something. Our api is very much a moving target though, ane
because of some design choices we cannot just generate this.”

<https://bugs.launchpad.net/dhis2/+bug/990783&gt;
https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio <pgracio@criticalsoftware.com> wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from
a legacy system into dhis2. Once this system only generates excel files as
output we would like to import the generated file directly(without
modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are
considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]
Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

···

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason

···

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Hi,

We have at least one use case where we’ll be pushing data, so it sounds like posting SDMX-HD or DXF to the dataValueSets resource is the right option for us.

Thanks!

Pascal

···

On 6 August 2013 16:02, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Hi Pascal

I think posting dxf to the dataValueSets resource is the simplest option - as described at http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html.

While SDMX-HD is still lying unmaintained there is little point in wasting excessive effort on it, unless it is of course to revamp the process, but that is not today’s proposition.

In most cases I prefer to use codes rather than uids as the resulting datavaluesets are a bit easier to read and debug and it is reasonably possible that you might have the same codes (particularly for facilities) in both systems. In which case you use the dataElementIdScheme=“code” and orgUnitIdScheme=“code” attributes as described at the bottom of http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html.

Regards

Bob

···

On 6 August 2013 15:12, Pascal Brandt pascal@jembi.org wrote:

Pascal

Hi,

We have at least one use case where we’ll be pushing data, so it sounds like posting SDMX-HD or DXF to the dataValueSets resource is the right option for us.

Thanks!


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 16:02, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Thanks Bob, we appreciate your input on this.

Ciao,

Pascal

···

On 6 August 2013 17:11, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Pascal

I think posting dxf to the dataValueSets resource is the simplest option - as described at http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html.

While SDMX-HD is still lying unmaintained there is little point in wasting excessive effort on it, unless it is of course to revamp the process, but that is not today’s proposition.

In most cases I prefer to use codes rather than uids as the resulting datavaluesets are a bit easier to read and debug and it is reasonably possible that you might have the same codes (particularly for facilities) in both systems. In which case you use the dataElementIdScheme=“code” and orgUnitIdScheme=“code” attributes as described at the bottom of http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html.

Regards

Bob


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
E-mail: pascal@jembi.org*

On 6 August 2013 15:12, Pascal Brandt pascal@jembi.org wrote:

Pascal

Hi,

We have at least one use case where we’ll be pushing data, so it sounds like posting SDMX-HD or DXF to the dataValueSets resource is the right option for us.

Thanks!


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 16:02, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Agree with what is being said here - from the description it seems producing a DXF file and post it to the web api would be easiest.

http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html

Lars

···

On Tue, Aug 6, 2013 at 5:14 PM, Pascal Brandt pascal@jembi.org wrote:

Thanks Bob, we appreciate your input on this.

Ciao,

Pascal


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 17:11, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Pascal

I think posting dxf to the dataValueSets resource is the simplest option - as described at http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html.

While SDMX-HD is still lying unmaintained there is little point in wasting excessive effort on it, unless it is of course to revamp the process, but that is not today’s proposition.

In most cases I prefer to use codes rather than uids as the resulting datavaluesets are a bit easier to read and debug and it is reasonably possible that you might have the same codes (particularly for facilities) in both systems. In which case you use the dataElementIdScheme=“code” and orgUnitIdScheme=“code” attributes as described at the bottom of http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html.

Regards

Bob


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On 6 August 2013 15:12, Pascal Brandt pascal@jembi.org wrote:

Pascal

Hi,

We have at least one use case where we’ll be pushing data, so it sounds like posting SDMX-HD or DXF to the dataValueSets resource is the right option for us.

Thanks!


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 16:02, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

Thanks Lars.

···

On 8 August 2013 11:18, Lars Helge Øverland larshelge@gmail.com wrote:

Agree with what is being said here - from the description it seems producing a DXF file and post it to the web api would be easiest.

http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html

Lars


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 5:14 PM, Pascal Brandt pascal@jembi.org wrote:

Thanks Bob, we appreciate your input on this.

Ciao,

Pascal


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 17:11, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Pascal

I think posting dxf to the dataValueSets resource is the simplest option - as described at http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html.

While SDMX-HD is still lying unmaintained there is little point in wasting excessive effort on it, unless it is of course to revamp the process, but that is not today’s proposition.

In most cases I prefer to use codes rather than uids as the resulting datavaluesets are a bit easier to read and debug and it is reasonably possible that you might have the same codes (particularly for facilities) in both systems. In which case you use the dataElementIdScheme=“code” and orgUnitIdScheme=“code” attributes as described at the bottom of http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html.

Regards

Bob


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On 6 August 2013 15:12, Pascal Brandt pascal@jembi.org wrote:

Pascal

Hi,

We have at least one use case where we’ll be pushing data, so it sounds like posting SDMX-HD or DXF to the dataValueSets resource is the right option for us.

Thanks!


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 6 August 2013 16:02, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Pascal,

I suppose this really depends on your scenario. The integration engine is useful for the case you describe, in that it can be scheduled to run on a periodic (but regular) basis from within DHIS2. If you need to run things on an ad-hoc basis, it may also be possible to manually trigger the engine, but I am not sure about this. Bob Joliffe would know best.

You could also push data to DHIS2 on a periodic basis as described in (2) of your mail. This is using the API of DHIS2. DXF is the internal format of DHIS2, while SDMX-HD is a more neutral “standard”. So, if your application can already “talk” SDMX-HD, then there is no need to develop an exporter to DXF.

(1) describes a cross transformation, which will transform SDMX-HD XML to something which DHIS2 can understand. DXF can of course be imported natively by DHIS2. This is essentially what is described here, where a custom XSL is used to transform SDMX-HD.

So, there are many ways, but it really depends on how you plan to do it.

As for the documentation on DXF, there is not really anything at this point, other than the stuff in the manual as far as I know (but contributions are always welcome) :slight_smile:

So, it really comes down to whether you are going to pull or push. If you are pulling data from your legacy app, you will need to use the integration engine if you are doing it on a periodic basis. If you want to import your applications XML, you would need to develop an XSL to transform it, similar to the way it is done with SDMX-HD. Otherwise, you alter your legacy application to export to a format which DHIS2 understands (like DXF or CSV) and you push this to DHIS2.

Thats my understanding anyway, but others may have better info.

Regards,

Jason


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*

On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt pascal@jembi.org wrote:

Hi,

Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets (DXF) options, but am still interested to hear what the DHIS2 devs think.

Ciao,

Pascal

On 6 August 2013 15:45, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

Please find my comments below.

Regards,

Paulo Grácio

From: Pascal Brandt [mailto:pascal@jembi.org]

Sent: terça-feira, 6 de Agosto de 2013 14:56
To: Paulo Grácio
Cc: Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl Fourie; Ryan Crichton

Subject: Re: [Dhis2-devs] Interoperability question

Hi all,

This thread is important for at least two separate projects that Jembi is working on.

What we’re looking for is a way to automatically import data into the DHIS. It’s not a once-off import, but it’s also not transactional (DHIS2 forms aren’t being completed on a regular basis). For a once-off, we could use CSV or DXF or one of the other options via the web interface and for transactional we could use the Web API.

What would be the recommended way to do this periodic bulk data load? I see the following options:

  1. Sending data values using SDMX-HD
  2. Sending large bulks of data values
  3. Integration Engine

[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To have it automatically loaded into DHIS2 the best solution might be to implement an adapter that translates data to SDMX-HD before send it to DHIS2. Using this approach you also isolate your application specific data format from other systems, communication is always done using standard format. ** * ***

The main point is that needs to be automatic (no user interaction with the UI) and we need to be able to initiate the process on an ad-hoc bases. Is there a way to load CSV or DXF using the Web API?

[Paulo Grácio] Please have a look at section 25.6. Sending data values of DHIS2 User Manual.

http://apps.dhis2.org/demo/api/dataValueSets


Also, I’m having some trouble finding the documentation for the DXF format, is it available somewhere?

[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we will try and provide something. Our api is very much a moving target though, ane because of some design choices we cannot just generate this.”

https://bugs.launchpad.net/dhis2/+bug/990783

Thanks for your help.

Regards,

Pascal

On 1 August 2013 18:50, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi Jason,

Basically what we are trying to achieved is a way to simple export data from a legacy system into dhis2. Once this system only generates excel files as output we would like to import the generated file directly(without modifications) to DHIS2 and have an import summary.

We know that we can do it using several different approaches and we are considering them J.

Regards,

Paulo Grácio

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: quinta-feira, 1 de Agosto de 2013 13:26
To: Paulo Grácio

Cc: DHIS 2 developers
Subject: Re: [Dhis2-devs] Interoperability question

Hi Paulo,

I am not sure it would be worth it, but looks like it might work. Why can’t you just simply export CSV instead? Seems that given the structure of your data, you could just simply use the CSV importer instead of setting up an integration route?

Regards,

Jason

On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio pgracio@criticalsoftware.com wrote:

Hi all,

We are wondering about the best form to import in DHIS2 data from legacy applications.

In chapter “26.4. Transforming data - a Java route” of the User Manual we see that there is a way to do this integration using Java routes, but we are having some difficulties in implementing a proof of concept.

In our PoC scenario we have a XLS file sent by an external application (DataValues.xls in attach). Our idea is to create a Java route in order to import all data values that exist in that file.

Following the tutorial “Customer 3: Excel via e-mail” presented in the Apache Camel page - http://camel.apache.org/tutorial-business-partners.html - we have been able to transform the XLS file in a list of DataValue objects.

Basically, we only had to create a XLSDataIn route and a ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of an XLS file and calls the ExcelConverterBean. The ExcelConverterBean iterates through the file, and for each data value row, creates a new DataValue object and adds that object to a list of DataValue objects.

Now we are having some issues because we don’t know the best way to insert these data values in DHIS2.

The questions we are having are:

  1.   The approach that we have used to implement our PoC is the correct one? I mean, XLS -> List<DataValue> -> import is correct or we should use XLS -> CSV -> import?
    
  1.   How should we proceed, after the XLS file has been processed by the ExcelConverterBean?
    

Thanks for your support,

Paulo Grácio

Technical Manager
Skype: paulojrgracio

Critical Software Mozambique

Dependable Technologies for Critical Systems

Critical Software Mozambique is a subsidiary of Critical Software, a CMMI® Level 5 rated Company

CMMI® is registered in the USPTO by CMU

Rua Pereira Marinho, 179
Bairro de Sommerchield
Maputo
Moçambique
Phone: (+258) 214 951 45

Fax: (+258) 214 951 46

DISCLAIMER: This message is confidential and may contain privileged information. It is for use only by the people or entities to whom it is addressed. If you are not an intended recipient, you should not disclose, distribute, copy, print, rely on or otherwise make use of this message. If an addressing or transmission error has misdirected it to you we would be grateful if you would please notify the sender by return, before deleting it from your system.


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


*Pascal Brandt
Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA

Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*


*Pascal Brandt

Senio**r Software Developer, Jembi Health Systems | SOUTH AFRICA
Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt

E-mail: pascal@jembi.org*