ETL process with DHIS2

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

···

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Thank you Knut.

Henri

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

···

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Hi Henri,
As Knut says, there is no “automatic” way of doing this, mostly because your Excel files are likely completely custom. Thus, you will need to transform your Excel files into something which DHIS2 can understand, possibly using CSV to import data.

I have developed a somewhat more technical example of how this can be accomplished with the R programming language, although regardless of you what use, the steps are basically the same.

You can find that here.

Regards,

Jason

···

On Mon, Mar 21, 2016 at 5:21 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Henri,
I read on this forum that someone was attempting to use Kettle (Pentaho Data Integration). I don’t know whether they succeeded.

Regards

Ant Snyman

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.

image

···

On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ant Snyman

Cell: 0824910449

Landline: 028 2713242

Health Information Systems Program - SA

Good point, Ant.

Here is a link to that conversation, with many interesting points:

https://lists.launchpad.net/dhis2-users/msg09234.html

Knut

image

···

On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman ant@hisp.org wrote:

Hi Henri,
I read on this forum that someone was attempting to use Kettle (Pentaho Data Integration). I don’t know whether they succeeded.

Regards

Ant Snyman

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ant Snyman

Cell: 0824910449

Landline: 028 2713242

Health Information Systems Program - SA

Hi - we did some work using Talend based on client desires. We found it to be an incredible pain in the butt. Although our group are typically senior software developers or technical architects we had to pull in vendor consultants to get training on how to use it and ultimately dumped it. Beware the marketing - they all promote self-service but I suspect you’ll either be bringing in their trainers, hiring their consultants or finding your own software developers to actually get the job done.

Aaron

···

On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring knutst@gmail.com wrote:

Good point, Ant.

Here is a link to that conversation, with many interesting points:

https://lists.launchpad.net/dhis2-users/msg09234.html

Knut


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman ant@hisp.org wrote:

Hi Henri,
I read on this forum that someone was attempting to use Kettle (Pentaho Data Integration). I don’t know whether they succeeded.

Regards

Ant Snyman

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ant Snyman

Cell: 0824910449

Landline: 028 2713242

Health Information Systems Program - SA

Aaron Gladders, 2Paths
http://www.2paths.com/

Hi.

Thank you Ant, Knut, Jason and Aaron for the tips. I’m sure the discussions we are having will make us continue to improve the DHIS2 platform, particularly for datawarehouse implementations.

I was also looking for Kettle and Talend for this purpose. Finally, I found DHIS2 very comfortable for Export and Import as it is possible to do this step by step (you choose which metadata you want to export/import).

Now I see the way is how to TRANSFORM the Excel databases to fit with the exported file. For now, I intend to do this manually…

Thank you.

Hi - we did some work using Talend based on client desires. We found it to be an incredible pain in the butt. Although our group are typically senior software developers or technical architects we had to pull in vendor consultants to get training on how to use it and ultimately dumped it. Beware the marketing - they all promote self-service but I suspect you’ll either be bringing in their trainers, hiring their consultants or finding your own software developers to actually get the job done.

Aaron

···

Le Mardi 22 mars 2016 7h58, Aaron Gladders agladders@2paths.com a écrit :

On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring knutst@gmail.com wrote:

Good point, Ant.

Here is a link to that conversation, with many interesting points:

https://lists.launchpad.net/dhis2-users/msg09234.html

Knut

On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman ant@hisp.org wrote:

Hi Henri,
I read on this forum that someone was attempting to use Kettle (Pentaho Data Integration). I don’t know whether they succeeded.

Regards

Ant Snyman

On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ant Snyman

Cell: 0824910449

Landline: 028 2713242

Health Information Systems Program - SA

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Aaron Gladders, 2Paths
http://www.2paths.com/


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

That is often the easiest way. Excel also comes with recordable macros and its own scripting language, VBA.

Good luck!

···

On Wed, Mar 23, 2016 at 9:01 AM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi.

Thank you Ant, Knut, Jason and Aaron for the tips. I’m sure the discussions we are having will make us continue to improve the DHIS2 platform, particularly for datawarehouse implementations.

I was also looking for Kettle and Talend for this purpose. Finally, I found DHIS2 very comfortable for Export and Import as it is possible to do this step by step (you choose which metadata you want to export/import).

Now I see the way is how to TRANSFORM the Excel databases to fit with the exported file. For now, I intend to do this manually…

Thank you.

Le Mardi 22 mars 2016 7h58, Aaron Gladders agladders@2paths.com a écrit :

Hi - we did some work using Talend based on client desires. We found it to be an incredible pain in the butt. Although our group are typically senior software developers or technical architects we had to pull in vendor consultants to get training on how to use it and ultimately dumped it. Beware the marketing - they all promote self-service but I suspect you’ll either be bringing in their trainers, hiring their consultants or finding your own software developers to actually get the job done.

Aaron

On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring knutst@gmail.com wrote:

Good point, Ant.

Here is a link to that conversation, with many interesting points:

https://lists.launchpad.net/dhis2-users/msg09234.html

Knut

On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman ant@hisp.org wrote:

Hi Henri,
I read on this forum that someone was attempting to use Kettle (Pentaho Data Integration). I don’t know whether they succeeded.

Regards

Ant Snyman

On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.

Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,

There is no specific 3rd party software that you have to use, people use many different tools. Perhaps most commonly, people use scripts in Python or R or similar to load data. You can import both metadata and data in CSV format, for example.

http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html

http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut

On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr wrote:

Hi to all.

I want to test the ETL process with DHIS2. Is there an automated way to do this? is there a recommended tiers soft to use for this? my operational DB to use are excel files.

Thank you in advance.

Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ant Snyman

Cell: 0824910449

Landline: 028 2713242

Health Information Systems Program - SA

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.


Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Aaron Gladders, 2Paths
http://www.2paths.com/


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

HI Henri,

sorry for for chipping in late, I am on vaccation. We are using kettle for
extracting data into DHIS2 as a prototype for a datawarehouse in a corporate
setting. A few of Aarons points on Talend apply for kettle as well. Things can
get tricky, but that is partly because of the nature of data import into DHIS2.
Support, community etc are not as vivid as the DHIS2 community, but you still
find some material on the net.

It runs and does the job. Currently we are batteling with memory issues as I am
trying to find some suitable method of packaging data partitions. Again, that is
not only kettles fault.

I just found out that the MoH is also using kettle in their HMIS setting, so I
am quite confident to have a few ressource persons in the country for
sustainable hand-over.

Compared to R etc I like the graphical data-flow, which helps when discussing
transformations with users. You can actually see what is going into and is
coming out of a process with a mouse-click.

Our current transformations are quite generique for extracting from a custom
data-source, hoping to put them on the net after some clean ups.

Generally speaking I'd prefere to have more extensive ETL capabilites within
DHIS2, though, e.g. to better synchronise and monitor load-jobs, not having to
allocate memory to two java engines, consistent management of data-elements
etc...

Regards,

Uwe

···

henri claude moungui <henrimoungui@yahoo.fr> hat am 23. März 2016 um 11:01
geschrieben:

Hi.Thank you Ant, Knut, Jason and Aaron for the tips. I'm sure the discussions
we are having will make us continue to improve the DHIS2 platform,
particularly for datawarehouse implementations.I was also looking for Kettle
and Talend for this purpose. Finally, I found DHIS2 very comfortable for
Export and Import as it is possible to do this step by step (you choose which
metadata you want to export/import). Now I see the way is how to TRANSFORM the
Excel databases to fit with the exported file. For now, I intend to do this
manually...
Thank you.

    Le Mardi 22 mars 2016 7h58, Aaron Gladders <agladders@2paths.com> a écrit
:

Hi - we did some work using Talend based on client desires. We found it to
be an incredible pain in the butt. Although our group are typically senior
software developers or technical architects we had to pull in vendor
consultants to get training on how to use it and ultimately dumped it. Beware
the marketing - they all promote self-service but I suspect you'll either be
bringing in their trainers, hiring their consultants or finding your own
software developers to actually get the job done.
Aaron

On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring <knutst@gmail.com> wrote:

Good point, Ant.
Here is a link to that conversation, with many interesting
points:Re: Importing DHS survey data in DHIS : Mailing list archive : dhis2-users team in Launchpad

Knut
On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman <ant@hisp.org> wrote:

Hi Henri,I read on this forum that someone was attempting to use Kettle
(Pentaho Data Integration). I don't know whether they succeeded.RegardsAnt
Snyman
On 21 March 2016 at 18:21, henri claude moungui <henrimoungui@yahoo.fr> wrote:

Thank you Knut.
Henri

    Le Lundi 21 mars 2016 15h25, Knut Staring <knutst@gmail.com> a écrit :

Hi Henri,
There is no specific 3rd party software that you have to use, people use many
different tools. Perhaps most commonly, people use scripts in Python or R or
similar to load data. You can import both metadata and data in CSV format, for
example.
http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html
http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut
On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui <henrimoungui@yahoo.fr> > wrote:

Hi to all.I want to test the ETL process with DHIS2. Is there an automated way
to do this? is there a recommended tiers soft to use for this? my operational
DB to use are excel files.Thank you in advance.
Henri Moungui

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

--
Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
Skype: knutstarhttp://dhis2.org

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

--

Ant SnymanCell: 0824910449Landline: 028 2713242
Health Information Systems Program - SA
This message and any attachments are subject to a disclaimer publishedat
Home - HISP-SA. Please read the disclaimer
before opening any attachment or taking any otheraction in terms of this
electronic transmission. If you cannot access thedisclaimer, kindly send an
email to disclaimer@hisp.org and a copy will be provided to you. Byreplying to
this e-mail or opening any attachment you agree to be bound by theprovisions
of the disclaimer.

--
Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
Skype: knutstarhttp://dhis2.org
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

--
Aaron Gladders, 2Paths
http://www.2paths.com/
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Thank you Uwe for sharing your experience. Better late than never *;) Clin d’œil

I agree with you. It is effectively the graphical data-flow on ETL that is missing on DHIS2…

Regards.

Henri

HI Henri,

sorry for for chipping in late, I am on vaccation. We are using kettle for
extracting data into DHIS2 as a prototype for a datawarehouse in a corporate
setting. A few of Aarons points on Talend apply for kettle as well. Things can
get tricky, but that is partly because of the nature of data import into DHIS2.
Support, community etc are not as vivid as the DHIS2 community, but you still
find some material on the net.

It runs and does the job. Currently we are batteling with memory issues as I am
trying to find some suitable method of packaging data partitions. Again, that is
not only kettles fault.

I just found out that the MoH is also using kettle in their HMIS setting, so I
am quite confident to have a few ressource persons in the country for
sustainable hand-over.

Compared to R etc I like the graphical data-flow, which helps when discussing
transformations with users. You can actually see what is going into and is
coming out of a process with a mouse-click.

Our current transformations are quite generique for extracting from a custom
data-source, hoping to put them on the net after some clean ups.

Generally speaking I’d prefere to have more extensive ETL capabilites within
DHIS2, though, e.g. to better synchronise and monitor load-jobs, not having to
allocate memory to two java engines, consistent management of data-elements
etc…

Regards,

Uwe

···

Le Jeudi 24 mars 2016 9h18, Uwe Wahser uwe@wahser.de a écrit :

henri claude moungui henrimoungui@yahoo.fr hat am 23. März 2016 um 11:01
geschrieben:

Hi.Thank you Ant, Knut, Jason and Aaron for the tips. I’m sure the discussions
we are having will make us continue to improve the DHIS2 platform,
particularly for datawarehouse implementations.I was also looking for Kettle
and Talend for this purpose. Finally, I found DHIS2 very comfortable for
Export and Import as it is possible to do this step by step (you choose which
metadata you want to export/import). Now I see the way is how to TRANSFORM the
Excel databases to fit with the exported file. For now, I intend to do this
manually…
Thank you.

Le Mardi 22 mars 2016 7h58, Aaron Gladders agladders@2paths.com a écrit
:

Hi - we did some work using Talend based on client desires. We found it to
be an incredible pain in the butt. Although our group are typically senior
software developers or technical architects we had to pull in vendor
consultants to get training on how to use it and ultimately dumped it. Beware
the marketing - they all promote self-service but I suspect you’ll either be
bringing in their trainers, hiring their consultants or finding your own
software developers to actually get the job done.
Aaron

On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring knutst@gmail.com wrote:

Good point, Ant.
Here is a link to that conversation, with many interesting
points:https://lists.launchpad.net/dhis2-users/msg09234.html

Knut
On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman ant@hisp.org wrote:

Hi Henri,I read on this forum that someone was attempting to use Kettle
(Pentaho Data Integration). I don’t know whether they succeeded.RegardsAnt
Snyman
On 21 March 2016 at 18:21, henri claude moungui henrimoungui@yahoo.fr wrote:

Thank you Knut.
Henri

Le Lundi 21 mars 2016 15h25, Knut Staring knutst@gmail.com a écrit :

Hi Henri,
There is no specific 3rd party software that you have to use, people use many
different tools. Perhaps most commonly, people use scripts in Python or R or
similar to load data. You can import both metadata and data in CSV format, for
example.
http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html
http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html

Knut
On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui henrimoungui@yahoo.fr > wrote:

Hi to all.I want to test the ETL process with DHIS2. Is there an automated way
to do this? is there a recommended tiers soft to use for this? my operational
DB to use are excel files.Thank you in advance.
Henri Moungui


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp


Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
Skype: knutstarhttp://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Ant SnymanCell: 0824910449Landline: 028 2713242
Health Information Systems Program - SA
This message and any attachments are subject to a disclaimer publishedat
[Home - HISP-SA. Please ](Home - HISP-SA. Please)read the disclaimer
before opening any attachment or taking any otheraction in terms of this
electronic transmission. If you cannot access thedisclaimer, kindly send an
email to disclaimer@hisp.org and a copy will be provided to you. Byreplying to
this e-mail or opening any attachment you agree to be bound by theprovisions
of the disclaimer.


Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
Skype: knutstarhttp://dhis2.org


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp


Aaron Gladders, 2Paths
http://www.2paths.com/


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp