DHIS OpenMRS integration

Hi Than,

Thanks, this is what I wanted to discuss with OpemMRS, not IXF/DXF a la standards. But the way I am doing is quite different. OpenMRS has Cohort builder, using it we can create aggregated values. But it is not what we want, we need to change behaviour of Cohort Builder to match DHIS periods, and finally map these values to DHIS analog - data elements. This way we reuse existing code and functionality.
Comments are welcome.

regards,
murod

···

From: Ngoc Thanh Nguyen thanh.hispvietnam@gmail.com
To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Bob Jolliffe bobjolliffe@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net; Sundeep Sahay sundeep.sahay@yahoo.com; Jørn Braa jornbraa@gmail.com
Sent: Wednesday, August 5, 2009 10:10:20 AM
Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Hi all,

I have been following all the discussions. The key issue is how to make OpenMRS and DHIS2 talk to each other. This is not about what standards should be used but THE DESIGN of AN interpreter (translator). I think we could discuss on how to design this mapping.

The key element in DHIS2 is the Dataelement and Value (often a number).

In OpenMRS, Concept is something similar to Dataelement. The answer for concept (Value) is other concept.

The Period in DHIS2 shares similarity with Encounter date time.

The Orgunit in DHIS2 is similar with Patient and location in OpenMRS.

Here are some examples of mapping:

  1. Simple mapping with only encounter

In OpenMRS, for outpatient department, every day many patients come to a clinic (hospital) to have check up.

This check up event will be stored as an encounter with date, time.

For DHIS2, one of the data element could be: “Number of patients come to the clinic this month?”

So how to map?

“select count(*) from encounter where encounter.date between FirstDateOfThisMonth and LastDateOfThisMonth”

  1. Mapping with Observation and concept

Ok, after the patient is registered, he come to see doctor in a check up room. The doctor will give diagnosis saying that he is infected by tuberculosis or malaria or any kinds of disease.

In OpenMRS, the diagnosis is stored as an observation with the Concept name is Diagnosis. The answer for this concept is also a list of concepts (ICD 10).

For DHIS2, many data element could be come from this scenario:

  • How many [TB] patient in a month?

  • How many [Malaria] patient in a month?

  • Up to some hundreds of data elements.

So how to map?

“select count(*) from obs where answer_concept is [TB] and date between FirstDateOfThisMonth and LastDateOfThisMonth”

  1. Break out by [sex] and [age]:

The query is the same but with a little modification

“select count(*) from encounter where encounter.date between FirstDateOfThisMonth and LastDateOfThisMonth”

"select count(*) from encounter inner join patient on encounter.patientid = patient.id where encounter.date between FirstDateOfThisMonth and LastDateOfThisMonth AND patient.age between [startage] and [endage] and patient.sex = [sex]

  1. Other queries:

Statistical reports require more than just clinical data. This requirements range from human resources, number of medical lab machines, financial management etc.

And if there are no such things in OpenMRS, DHIS2 must pull data from other sources.

Regards,

Thanh

On Mon, Aug 3, 2009 at 8:02 PM, Murodullo Latifov murodlatifov@yahoo.com wrote:

Hi,

Discussion was mainly around IXF/DXF standards for representing data, which I ignored on my initial implementation, and likely will do so for now. I am concentrated to make systems talk, after we can think of standards. This was agreement with OpenMRS people, who insisted on standards, but after long debates I made them agree to go my way. I am actually using XML for data exchange, which is standard in a sense. Will come back to discussions while progress is made.

This was not really my impression. There was a breakout discussion of me, Ola, Murod and Paul (from OpenMRS). Maybe there were long debates afterwards which I missed. Anyway the prevailing view was that before we look at a new ad hoc way of doing things we need to be sure that there is not an existing standard way which is adequate. There was nobody who thought creating a new xml representation just for this openmrs-dhis integration is a good idea. Unfortunately Ola and I also had not seen Murod’s work in advance so it was difficult to present a comon dhis view. Murod can you share more of what you have done to the wider group?

Main issue was and is to map DHIS data to OMRS data. None of listed standards currently provide it. There should be a medium to dictate standard and uniform keys, names, and other attributes for both systems, from all available standards only SDMX-HD has some sample data and is good candidate to provide such service. Parsing is not an issue, validity of content is. That is where systems talk. This standardization is multy step and long process. omrs should follow ICD10 and dhis - SDMX-HD, there should be mapping between SDMX-HD and ICD10. This is my view of standards and might not be correct. Project I created is intended to link two systems in closest possible way, not to build standards or discuss standards. Taking this into account the rest of messages from this line onward are void. When there is proper standard and implementation instructions we can easily shift.

In terms of discussion, the following options were considered regrading
data format:

  1. SDMX_HD - Paul had not been aware of the deficiencies which I had pointed out. He was also at pains to insist that OpenMRS was not committed to this format yet, but like us, felt that it is better to align with a WHO effort if is feasible. If its not, he’s happy not to go that way. I asked him to get a second opinion on some of the concerns I had raised from the OpenMRS team.

  2. IXF - Paul said that IXFv2 did not have the problem which SDMX has - that is that new codes and codelists translate to new attributes in the data exchange schema. Given that we have both already implemented IXF parsers this might still be the basis of future interoperability. Ola mentioned that Lars had implemented IXF parser in DHIS but that he cursed it often :slight_smile: Lars you would know best whether this is a good idea or not.

  3. DXF - we didn’t discuss much. A pity, but time was short. This, or an enhanced version, will be a fallback if 1 and 2 above are not workable.

  4. Merger of 2 and 3 - this was quite an interesting thought. One of the problems with SDMX HD is that it is based on an ISO standard. Meaning that there is only a certain amount of scope to change things for the health domain - much of the rest is fixed by the ISO parent SDMX. IXF on the other hand can be taken and fixed, developed and improved by its stakeholders. It might be possible to take DXF and new ideas which have been suggested for DXF2 including elements of Murod’s schema for example and develop these as IXFv4.

Paul felt we need to create some sort of grid showing the pros and cons of these approaches and discuss on that basis. There was some possibility of making this interoperability problem a topic of the September OpenMRS implementors meeting in Cape Town.

Of course all of the above is only formats - there is more to it than that. Murod did identify the minimum information set that would need to form part of an exchange to get data from OpenMRS into DHIS. That is useful. And the idea of writing an OpenMRS module to do the job - rather than waiting for OpenMRS folk to do it - makes standards less critical, but an expensive way to consider interop in general. Though Paul had some suggestions around how the OpenMRS inference engine should be used which was a bit beyond “beginner” OpenMRS. And Saptarshi had some thoughts around using the OpenMRS cohort builder to assist with aggregation which I didn’t fully understand, but it sounded convincing :-). We are all agreed that aggregation happens on the OpenMRS side.

Its a pity we didn’t get much to web services and REST. I guess we used Murod’s work as the catalyst for discussion and ended up having lengthy debates on standard vs ad hoc xml instead.

Regards
Bob

murod


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Best regards,
Thanh


Center for HIS development
957, 3/2 st., Dist 11, Hochiminh city
Tel: 08-956 0150
Cell: 098 221 8623


Thanks Bob, I know this, so what it adds to new module? Does it do what it is supposed to do? If it does, what it does?

···

From: Bob Jolliffe bobjolliffe@gmail.com
To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Jo Størset storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net
Sent: Tuesday, August 4, 2009 10:04:10 PM
Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

2009/8/4 Murodullo Latifov murodlatifov@yahoo.com

Hi Bob,

Output of module would be what we are using for import into DHIS now, so no need to create it in DHIS again.

This is what is being called DXF.

Cheers

Bob

Yes, it would be same ids coming back with value assigned. Changing XML to look like IXF/DXF or SDMX-HD is not a big issue, we can do it in many levels before output.

regards,
murod


From: Bob Jolliffe bobjolliffe@gmail.com

To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Jo Størset
storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Sent: Tuesday, August 4, 2009 6:40:11 PM

Thanks. It is very similar to existing dxf (which has the same problematic use of IDs). What about the data going the other way? Is it also like DXF?

Regards
Bob

2009/8/4 Murodullo Latifov murodlatifov@yahoo.com

Hi Bob,

Here is my XML file. I have ids on it, and its problematic as ids differ from implementation to implementation.

regards,
murod


From: Bob Jolliffe bobjolliffe@gmail.com

To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Jo Størset storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net

Sent: Tuesday, August 4, 2009
5:31:29 PM

Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Hi Murod

Do you have an example of the XML you are using? I want to see how far it might be from the DXF we are already using, should it be incorporated or what … I remember seeing a small snippet but would be good to share a file or two.

Regards
Bob

2009/8/3 Murodullo Latifov murodlatifov@yahoo.com

Hi,

Discussion was mainly around IXF/DXF standards for representing data, which I ignored on my initial implementation, and likely will do so for now. I am concentrated to make systems talk, after we can think of standards. This was agreement with OpenMRS people, who insisted on standards, but after long debates I made them agree to go my way. I am actually using XML for data exchange, which is standard in a sense. Will come back to discussions while progress is made.

This was not really my impression. There was a breakout discussion of me, Ola, Murod and Paul (from OpenMRS). Maybe there were long debates afterwards which I missed. Anyway the prevailing view was that before we look at a new ad hoc way of doing things we need to be sure that there is not an existing standard way which is adequate. There was nobody who thought creating a new xml representation just for this openmrs-dhis integration is a good idea. Unfortunately Ola and I also had not seen Murod’s work in advance so it was difficult to present a comon dhis view. Murod can you share more of what you have done to the wider group?

Main issue was and is to map DHIS data to OMRS data. None of listed standards currently provide it. There should be a medium to dictate standard and uniform keys, names, and other attributes for both systems, from all available standards only SDMX-HD has some sample data and is good candidate to provide such service. Parsing is not an issue, validity of content is. That is where systems talk. This standardization is multy step and long process. omrs should follow ICD10 and dhis - SDMX-HD, there should be mapping between SDMX-HD and ICD10. This is my view of standards and might not be correct. Project I created is intended to link two systems in closest possible way, not to build standards or discuss standards. Taking this into account the rest of messages from this line onward are void. When there is proper standard and implementation instructions we can easily shift.

In terms of discussion, the following options were considered regrading
data format:

  1. SDMX_HD - Paul had not been aware of the deficiencies which I had pointed out. He was also at pains to insist that OpenMRS was not committed to this format yet, but like us, felt that it is better to align with a WHO effort if is feasible. If its not, he’s happy not to go that way. I asked him to get a second opinion on some of the concerns I had raised from the OpenMRS team.

  2. IXF - Paul said that IXFv2 did not have the problem which SDMX has - that is that new codes and codelists translate to new attributes in the data exchange schema. Given that we have both already implemented IXF parsers this might still be the basis of future interoperability. Ola mentioned that Lars had implemented IXF parser in DHIS but that he cursed it often :slight_smile: Lars you would know best whether this is a good idea or not.

  3. DXF - we didn’t discuss much. A pity, but time was short. This, or an enhanced version, will be a fallback if 1 and 2 above are not workable.

  4. Merger of 2 and 3 - this was quite an interesting thought. One of the problems with SDMX HD is that it is based on an ISO standard. Meaning that there is only a certain amount of scope to change things for the health domain - much of the rest is fixed by the ISO parent SDMX. IXF on the other hand can be taken and fixed, developed and improved by its stakeholders. It might be possible to take DXF and new ideas which have been suggested for DXF2 including elements of Murod’s schema for example and develop these as IXFv4.

Paul felt we need to create some sort of grid showing the pros and cons of these approaches and discuss on that basis. There was some possibility of making this interoperability problem a topic of the September OpenMRS implementors meeting in Cape Town.

Of course all of the above is only formats - there is more to it than that. Murod did identify the minimum information set that would need to form part of an exchange to get data from OpenMRS into DHIS. That is useful. And the idea of writing an OpenMRS module to do the job - rather than waiting for OpenMRS folk to do it - makes standards less critical, but an expensive way to consider interop in general. Though Paul had some suggestions around how the OpenMRS inference engine should be used which was a bit beyond “beginner” OpenMRS. And Saptarshi had some thoughts around using the OpenMRS cohort builder to assist with aggregation which I didn’t fully understand, but it sounded convincing :-). We are all agreed that aggregation happens on the OpenMRS side.

Its a pity we didn’t get much to web services and REST. I guess we used Murod’s work as the catalyst for discussion and ended up having lengthy debates on standard vs ad hoc xml instead.

Regards
Bob

murod


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Murod

Thanks Bob, I know this

OK. I was just unsure from your previous mail where it seems you were implying that there is what dhis does now and then there is something else called dxf. So just clarifying dxf is what dhis does now.

, so what it adds to new module? Does it do what it is supposed to do? If it does, what it does?

I’m not sure what you mean. As you know what dxf does now is it allows dhis to send datavalues and accompanying “metadata” backwards and forwards between dhis instances in an xml formatted file wrapped in a zip container. Does it do what it is supposed to do? Yes.

Can or should it be coaxed into doing something more than that: ie being a default format which other applications could also use for pumping data into dhis? Like what you were doing. I think it can, but there are a few modifications one might want to make to make it (i) more efficient and (ii) more understandable to 3rd party users. Though these modifications could be seen as nice to have. Whether DXF should be used for this depends on whether the requirements are already adequately covered by something existing, like IXF or SDMX HD. Its tricky. IXF it seems is no longer maintained and is hard to find any documentation. SDMX HD is enjoying lots of current support (basking in the sunlight of WHO), is a very rich grammar. What SDMX does very well is provide the ability to define “Key Families” which relates to your point about semantic interoperability above. What SDMX HD goes on to do with that facility does not always make a lot of sense. But more on that later.

On the question of “new module” I’d rather not see a new module specifically for openmrs integration. We should just have modules for importing data (and the fewer the better). And perhaps modules which expose services, like for example request for data element definitions.

Regards
Bob

···

2009/8/5 Murodullo Latifov murodlatifov@yahoo.com


From: Bob Jolliffe bobjolliffe@gmail.com
To: Murodullo Latifov murodlatifov@yahoo.com

Cc: Jo Størset storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Sent: Tuesday, August 4, 2009 10:04:10 PM

2009/8/4 Murodullo Latifov murodlatifov@yahoo.com

Hi Bob,

Output of module would be what we are using for import into DHIS now, so no need to create it in DHIS again.

This is what is being called DXF.

Cheers

Bob

Yes, it would be same ids coming back with value assigned. Changing XML to look like IXF/DXF or SDMX-HD is not a big issue, we can do it in many levels before output.

regards,
murod


From: Bob Jolliffe bobjolliffe@gmail.com

To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Jo Størset storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Sent: Tuesday, August 4, 2009 6:40:11 PM

Thanks. It is very similar to existing dxf (which has the same problematic use of IDs). What about the data going the other way? Is it also like DXF?

Regards
Bob

2009/8/4 Murodullo Latifov murodlatifov@yahoo.com

Hi Bob,

Here is my XML file. I have ids on it, and its problematic as ids differ from implementation to implementation.

regards,
murod


From: Bob Jolliffe bobjolliffe@gmail.com

To: Murodullo Latifov murodlatifov@yahoo.com
Cc: Jo Størset storset@gmail.com; DHIS 2 developers dhis2-devs@lists.launchpad.net

Sent: Tuesday, August 4, 2009 5:31:29 PM

Subject: Re: [Dhis2-devs] DHIS OpenMRS integration

Hi Murod

Do you have an example of the XML you are using? I want to see how far it might be from the DXF we are already using, should it be incorporated or what … I remember seeing a small snippet but would be good to share a file or two.

Regards
Bob

2009/8/3 Murodullo Latifov murodlatifov@yahoo.com

Hi,

Discussion was mainly around IXF/DXF standards for representing data, which I ignored on my initial implementation, and likely will do so for now. I am concentrated to make systems talk, after we can think of standards. This was agreement with OpenMRS people, who insisted on standards, but after long debates I made them agree to go my way. I am actually using XML for data exchange, which is standard in a sense. Will come back to discussions while progress is made.

This was not really my impression. There was a breakout discussion of me, Ola, Murod and Paul (from OpenMRS). Maybe there were long debates afterwards which I missed. Anyway the prevailing view was that before we look at a new ad hoc way of doing things we need to be sure that there is not an existing standard way which is adequate. There was nobody who thought creating a new xml representation just for this openmrs-dhis integration is a good idea. Unfortunately Ola and I also had not seen Murod’s work in advance so it was difficult to present a comon dhis view. Murod can you share more of what you have done to the wider group?

Main issue was and is to map DHIS data to OMRS data. None of listed standards currently provide it. There should be a medium to dictate standard and uniform keys, names, and other attributes for both systems, from all available standards only SDMX-HD has some sample data and is good candidate to provide such service. Parsing is not an issue, validity of content is. That is where systems talk. This standardization is multy step and long process. omrs should follow ICD10 and dhis - SDMX-HD, there should be mapping between SDMX-HD and ICD10. This is my view of standards and might not be correct. Project I created is intended to link two systems in closest possible way, not to build standards or discuss standards. Taking this into account the rest of messages from this line onward are void. When there is proper standard and implementation instructions we can easily shift.

In terms of discussion, the following options were considered regrading data format:

  1. SDMX_HD - Paul had not been aware of the deficiencies which I had pointed out. He was also at pains to insist that OpenMRS was not committed to this format yet, but like us, felt that it is better to align with a WHO effort if is feasible. If its not, he’s happy not to go that way. I asked him to get a second opinion on some of the concerns I had raised from the OpenMRS team.

  2. IXF - Paul said that IXFv2 did not have the problem which SDMX has - that is that new codes and codelists translate to new attributes in the data exchange schema. Given that we have both already implemented IXF parsers this might still be the basis of future interoperability. Ola mentioned that Lars had implemented IXF parser in DHIS but that he cursed it often :slight_smile: Lars you would know best whether this is a good idea or not.

  3. DXF - we didn’t discuss much. A pity, but time was short. This, or an enhanced version, will be a fallback if 1 and 2 above are not workable.

  4. Merger of 2 and 3 - this was quite an interesting thought. One of the problems with SDMX HD is that it is based on an ISO standard. Meaning that there is only a certain amount of scope to change things for the health domain - much of the rest is fixed by the ISO parent SDMX. IXF on the other hand can be taken and fixed, developed and improved by its stakeholders. It might be possible to take DXF and new ideas which have been suggested for DXF2 including elements of Murod’s schema for example and develop these as IXFv4.

Paul felt we need to create some sort of grid showing the pros and cons of these approaches and discuss on that basis. There was some possibility of making this interoperability problem a topic of the September OpenMRS implementors meeting in Cape Town.

Of course all of the above is only formats - there is more to it than that. Murod did identify the minimum information set that would need to form part of an exchange to get data from OpenMRS into DHIS. That is useful. And the idea of writing an OpenMRS module to do the job - rather than waiting for OpenMRS folk to do it - makes standards less critical, but an expensive way to consider interop in general. Though Paul had some suggestions around how the OpenMRS inference engine should be used which was a bit beyond “beginner” OpenMRS. And Saptarshi had some thoughts around using the OpenMRS cohort builder to assist with aggregation which I didn’t fully understand, but it sounded convincing :-). We are all agreed that aggregation happens on the OpenMRS side.

Its a pity we didn’t get much to web services and REST. I guess we used Murod’s work as the catalyst for discussion and ended up having lengthy debates on standard vs ad hoc xml instead.

Regards
Bob

murod


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp