Proposal for new DXF format

Hi,

As you know I was working to create metadata that is unique (in terms of metaobjects) all over the implementation. Details of how metadata is created was posted previously. Now, we have XML representation of metadata as no “Wow Snap” typew of things. Attached is dhis metadata (dxf3.xsd), which ensures that every XML import/export complying with this XSD will be always valid for W3. Every external source suplying data to DHIS2 following this XSD will not fail, and so on.

Also each metadata screen had icon for adding new metadata object. Once object is added to metadata, it is shown in green color to distinguish it from ordinary objects. Also views for metadata view were created in hierarchy and relations along with metadta export in xml (.xml). Changes made to master database could be easily transfered to local installations without any failure as long as we follow this XSD, which is built in DHIS2 implementation (one i am using). XSD counts for all relations in DHIS2 metadata and restores them in similar manner without failure. If someting is not counted for, it fails rather creating false output, that fails later for sure.

Much more to this shouls be added as we hear feedback from interested devs/implementers.
All code is some line to call lib functions, no need to take care of string beign XML compliant not to “Wow snap” users in specific utils of date and minutes handling.

So, I propose this XSD to be next version od DXF with feedbacks from users/devs incorporated. Most, and noticeable part of it is to respect duely relations between meta objects. My understanding is that unless we adhear to these issues, we will be solving individual issues of cases.

Please revise and comment as it is important for me to take further steps.

regards,
murod

dxf3.xsd (13.3 KB)

Hi Murod,
I think you raise some really good points, but being more towards the
implementers side than the developers side, it would be very useful to
see what you describe in action. Is the "local copy" which you refer
to available on Launchpad so that maybe those of us that are
interested could take a look at what you have done thus far?

Regards,
Jason

···

On Thu, Mar 10, 2011 at 5:26 PM, Murod Latifov <mlatifov@gmail.com> wrote:

Hi,

As you know I was working to create metadata that is unique (in terms of
metaobjects) all over the implementation. Details of how metadata is created
was posted previously. Now, we have XML representation of metadata as no
"Wow Snap" typew of things. Attached is dhis metadata (dxf3.xsd), which
ensures that every XML import/export complying with this XSD will be always
valid for W3. Every external source suplying data to DHIS2 following this
XSD will not fail, and so on.
Also each metadata screen had icon for adding new metadata object. Once
object is added to metadata, it is shown in green color to distinguish it
from ordinary objects. Also views for metadata view were created in
hierarchy and relations along with metadta export in xml (.xml). Changes
made to master database could be easily transfered to local installations
without any failure as long as we follow this XSD, which is built in DHIS2
implementation (one i am using). XSD counts for all relations in DHIS2
metadata and restores them in similar manner without failure. If someting is
not counted for, it fails rather creating false output, that fails later for
sure.
Much more to this shouls be added as we hear feedback from interested
devs/implementers.
All code is some line to call lib functions, no need to take care of string
beign XML compliant not to "Wow snap" users in specific utils of date and
minutes handling.
So, I propose this XSD to be next version od DXF with feedbacks from
users/devs incorporated. Most, and noticeable part of it is to respect duely
relations between meta objects. My understanding is that unless we adhear to
these issues, we will be solving individual issues of cases.
Please revise and comment as it is important for me to take further steps.

regards,
murod

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260974901293

Hi Jason,

Yes, all my work is publicly available from bzr branch lp:~murodlatifov/+junk/dhis2. I am using JAXB for validating and marshalling XML. Reason for having separate branch is validation, which fails for cyrrilyc chars. Also I made locale handling differently. Last week up to now I only commited to my local Bazaar only, as was strugling with third party installers in NDIS module for installer. They are huge and meaningless to be kept in bazaar. But metadata part should be there by now.

Looking forward for your comments.
Most of metadata handling is in dhis-web-datadictionary/dataset/orgunit/ modules under “meta” packages. There are two menu options in “Data Elements and Indicators”(Data Dictionary) menu with same nane at the bottom :), one brows metaobjects in tabs, other produces xml representation of metaobjects according to given xsd. There are also autogeneretaed Java objects for each object in xsd to ensure valid output. Package “org.hisp.dhis.meta” in dhis-web-datadictionary has these objects, they are created each time you run mvn from src/main/resources/dxf3.xsd and placed in target generated sources. You need to manually copy them over to “org.hisp.dhis.meta” if modified xsd.

regards,
murod

···

On Thu, Mar 10, 2011 at 7:38 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Murod,

I think you raise some really good points, but being more towards the

implementers side than the developers side, it would be very useful to

see what you describe in action. Is the “local copy” which you refer

to available on Launchpad so that maybe those of us that are

interested could take a look at what you have done thus far?

Regards,

Jason

On Thu, Mar 10, 2011 at 5:26 PM, Murod Latifov mlatifov@gmail.com wrote:

Hi,

As you know I was working to create metadata that is unique (in terms of

metaobjects) all over the implementation. Details of how metadata is created

was posted previously. Now, we have XML representation of metadata as no

“Wow Snap” typew of things. Attached is dhis metadata (dxf3.xsd), which

ensures that every XML import/export complying with this XSD will be always

valid for W3. Every external source suplying data to DHIS2 following this

XSD will not fail, and so on.

Also each metadata screen had icon for adding new metadata object. Once

object is added to metadata, it is shown in green color to distinguish it

from ordinary objects. Also views for metadata view were created in

hierarchy and relations along with metadta export in xml (.xml). Changes

made to master database could be easily transfered to local installations

without any failure as long as we follow this XSD, which is built in DHIS2

implementation (one i am using). XSD counts for all relations in DHIS2

metadata and restores them in similar manner without failure. If someting is

not counted for, it fails rather creating false output, that fails later for

sure.

Much more to this shouls be added as we hear feedback from interested

devs/implementers.

All code is some line to call lib functions, no need to take care of string

beign XML compliant not to “Wow snap” users in specific utils of date and

minutes handling.

So, I propose this XSD to be next version od DXF with feedbacks from

users/devs incorporated. Most, and noticeable part of it is to respect duely

relations between meta objects. My understanding is that unless we adhear to

these issues, we will be solving individual issues of cases.

Please revise and comment as it is important for me to take further steps.

regards,

murod


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+260974901293

Hi Murod

Don't have time for long reply right now and i will look at your
branch more closely later. But a few quick responses :
1. Having a schema driven validation is a good thing, both for
integrity and also for publishing. I think we all agree with that.
Choices around how and when to bind the schema to java objects are not
quite so clearcut and reflect tradeoffs between flexibility,
robustness, adaptability, maintenance etc.

2. There are some aspects of your schema which might probably be
improved but that is natural. would be strange if not the case. For
example elements like <xsd:element name="members" type="xsd:int"
minOccurs="1" maxOccurs="unbounded" />

3. Other than (i) having a schema and (ii) the improvement that you
have done away with the separate association elements, I think there
are still problems to be solved with identifiers for objects (and
identification of the metadata set itself). I know there was
something in your previous post about metaobject generating unique ids
but I'm not sure I fully understood the rationale. To be honest this
looks much more like plain old dxf than i had expected - given your
previous discussion about metadata objects i was expecting you would
have proceeded in a more radical direction serializing those "weakly
typed" objects.

There are a lot of different use cases and stakeholders involved so I
doubt that your solution could be lifted as is into the mainstream,
but I think it is doing a lot of things in the right direction. I am
going to be in Oslo at the end of the month and i hope to be able to
hammer some things out with Jo re his requirements. Are you there?

Regards
Bob

···

On 10 March 2011 21:29, Murod Latifov <mlatifov@gmail.com> wrote:

Hi Jason,

Yes, all my work is publicly available from bzr branch
lp:~murodlatifov/+junk/dhis2. I am using JAXB for validating and marshalling
XML. Reason for having separate branch is validation, which fails for
cyrrilyc chars. Also I made locale handling differently. Last week up to now
I only commited to my local Bazaar only, as was strugling with third party
installers in NDIS module for installer. They are huge and meaningless to be
kept in bazaar. But metadata part should be there by now.
Looking forward for your comments.
Most of metadata handling is in dhis-web-datadictionary/dataset/orgunit/
modules under "meta" packages. There are two menu options in "Data Elements
and Indicators"(Data Dictionary) menu with same nane at the bottom :), one
brows metaobjects in tabs, other produces xml representation of metaobjects
according to given xsd. There are also autogeneretaed Java objects for each
object in xsd to ensure valid output. Package "org.hisp.dhis.meta" in
dhis-web-datadictionary has these objects, they are created each time you
run mvn from src/main/resources/dxf3.xsd and placed in target generated
sources. You need to manually copy them over to "org.hisp.dhis.meta" if
modified xsd.

regards,
murod

On Thu, Mar 10, 2011 at 7:38 PM, Jason Pickering > <jason.p.pickering@gmail.com> wrote:

Hi Murod,
I think you raise some really good points, but being more towards the
implementers side than the developers side, it would be very useful to
see what you describe in action. Is the "local copy" which you refer
to available on Launchpad so that maybe those of us that are
interested could take a look at what you have done thus far?

Regards,
Jason

On Thu, Mar 10, 2011 at 5:26 PM, Murod Latifov <mlatifov@gmail.com> wrote:
> Hi,
>
> As you know I was working to create metadata that is unique (in terms of
> metaobjects) all over the implementation. Details of how metadata is
> created
> was posted previously. Now, we have XML representation of metadata as no
> "Wow Snap" typew of things. Attached is dhis metadata (dxf3.xsd), which
> ensures that every XML import/export complying with this XSD will be
> always
> valid for W3. Every external source suplying data to DHIS2 following
> this
> XSD will not fail, and so on.
> Also each metadata screen had icon for adding new metadata object. Once
> object is added to metadata, it is shown in green color to distinguish
> it
> from ordinary objects. Also views for metadata view were created in
> hierarchy and relations along with metadta export in xml (.xml). Changes
> made to master database could be easily transfered to local
> installations
> without any failure as long as we follow this XSD, which is built in
> DHIS2
> implementation (one i am using). XSD counts for all relations in DHIS2
> metadata and restores them in similar manner without failure. If
> someting is
> not counted for, it fails rather creating false output, that fails later
> for
> sure.
> Much more to this shouls be added as we hear feedback from interested
> devs/implementers.
> All code is some line to call lib functions, no need to take care of
> string
> beign XML compliant not to "Wow snap" users in specific utils of date
> and
> minutes handling.
> So, I propose this XSD to be next version od DXF with feedbacks from
> users/devs incorporated. Most, and noticeable part of it is to respect
> duely
> relations between meta objects. My understanding is that unless we
> adhear to
> these issues, we will be solving individual issues of cases.
> Please revise and comment as it is important for me to take further
> steps.
>
> regards,
> murod
>
> _______________________________________________
> Mailing list: DHIS 2 developers in Launchpad
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : DHIS 2 developers in Launchpad
> More help : ListHelp - Launchpad Help
>
>

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260974901293

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

Hi Bob,

Thanks for comments and initiating discussions around it. See my responses inline.

Hi Murod

Don’t have time for long reply right now and i will look at your

branch more closely later. But a few quick responses :

  1. Having a schema driven validation is a good thing, both for

integrity and also for publishing. I think we all agree with that.

Choices around how and when to bind the schema to java objects are not

quite so clearcut and reflect tradeoffs between flexibility,

robustness, adaptability, maintenance etc.

Java binding autogenerated from xsd, and it is a single point for maintenance.

  1. There are some aspects of your schema which might probably be

improved but that is natural. would be strange if not the case. For

example elements like <xsd:element name=“members” type=“xsd:int”

minOccurs=“1” maxOccurs=“unbounded” />

DataSet could have 1 ore more DataElements. Integer values are metaId refencing thai instance of DE. inside makes sure that each metaId being used is beforehand initialized. See attached sample output for details.

  1. Other than (i) having a schema and (ii) the improvement that you

have done away with the separate association elements, I think there

are still problems to be solved with identifiers for objects (and

identification of the metadata set itself). I know there was

something in your previous post about metaobject generating unique ids

but I’m not sure I fully understood the rationale. To be honest this

looks much more like plain old dxf than i had expected - given your

previous discussion about metadata objects i was expecting you would

have proceeded in a more radical direction serializing those "weakly

typed" objects.

In sample meta.xml all ids are metaids, not real dhis object id. So senario is that during applying this metadata system will call say getDataElementByMetaCode(metaId). If such is found it is updated, else new DataElement is created and its metaCode set to this metaId (its own id could come from dhis and can be any integer), other properties also applied, and from now on this newly created DE will be refered to as instance of MetaObject(metaId). This way we will apply unique id accross different installations of the same implementation.

While exporting the same senario in reverse will happen. Once DataValues are selected for export, their ids are replaced with their metaIds, which are unique and known to all instances of implementation, other end will do the same but other way around, it converts metaIds to its internal dhis ids of that [DataValue, category, period??] Instance if case is datavalue. This part still has to be built as well as import/export using xsd. xsd also does not have DataValue object yet. But I wanted to hear other voices before making further steps.

So all interface to external sources are via metaIds, internally things will go as dhis does.

There are a lot of different use cases and stakeholders involved so I

doubt that your solution could be lifted as is into the mainstream,

but I think it is doing a lot of things in the right direction.

This xsd is quite generic and pure reflection of dhis data model and its relations, so I doubt if it does not fit any needs. So your sample in 2 is analogous to List, but in List format, still following original objects cardinalities.

I am

going to be in Oslo at the end of the month and i hope to be able to

hammer some things out with Jo re his requirements. Are you there?

I am working with local team here to finalize metadata, I am not sure if I could come.

regards,
murod

meta.xml (114 KB)

···

On Fri, Mar 11, 2011 at 11:08 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Regards

Bob

On 10 March 2011 21:29, Murod Latifov mlatifov@gmail.com wrote:

Hi Jason,

Yes, all my work is publicly available from bzr branch

lp:~murodlatifov/+junk/dhis2. I am using JAXB for validating and marshalling

XML. Reason for having separate branch is validation, which fails for

cyrrilyc chars. Also I made locale handling differently. Last week up to now

I only commited to my local Bazaar only, as was strugling with third party

installers in NDIS module for installer. They are huge and meaningless to be

kept in bazaar. But metadata part should be there by now.

Looking forward for your comments.

Most of metadata handling is in dhis-web-datadictionary/dataset/orgunit/

modules under “meta” packages. There are two menu options in "Data Elements

and Indicators"(Data Dictionary) menu with same nane at the bottom :), one

brows metaobjects in tabs, other produces xml representation of metaobjects

according to given xsd. There are also autogeneretaed Java objects for each

object in xsd to ensure valid output. Package “org.hisp.dhis.meta” in

dhis-web-datadictionary has these objects, they are created each time you

run mvn from src/main/resources/dxf3.xsd and placed in target generated

sources. You need to manually copy them over to “org.hisp.dhis.meta” if

modified xsd.

regards,

murod

On Thu, Mar 10, 2011 at 7:38 PM, Jason Pickering > > > jason.p.pickering@gmail.com wrote:

Hi Murod,

I think you raise some really good points, but being more towards the

implementers side than the developers side, it would be very useful to

see what you describe in action. Is the “local copy” which you refer

to available on Launchpad so that maybe those of us that are

interested could take a look at what you have done thus far?

Regards,

Jason

On Thu, Mar 10, 2011 at 5:26 PM, Murod Latifov mlatifov@gmail.com wrote:

Hi,

As you know I was working to create metadata that is unique (in terms of

metaobjects) all over the implementation. Details of how metadata is

created

was posted previously. Now, we have XML representation of metadata as no

“Wow Snap” typew of things. Attached is dhis metadata (dxf3.xsd), which

ensures that every XML import/export complying with this XSD will be

always

valid for W3. Every external source suplying data to DHIS2 following

this

XSD will not fail, and so on.

Also each metadata screen had icon for adding new metadata object. Once

object is added to metadata, it is shown in green color to distinguish

it

from ordinary objects. Also views for metadata view were created in

hierarchy and relations along with metadta export in xml (.xml). Changes

made to master database could be easily transfered to local

installations

without any failure as long as we follow this XSD, which is built in

DHIS2

implementation (one i am using). XSD counts for all relations in DHIS2

metadata and restores them in similar manner without failure. If

someting is

not counted for, it fails rather creating false output, that fails later

for

sure.

Much more to this shouls be added as we hear feedback from interested

devs/implementers.

All code is some line to call lib functions, no need to take care of

string

beign XML compliant not to “Wow snap” users in specific utils of date

and

minutes handling.

So, I propose this XSD to be next version od DXF with feedbacks from

users/devs incorporated. Most, and noticeable part of it is to respect

duely

relations between meta objects. My understanding is that unless we

adhear to

these issues, we will be solving individual issues of cases.

Please revise and comment as it is important for me to take further

steps.

regards,

murod


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+260974901293


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp