Dynamic attributes

Hi,

I see there is development going on for dynamic attributes [1] on some meta objects (DataElement, Indicator, DataSet, OrganisationUnit?).

Some questions:

- What is the planned use cases for this?
- Is this significantly different from the attributes in the tracking module (I guess these might be more dynamic, but not sure)?
- It doesn't seem to be linked to the general classes of nameable/identifiable object. Is it meant to be implemented just for specific meta object types?

and then I guess my main interest:

- How should these be handled in the communication with external systems
- is it a general component that we should think through the use of in relation to things like the mobile solutions?

Would be great to get some background info.

Jo

[1] https://blueprints.launchpad.net/dhis2/+spec/dynamic-attributes-for-meta-objects

Hi

- What is the planned use cases for this?

This is part of my work for Rwanda, the system they are using today
has quite a bit more attributes connected to each data element (or
indicator as it was known in that system), and it was decided that
instead of expanding the set of attributes connected to meta-data
today, we would rather create a dynamic system were they would be
allowed to create their own attributes.

For the specific use case, what they want is:

1) Add more meta-attributes to data elements (but we also add support
for indicators, organisation units)
2) Create a data repository report (data dictionary) in PDF. For this,
I will expand the "Export to PDF" functionality that already exists in
data elements.

- Is this significantly different from the attributes in the tracking module (I guess these might be more dynamic, but not sure)?

Unsure, I have not looked into this. But Lars gave me the go-ahead on
this (and I guess they are different enough).

- It doesn't seem to be linked to the general classes of nameable/identifiable object. Is it meant to be implemented just for specific meta object types?

I'm a bit unsure what you mean here.. Both attribute and
attributeoption implement AbstractIdentifiableObject..

- How should these be handled in the communication with external systems
- is it a general component that we should think through the use of in relation to things like the mobile solutions?

I guess that would be up to your team :wink: I have not given that part of
it any further thoughts.

···

--
Morten

Would be great to get some background info.

Jo

[1] https://blueprints.launchpad.net/dhis2/+spec/dynamic-attributes-for-meta-objects

For the specific use case, what they want is:

1) Add more meta-attributes to data elements (but we also add support
for indicators, organisation units)
2) Create a data repository report (data dictionary) in PDF. For this,
I will expand the "Export to PDF" functionality that already exists in
data elements.

This is also in fact a very general requirement we see many places -
they want a "data dictionary" with dynamic attributes attached to data
elements, indicators, orgunits to be able to define more comprehensive
information beyond name, description etc.

- Is this significantly different from the attributes in the tracking module (I guess these might be more dynamic, but not sure)?

Unsure, I have not looked into this. But Lars gave me the go-ahead on
this (and I guess they are different enough).

Different in the sense that PatientAttribute and PatientAttributeValue
have Patient-specific associations. We might investigate if this can
be harmonized in the future though.

- It doesn't seem to be linked to the general classes of nameable/identifiable object. Is it meant to be implemented just for specific meta object types?

We are planning to implement for data element, indicator, org unit
only at least for this release. But it can easily be extended if we
need it for more objects. The associations are unidirectional from the
DataElement/IdentifiableObject -> AttributeValue in order to keep the
AttributeValue object clean.

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo

Breaking out from our current hard coded set of object attributes is
certainly a great step forward. There is also some advantage in
trying to think a bit generally about what we are doing. We are
seeing lots of progress in the right direction. I was about to start
a discussion about names earlier, but maybe it makes sense to consider
some of these things together.

In general an entity within the dhis model universe has:
(i) a set of identifiers
(ii) a set of names
(iii) a set of attributes
(iv) a set of relations with other attributes

One could think of names as attributes, but there are reasonable
semantic reasons to consider them separately.

On top of this generic view we have classes or categories of entity.
Such as DataElement, OrgUnit etc, where the class of the entity
determines the set of attributes and the type of relations between
them.

I think from an xml metadata representation perspective, this is
probably how it should be modelled. The fact that some of the
attributes may be fixed database fields and some dynamically
configured attributes is incidental and purely historical. If you had
the benefit of hindsight and were designing a datadictionary from
scratch, you might never have had the fixed fields in the database.
Or you might have come up with different ones. On import/export I
think the distinction should simply be masked by the API. Fixed or
non-fixed is an internal arrangement of little interest outside the
system.

Bob

PS. One of the possible uses of attributes which I am seeing is the
in fact the possibility of dynamically adding new identifiers for
objects.

···

On 8 September 2011 15:55, Jo Størset <storset@gmail.com> wrote:

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Morten

On a more concrete note than before ... :slight_smile:

I've finally had a chance to look more closely at what you are doing.
Can I make a small suggestion? Currently you are replicating a lot of
code as you implement attributes across dataelements, indicators and
orgunits. Currently these all extend AbstractNameable object.

What about creating AbstractExtendableObject which extends
AbstractNameableObject? Then include your "Set<AttributeValue>
attributeValues" and related methods in there. Then DataElements,
Indicators or what have you become extendable by simply extending that
class?

Bob

···

On 14 September 2011 12:27, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Breaking out from our current hard coded set of object attributes is
certainly a great step forward. There is also some advantage in
trying to think a bit generally about what we are doing. We are
seeing lots of progress in the right direction. I was about to start
a discussion about names earlier, but maybe it makes sense to consider
some of these things together.

In general an entity within the dhis model universe has:
(i) a set of identifiers
(ii) a set of names
(iii) a set of attributes
(iv) a set of relations with other attributes

One could think of names as attributes, but there are reasonable
semantic reasons to consider them separately.

On top of this generic view we have classes or categories of entity.
Such as DataElement, OrgUnit etc, where the class of the entity
determines the set of attributes and the type of relations between
them.

I think from an xml metadata representation perspective, this is
probably how it should be modelled. The fact that some of the
attributes may be fixed database fields and some dynamically
configured attributes is incidental and purely historical. If you had
the benefit of hindsight and were designing a datadictionary from
scratch, you might never have had the fixed fields in the database.
Or you might have come up with different ones. On import/export I
think the distinction should simply be masked by the API. Fixed or
non-fixed is an internal arrangement of little interest outside the
system.

Bob

PS. One of the possible uses of attributes which I am seeing is the
in fact the possibility of dynamically adding new identifiers for
objects.

On 8 September 2011 15:55, Jo Størset <storset@gmail.com> wrote:

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Bob

I've finally had a chance to look more closely at what you are doing.
Can I make a small suggestion? Currently you are replicating a lot of
code as you implement attributes across dataelements, indicators and
orgunits. Currently these all extend AbstractNameable object.

Currently, theres not too much duplication... just one added one field
called attributeValues (with getter/setter). If you're talking about
the newly committed code for making attributes/attributeMap available
(three duplications there), that code is not finished, I just had to
commit it since I'm working on something else now. Yes, that
functionality should be made more generic, as its basically doing the
same thing three times (and more times, if we also add these to
datasets, etc)

What about creating AbstractExtendableObject which extends
AbstractNameableObject? Then include your "Set<AttributeValue>
attributeValues" and related methods in there. Then DataElements,
Indicators or what have you become extendable by simply extending that
class?

Yes, I don't really want to mess too much with the original domain
classes, but making it a bit more generic is on the agenda. This is
very much WIP and more refactoring could/should happen. :slight_smile:

···

--
Morten

Bob

On 14 September 2011 12:27, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Breaking out from our current hard coded set of object attributes is
certainly a great step forward. There is also some advantage in
trying to think a bit generally about what we are doing. We are
seeing lots of progress in the right direction. I was about to start
a discussion about names earlier, but maybe it makes sense to consider
some of these things together.

In general an entity within the dhis model universe has:
(i) a set of identifiers
(ii) a set of names
(iii) a set of attributes
(iv) a set of relations with other attributes

One could think of names as attributes, but there are reasonable
semantic reasons to consider them separately.

On top of this generic view we have classes or categories of entity.
Such as DataElement, OrgUnit etc, where the class of the entity
determines the set of attributes and the type of relations between
them.

I think from an xml metadata representation perspective, this is
probably how it should be modelled. The fact that some of the
attributes may be fixed database fields and some dynamically
configured attributes is incidental and purely historical. If you had
the benefit of hindsight and were designing a datadictionary from
scratch, you might never have had the fixed fields in the database.
Or you might have come up with different ones. On import/export I
think the distinction should simply be masked by the API. Fixed or
non-fixed is an internal arrangement of little interest outside the
system.

Bob

PS. One of the possible uses of attributes which I am seeing is the
in fact the possibility of dynamically adding new identifiers for
objects.

On 8 September 2011 15:55, Jo Størset <storset@gmail.com> wrote:

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Bob

I've finally had a chance to look more closely at what you are doing.
Can I make a small suggestion? Currently you are replicating a lot of
code as you implement attributes across dataelements, indicators and
orgunits. Currently these all extend AbstractNameable object.

Currently, theres not too much duplication... just one added one field
called attributeValues (with getter/setter). If you're talking about
the newly committed code for making attributes/attributeMap available
(three duplications there), that code is not finished, I just had to
commit it since I'm working on something else now. Yes, that
functionality should be made more generic, as its basically doing the
same thing three times (and more times, if we also add these to
datasets, etc)

What about creating AbstractExtendableObject which extends
AbstractNameableObject? Then include your "Set<AttributeValue>
attributeValues" and related methods in there. Then DataElements,
Indicators or what have you become extendable by simply extending that
class?

Yes, I don't really want to mess too much with the original domain
classes, but making it a bit more generic is on the agenda. This is
very much WIP and more refactoring could/should happen. :slight_smile:

Sure. I know.

I'm just making sure I keep in touch because I'm also in the process
of working out how best to represent these things for the purpose of
of importing/exporting them. So what you do here has some significant
bearing on what I do. The few bits of duplication you have currently
will fan out further when we serialize/deserailize to xml.

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Regards
Bob

···

On 14 September 2011 14:05, Morten Olav Hansen <mortenoh@gmail.com> wrote:

--
Morten

Bob

On 14 September 2011 12:27, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Breaking out from our current hard coded set of object attributes is
certainly a great step forward. There is also some advantage in
trying to think a bit generally about what we are doing. We are
seeing lots of progress in the right direction. I was about to start
a discussion about names earlier, but maybe it makes sense to consider
some of these things together.

In general an entity within the dhis model universe has:
(i) a set of identifiers
(ii) a set of names
(iii) a set of attributes
(iv) a set of relations with other attributes

One could think of names as attributes, but there are reasonable
semantic reasons to consider them separately.

On top of this generic view we have classes or categories of entity.
Such as DataElement, OrgUnit etc, where the class of the entity
determines the set of attributes and the type of relations between
them.

I think from an xml metadata representation perspective, this is
probably how it should be modelled. The fact that some of the
attributes may be fixed database fields and some dynamically
configured attributes is incidental and purely historical. If you had
the benefit of hindsight and were designing a datadictionary from
scratch, you might never have had the fixed fields in the database.
Or you might have come up with different ones. On import/export I
think the distinction should simply be masked by the API. Fixed or
non-fixed is an internal arrangement of little interest outside the
system.

Bob

PS. One of the possible uses of attributes which I am seeing is the
in fact the possibility of dynamically adding new identifiers for
objects.

On 8 September 2011 15:55, Jo Størset <storset@gmail.com> wrote:

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Sure. I can tomorrow or Friday. Lars is in Uganda, but I'm sure he'll
respond later if he has time for a conference call.

···

--
Morten

Regards
Bob

--
Morten

Bob

On 14 September 2011 12:27, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Breaking out from our current hard coded set of object attributes is
certainly a great step forward. There is also some advantage in
trying to think a bit generally about what we are doing. We are
seeing lots of progress in the right direction. I was about to start
a discussion about names earlier, but maybe it makes sense to consider
some of these things together.

In general an entity within the dhis model universe has:
(i) a set of identifiers
(ii) a set of names
(iii) a set of attributes
(iv) a set of relations with other attributes

One could think of names as attributes, but there are reasonable
semantic reasons to consider them separately.

On top of this generic view we have classes or categories of entity.
Such as DataElement, OrgUnit etc, where the class of the entity
determines the set of attributes and the type of relations between
them.

I think from an xml metadata representation perspective, this is
probably how it should be modelled. The fact that some of the
attributes may be fixed database fields and some dynamically
configured attributes is incidental and purely historical. If you had
the benefit of hindsight and were designing a datadictionary from
scratch, you might never have had the fixed fields in the database.
Or you might have come up with different ones. On import/export I
think the distinction should simply be masked by the API. Fixed or
non-fixed is an internal arrangement of little interest outside the
system.

Bob

PS. One of the possible uses of attributes which I am seeing is the
in fact the possibility of dynamically adding new identifiers for
objects.

On 8 September 2011 15:55, Jo Størset <storset@gmail.com> wrote:

Thanks for the responses.

So I guess this might very well be a start of a general evolution in the meta model? It might be worth already now thinking a little through the possible consequences of this direction going forward. I think this is something we need :slight_smile:

As we work through how to represent the meta model in external formats (xml and so on), this might have implications on how generic a model we should create for these formats (and I guess the attribute definitions themselves have to represented in some way). In that context it might make sense to generalize both the objects these attributes apply to and the way we represent atrributes more generally. And also at least look at and specify likely future harmonization now?

Changing the external formats once they are created is much more difficult than changing implementation details, so it might be worth it to think a little bit broader already now.

A minor point - when I saw the term dynamic attributes, I was immediately thinking free-form attributes (more like tags you could put on individual org units). But I guess this is about predefined attribute types applied to (specific) meta objects(?) Maybe just calling it "attributes" might be good enough, if that is what it is?

Jo
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

I should also be able to make it.. Just give the time in advance :slight_smile:

Jo

···

Den 14. sep. 2011 kl. 15.32 skrev Morten Olav Hansen:

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Sure. I can tomorrow or Friday. Lars is in Uganda, but I'm sure he'll
respond later if he has time for a conference call.

OK. Lets try Friday morning 09h30 my time - that's 10h30 Oslo time

Dial-in number: +1-424-203-8000
or Skype call to: freeconferencing.8000
Code: 553472

I know you are under pressure to finish this stuff so the intent is
not to slow things up. Just keep us all pointing in same direction
:slight_smile:

Cheers
Bob

···

On 14 September 2011 14:40, Jo Størset <storset@gmail.com> wrote:

Den 14. sep. 2011 kl. 15.32 skrev Morten Olav Hansen:

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Sure. I can tomorrow or Friday. Lars is in Uganda, but I'm sure he'll
respond later if he has time for a conference call.

I should also be able to make it.. Just give the time in advance :slight_smile:

Jo

Hi All,

 I was just watching this thread and wanted to share our possibly

similar experiences from iHRIS. Admittedly I have not looked at the
details of how DHIS is implementing dynamic attributes in the
database, but I would suspect the design pattern is similar to how
iHRIS has done its form and fields. (I have also seen this design
pattern in openMRS).

Although this flexibility has been great, it did have a significant

impact when it came to reporting time. Even with proper indices,
MySQL couldn’t handle multiple joins and the reports quickly became
essentially unusable. As a solution we had to implement a middle
layer (form caching) which sits between the data storage system and
the reporting system. We were able to do this only only after
addaing a sufficient amount of meta-data about our fields and forms
(e.g. the relationship between them, the SQL data types, etc.)

I don't know how heavily you plan to make use of the dynamic

attributes in reporting, so it may be a non-issue for you all, but
it may be a topic for your design call.

Cheers.

-carl
···

storset@gmail.comhttps://launchpad.net/~dhis2-devsdhis2-devs@lists.launchpad.nethttps://launchpad.net/~dhis2-devshttps://help.launchpad.net/ListHelp

Hi All,
I was just watching this thread and wanted to share our possibly similar
experiences from iHRIS. Admittedly I have not looked at the details of how
DHIS is implementing dynamic attributes in the database, but I would
suspect the design pattern is similar to how iHRIS has done its form and
fields. (I have also seen this design pattern in openMRS).

Although this flexibility has been great, it did have a significant impact
when it came to reporting time. Even with proper indices, MySQL couldn't
handle multiple joins and the reports quickly became essentially unusable.
As a solution we had to implement a middle layer (form caching) which sits
between the data storage system and the reporting system. We were able to
do this only only after addaing a sufficient amount of meta-data about our
fields and forms (e.g. the relationship between them, the SQL data types,
etc.)

I don't know how heavily you plan to make use of the dynamic attributes in
reporting, so it may be a non-issue for you all, but it may be a topic for
your design call.

Hi Carl

Thanks for the observation. I am also not sure to what extent these
will be used in reporting - though I can only imagine they will be
used for something :slight_smile:

Reducing to the extreme we would simply have an identifier, the class
of the object and its attributes. And from an external perspective
this is a neat and orthogonal way of viewing them. I even recall
Murod has done something fairly similar. But as you point out, this
neat reductionism can come back to bite on the performance side. So
what is more likely to emerge as a pragmatic extension which builds on
the history of what is already there, is a hybrid of "core" attributes
and "dynamic" ones. The core attributes being hard coded columns in
tables and the dynamic ones being linked through database relations.
Which gives rise to the question of which should be core and which
should be dynamic and why. At first pass the answer to that is
probably straightforward - the core attributes already exist as column
names, and are already heavily used in reporting. In time we might
find a more rational basis to distinguish.

From an external exchange perspective (I'm trying to make my life

really simple) there is a level at which these are all simply
attributes. Whether a particular system, such as dhis, strategically,
historically or accidentally chooses to represent some of these as
database fields and some of them as dynamic attributes is an internal
detail to that system. So when I see an orgunit I might see something
like:

<orgunit identifier(s)=".. ".
   type="Clinic"
   name='Bob's Clinic'
   alternateName='Robert's Fancy Clinic for the bewildered'
   geoCode="..."
   contactPerson='Bob Jolliffe'
   bedcapacity=0 etc />

It is up to the consumer of such an element to decide how to actually
store this stuff internally and different systems might well do it
differently. In DHIS I guess we would collect up all the fixed fields
we have and store them, then take whatever is left and store them as
dynamic or extended attributes. From the producer side we want to be
able to identify the object and then dump all its attributes -
implementing from DHIS perspective would imply simply dumping all the
core attributes followed by whatever dynamic attributes it might have,
without necessarily exposing this detail in what gets spat out..

Managing the performance/flexibility tradeoff is tricky. Particularly
as we want to combine both the qualities of a performing data
warehouse and a flexible metadata repository into a single system. I
think the hybrid approach will take us so far. Its good to see Morten
working on it.

Cheers
Bob

···

On 14 September 2011 15:32, Carl Leitner <litlfred@ibiblio.org> wrote:

Cheers.
-carl

On 09/14/2011 10:13 AM, Bob Jolliffe wrote:

OK. Lets try Friday morning 09h30 my time - that's 10h30 Oslo time

Dial-in number: +1-424-203-8000
or Skype call to: freeconferencing.8000
Code: 553472

I know you are under pressure to finish this stuff so the intent is
not to slow things up. Just keep us all pointing in same direction
:slight_smile:

Cheers
Bob

On 14 September 2011 14:40, Jo Størset <storset@gmail.com> wrote:

Den 14. sep. 2011 kl. 15.32 skrev Morten Olav Hansen:

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Sure. I can tomorrow or Friday. Lars is in Uganda, but I'm sure he'll
respond later if he has time for a conference call.

I should also be able to make it.. Just give the time in advance :slight_smile:

Jo

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Final thought from me for now ...

Hi All,
I was just watching this thread and wanted to share our possibly similar
experiences from iHRIS. Admittedly I have not looked at the details of how
DHIS is implementing dynamic attributes in the database, but I would
suspect the design pattern is similar to how iHRIS has done its form and
fields. (I have also seen this design pattern in openMRS).

Although this flexibility has been great, it did have a significant impact
when it came to reporting time. Even with proper indices, MySQL couldn't
handle multiple joins and the reports quickly became essentially unusable.
As a solution we had to implement a middle layer (form caching) which sits
between the data storage system and the reporting system.

There are some places where we do something similar. Most notably the
DHIS resource tables can be thought of as something like this
specifically for reporting.

···

On 14 September 2011 17:11, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

On 14 September 2011 15:32, Carl Leitner <litlfred@ibiblio.org> wrote:

We were able to

do this only only after addaing a sufficient amount of meta-data about our
fields and forms (e.g. the relationship between them, the SQL data types,
etc.)

I don't know how heavily you plan to make use of the dynamic attributes in
reporting, so it may be a non-issue for you all, but it may be a topic for
your design call.

Hi Carl

Thanks for the observation. I am also not sure to what extent these
will be used in reporting - though I can only imagine they will be
used for something :slight_smile:

Reducing to the extreme we would simply have an identifier, the class
of the object and its attributes. And from an external perspective
this is a neat and orthogonal way of viewing them. I even recall
Murod has done something fairly similar. But as you point out, this
neat reductionism can come back to bite on the performance side. So
what is more likely to emerge as a pragmatic extension which builds on
the history of what is already there, is a hybrid of "core" attributes
and "dynamic" ones. The core attributes being hard coded columns in
tables and the dynamic ones being linked through database relations.
Which gives rise to the question of which should be core and which
should be dynamic and why. At first pass the answer to that is
probably straightforward - the core attributes already exist as column
names, and are already heavily used in reporting. In time we might
find a more rational basis to distinguish.

From an external exchange perspective (I'm trying to make my life
really simple) there is a level at which these are all simply
attributes. Whether a particular system, such as dhis, strategically,
historically or accidentally chooses to represent some of these as
database fields and some of them as dynamic attributes is an internal
detail to that system. So when I see an orgunit I might see something
like:

<orgunit identifier(s)=".. ".
type="Clinic"
name='Bob's Clinic'
alternateName='Robert's Fancy Clinic for the bewildered'
geoCode="..."
contactPerson='Bob Jolliffe'
bedcapacity=0 etc />

It is up to the consumer of such an element to decide how to actually
store this stuff internally and different systems might well do it
differently. In DHIS I guess we would collect up all the fixed fields
we have and store them, then take whatever is left and store them as
dynamic or extended attributes. From the producer side we want to be
able to identify the object and then dump all its attributes -
implementing from DHIS perspective would imply simply dumping all the
core attributes followed by whatever dynamic attributes it might have,
without necessarily exposing this detail in what gets spat out..

Managing the performance/flexibility tradeoff is tricky. Particularly
as we want to combine both the qualities of a performing data
warehouse and a flexible metadata repository into a single system. I
think the hybrid approach will take us so far. Its good to see Morten
working on it.

Cheers
Bob

Cheers.
-carl

On 09/14/2011 10:13 AM, Bob Jolliffe wrote:

OK. Lets try Friday morning 09h30 my time - that's 10h30 Oslo time

Dial-in number: +1-424-203-8000
or Skype call to: freeconferencing.8000
Code: 553472

I know you are under pressure to finish this stuff so the intent is
not to slow things up. Just keep us all pointing in same direction
:slight_smile:

Cheers
Bob

On 14 September 2011 14:40, Jo Størset <storset@gmail.com> wrote:

Den 14. sep. 2011 kl. 15.32 skrev Morten Olav Hansen:

Can I suggest we fit in a brief design call this week to touch base?
I have a freeconference number we can use. Maybe tomorrow or Friday?
Lars are you around and/or available as well?

Sure. I can tomorrow or Friday. Lars is in Uganda, but I'm sure he'll
respond later if he has time for a conference call.

I should also be able to make it.. Just give the time in advance :slight_smile:

Jo

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi,

···

Den 14. sep. 2011 kl. 16.13 skrev Bob Jolliffe:

OK. Lets try Friday morning 09h30 my time - that's 10h30 Oslo time

Dial-in number: +1-424-203-8000
or Skype call to: freeconferencing.8000
Code: 553472

I know you are under pressure to finish this stuff so the intent is
not to slow things up. Just keep us all pointing in same direction
:slight_smile:

just thought I'd give a quick recap of the meeting, but haven't really got time to write it up :slight_smile:

Morten's approach seems reasonable, once we cleared up some misunderstandings around the consistency of the model.

Only specific meta objects (indicators, org units, data elements..) will have attributes, and the values are stored in a per-object-type way. This makes it consistent, and fine as long as not too many types need attributes.

If it turns out that we want these attributes for many types, this apporach will not scale too well. So in that case we should revisit this design and abstract it into a more general implementation.

A decent recap, Bob and Morten?

Jo

A decent recap, Bob and Morten?

Yes, that seems to be a good recap. Its good enough.... for now :wink:

···

--
Morten

Jo

Could we also add "users" to the objects with dynamic attributes? We
need some more fields for alternative telephone numbers, fax, etc etc.

Thanks,
Jason

···

On Fri, Sep 16, 2011 at 12:23 PM, Morten Olav Hansen <mortenoh@gmail.com> wrote:

A decent recap, Bob and Morten?

Yes, that seems to be a good recap. Its good enough.... for now :wink:

--
Morten

Jo

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

A decent recap, Bob and Morten?

Yes thanks for calling in and good recap. Scope creep on which
objects can have extensible attributes could render it cumbersome
eventually, but while the number of such objects is few it should work
just fine.

One other thing we also discussed was how these attributes would be
serialized to xml. Given that there is no constraint on the attribute
name which would ensure that it is a 'good' xml element/attribute
name, we can't simply do the natural thing of:

<orgunit>
<name>Bobs Clinic</name>
...
<no_of_beds>0<no_of_beds>
</orginit>

So we will have to do:
<orgunit>
<name>Bobs Clinic</name>
...
<extensionAttributes>
  <attribute name="number of beds">0</attribute>
  <attribute name=...>...</attribute>
</extensionAttributes>
</orginit>

I think this *might* be ok. As long as it doesn't lead us into the
old mess of having "number of beds", "no of beds" etc as synonyms for
the same thing. But that's a background worry I have which is more to
do with that we are essentially using name as identifier again, but
maybe I'm worrying too much. Attribute is an identifiable object so
we could technically improve upon this in time.

Good discussion. 30 minutes. I hope people didn't think it was a
waste of time. I'm happy to host another one next week at 09h30 if
anyone has a pressing design issue which might benefit from a run
through.

Cheers
Bob

···

On 16 September 2011 11:23, Morten Olav Hansen <mortenoh@gmail.com> wrote:

Yes, that seems to be a good recap. Its good enough.... for now :wink:

--
Morten

Jo

Could we also add "users" to the objects with dynamic attributes? We
need some more fields for alternative telephone numbers, fax, etc etc.

and so it begins ...

Sounds reasonable to me :slight_smile: Or use ldap.

Bob

···

On 16 September 2011 11:54, Jason Pickering <jason.p.pickering@gmail.com> wrote:

Thanks,
Jason

On Fri, Sep 16, 2011 at 12:23 PM, Morten Olav Hansen <mortenoh@gmail.com> wrote:

A decent recap, Bob and Morten?

Yes, that seems to be a good recap. Its good enough.... for now :wink:

--
Morten

Jo

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp