I have brought up this discussion a couple of times before and good to see it moving now.
I like the idea of allowing multiple catcombos in one data entry form, and I think this should also go for default non-custom forms, as in many cases it would be a matter of listing multiple tables on a page, one table per catcombo.
We are facing exactly this challenge now that we are working on revising the Sierra Leone database for 2010. We want to use multidimensional data elements as much as possible and that typically means having 1-5 catcombos per paper form. At the same time we want this database to be compatible with the new SDMX-HD reporting so that it can easily work together with OpenMRS and other software. Then we need to make sure that our datasets only have 1 catcombo, if not the sdmx export will have trouble finding and structuring the data. Our problem then becomes how to create data entry forms that look exactly like the paper forms when such a form typically will have more then 1 catcombo? So we are under a bit of pressure here to be able to come up with such forms before they start entering data for 2010 in beginning of February next year.
I agree with Bob’s view that a data entry form is the combined thing, what you see and where you enter data, typically the same as the paper form. In order to support multiple catcombos (which to me means multiple datasets), then I think it is natural to be able to link a data entry form to n data sets. And a dataset can be used in multiple forms, that still needs to be supported. The datasets become the crucial database design part, where we define how data is modelled in the DHIS. The forms as I see it are simply views on top of that which can make data entry easier and be more like the paper experience.
It seems we are thinking along the same lines, just use different names for the various parts.
I think a group of data elements sharing the same catcombo should be a dataset, as this also best matches the terminology of SDMX-HD where one dataset will contain data that share the same dimensions.
**Whatever the names we go for my main requirements are:**1) need for an artefact that groups together data elements that share the same category combo and also define the allowed periodicity (periodType) for these data elements
when entering data it should be possible to see multiple tables with different dimensions in the same form
these tables should be generated by default (like today for 1 catcombo) from catcombos without a need to design them manually
it should be able to make minor adjustments to what the default data entry form looks (not completely custom form) by e.g. adding headings for each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table)
Like today offer a custom form designer with full freedom to link to data elements+catoptioncombos from multiple catcombos
When exporting data we again need easily to be able to pick data elements that share the same category combo (“sdmx datasets”)
In report tables we need to filter by category combos and get options from these (already implemented)
It also seems like this improvement would be equally important to both routine and patient data.
Ola
···
I like the idea of separating forms (or screens) from data sets and to allow multiple data sets (or groups of data elements with the same catcombo) in one form.
I like your suggestion very much but want (again I know!) to suggest a slightly different slant. I tend to think of a form as being the composite thing (like a paper form which might even have a reference number - form ANC-B3). And that form might have different screens and/or sections. Rather than the screen having multiple forms. Just a matter of naming I know, but if we are going to refactor like this, we do need to think about what is the least confusing to users.
Otherwise I am fully behind the suggestion of subsetting by common categorycombos - it does make sense. Possibly we need to implement a getDataElementsByCategoryCombo() method on DataSet to make it easier to select appropriate dataelements.
What I wanted to say is - think of separating dataentry screens from dataentry forms they shouldn’t be the same. A dataentry screen contains dataentryforms. And there shouldn’t be any object called dataentryScreen but only dataentryForm. Dataentry screen is more of a presentation and something to be implemented using action classes at the web layer.
When you have a dataset having dataelements with different category combos then difficult to display them using a single table. So what I am suggesting is to group these dataelements (those having the same catcombo) and create them dataentryform… like this a dataset can have have multiple dataentry forms. DataEntry form is to have a group of dataelements all sharing same categorycombo. Finally the dataset is free to have a set of dataforms, but only one dataentryscreen. I have attached you a sample form … we need to handle such a requirement.
You are saying one dataset say dataset1 can have multiple dataentry screens like dataentryscreen1,dataentryscreen2,dataentryscreen3.
When we select dataset1 in the dataentry module we should display these 3 screens for dataentry?
These 3 screens can have same dataelements or different dataelements?
On Wed, Dec 9, 2009 at 9:53 PM, Abyot Gizaw abyota@gmail.com wrote:
and also assigning multiple forms for datasets or programstages. For example a dataset might have dataelements having different categorycombos… for such cases it makes sense to group those datalements having the same categorycombo together then create a form… in the end such datasets will have a set of forms. So when displaying the dataentry screen we can display a list of forms and come up a nice looking dataentry screen.
Regarding DataEntryScreen, as we (Abyot & Me) discussed on his last visit we agreed to make CustomDataEntryScreen as Genric.
For that what we started is:
We changed DataEntryForm object to contain its id, name and html code only. (before it was bound with datasetid).
All the CustomScreen associations we kept in another table dataentryformassociation which contains associationtablename( represents which object DataSet/ProgramStage etc), associationid ( represents datasetid/programstage id), dataentryformid.
From the GUI:
If you goto dataset screen there will be icon to design custom screen for dataset like before. only change is before it was saving in dataentryform table. now it will save the design in dataentryform table and association will be saved in dataentryformassociation table.
Similarly for ProgramState also we can design screens.
Breifly that is what we are doing regarding CustomScreens. Is it OK or any changes?
I am sorry, the message of that commit was missing some information.
I had some problem while committing the source. It was locked, and I tried to use the break-lock command many times and finally it worked. That is why I did not write the full commit message.
So, that commit include
Custom DataEntryForm design form for ProgramStage ( this is what the FCK editor is for ).
I Added class DataEntryFormAssociation. And yes Abyot, when I was creating that class, I also wanted to create another package for all the dataentryform things. But that was like just a few days I touch the dhis code and I did not want to change so much. I just tried to follow the old work flow. Please do not worry, after read your email, I changed it immediately ( really happy to do that ) . You will see it in the next commit.
Custom data entry screen for case ( patient ) . This is not completely done, I am still working on it.
The reason for this commit is to merge my local changes with your trunk. Because I have been coding for about two weeks on this module with my local source code, not on any branch…so it really need to be merged as soon as possible.
And actually , I have not been in the dhis2-devs@lists.launchpad.net util Abyot send me and Bharath the email. I have just subscribed to the list yesterday. I did not know about that mailing list. When you added me to the core-dev team, I looked for mailing list of this team, but I could not.
About Abyot 's question :
The other thing, if you remember we even talked about having a set of forms for datasets/programstages. Because sometimes you might have multiple tables/sections. Do you think you can include these things?
Sorry, I do not know about this… Please explain it again if you have time.
Thank you,
Thanks Viet for your reply… Don’t misunderstand, we’re happy to see u contributing and I’m exited to see the new code/functionality. Its just important that Abyot understands whats going on and gets the change to review things as he is leading the patient system work… Now that you have enlisted for the dev list that’s will be easier:-) Tell me if you there is other stuff u are wondering about.
I am sorry I have set you off on this path of datasets with common categorycombos but I am now having doubts … see below.
Hi,
I have brought up this discussion a couple of times before and good to see it moving now.
I like the idea of allowing multiple catcombos in one data entry form, and I think this should also go for default non-custom forms, as in many cases it would be a matter of listing multiple tables on a page, one table per catcombo.
We are facing exactly this challenge now that we are working on revising the Sierra Leone database for 2010. We want to use multidimensional data elements as much as possible and that typically means having 1-5 catcombos per paper form. At the same time we want this database to be compatible with the new SDMX-HD reporting so that it can easily work together with OpenMRS and other software. Then we need to make sure that our datasets only have 1 catcombo, if not the sdmx export will have trouble finding and structuring the data. Our problem then becomes how to create data entry forms that look exactly like the paper forms when such a form typically will have more then 1 catcombo? So we are under a bit of pressure here to be able to come up with such forms before they start entering data for 2010 in beginning of February next year.
I agree with Bob’s view that a data entry form is the combined thing, what you see and where you enter data, typically the same as the paper form. In order to support multiple catcombos (which to me means multiple datasets), then I think it is natural to be able to link a data entry form to n data sets. And a dataset can be used in multiple forms, that still needs to be supported. The datasets become the crucial database design part, where we define how data is modelled in the DHIS. The forms as I see it are simply views on top of that which can make data entry easier and be more like the paper experience.
After considerable reflection I have softened in some respects on this on hardened in others. I do think we need to have a notion of a set of reported data - whether that data has arrived through a data entry screen/form or sucked in from the ether. That is a particular set of datavalues rather than a set of dataelements. The Dataset has always been approximately (but not quite) this abstraction - I guess I am talking about a Dataset instance - a ReportedDataSet. I think this is in some ways similar to our completedDataSetRegistration.
But I have also come to understand the fact that SDMX-HD datasets are obliged to be of the same dimensionality as being a limitation of SDMX and not necessarily the shining example of best practice in our field. When we read in data (say an sdmx package from openmrs which might be a group of SDMX datasets) its not really that important that we have to match 1-1 those SDMX datasets onto DHIS datasets. In fact there is no loss of meaningful information if they all go into the same DHIS dataset. As long as we can pull back out values byCategoryCombo() then that is fine.
And I suspect the same thing is true in reverse - when we are reporting data rather than receiving reported data. That is a more complicated problem which I haven’t fully got my head around yet (its different because we would be dealing mostly with aggregated indicator values with no categorycombos) but again DHIS just needs to produce the reported data set. If that data has to be subdivided further to parcel into SDMX messages that is a downstream issue which shouldn’t necessarily affect constraints on our model.
So what about the forms/screens? Well I think we must cleanly separate model-view issues. The requirement to group dataentry elements in different sized tables is primarily a UI issue which should be taken care of at the UI layer. The UI might have screens with forms or screens with sections or what have you. And those sections might have a common dimensionality requirement but I don’t think this has necessarily got to do with how we model datasets. Currently I think sections are a property of the DataSet - the SectionManager has getSectionsByDataSet() for example. This I am now suspecting is the wrong approach. One data set for reporting instrument (whether we call that a screen or a form is immaterial as long as we agree on something) makes sense. And each section on that form should have a requirement that only dataelements from the dataset which share the same dimensionality are eligle for inclusion in that section. But these are sections on the form not anything structurally meaningful to the DataSet. I think this can be enforced without fragmenting our current model.
So in summary (my list):
discount my earlier concerns about sdmx datasets - they are not the same as dhis datasets and we can workaround the difference.
think about something to represent an actual set of reported (electronic or screen-entered) values. There is metadata regarding this from sdmx which we might currently have to discard. It might be a refinement of completedDataSetRegistartion.
sections on a form should have the constraint that they only allow dataelements of similar dimensionality.
If the UI (the web forms) requires a breakdown of datasets by categorycombo (or dimensionality) for the purpose of layout then keep that mechanism in the view layer. Sometimes the view needs its own micro-model …
It seems we are thinking along the same lines, just use different names for the various parts.
I think a group of data elements sharing the same catcombo should be a dataset, as this also best matches the terminology of SDMX-HD where one dataset will contain data that share the same dimensions.
**Whatever the names we go for my main requirements are:**1) need for an artefact that groups together data elements that share the same category combo and also define the allowed periodicity (periodType) for these data elements
when entering data it should be possible to see multiple tables with different dimensions in the same form
these tables should be generated by default (like today for 1 catcombo) from catcombos without a need to design them manually
it should be able to make minor adjustments to what the default data entry form looks (not completely custom form) by e.g. adding headings for each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table)
Like today offer a custom form designer with full freedom to link to data elements+catoptioncombos from multiple catcombos
When exporting data we again need easily to be able to pick data elements that share the same category combo (“sdmx datasets”)
In report tables we need to filter by category combos and get options from these (already implemented)
It also seems like this improvement would be equally important to both routine and patient data.
Ola
I like the idea of separating forms (or screens) from data sets and to allow multiple data sets (or groups of data elements with the same catcombo) in one form.
I like your suggestion very much but want (again I know!) to suggest a slightly different slant. I tend to think of a form as being the composite thing (like a paper form which might even have a reference number - form ANC-B3). And that form might have different screens and/or sections. Rather than the screen having multiple forms. Just a matter of naming I know, but if we are going to refactor like this, we do need to think about what is the least confusing to users.
Otherwise I am fully behind the suggestion of subsetting by common categorycombos - it does make sense. Possibly we need to implement a getDataElementsByCategoryCombo() method on DataSet to make it easier to select appropriate dataelements.
What I wanted to say is - think of separating dataentry screens from dataentry forms they shouldn’t be the same. A dataentry screen contains dataentryforms. And there shouldn’t be any object called dataentryScreen but only dataentryForm. Dataentry screen is more of a presentation and something to be implemented using action classes at the web layer.
When you have a dataset having dataelements with different category combos then difficult to display them using a single table. So what I am suggesting is to group these dataelements (those having the same catcombo) and create them dataentryform… like this a dataset can have have multiple dataentry forms. DataEntry form is to have a group of dataelements all sharing same categorycombo. Finally the dataset is free to have a set of dataforms, but only one dataentryscreen. I have attached you a sample form … we need to handle such a requirement.
You are saying one dataset say dataset1 can have multiple dataentry screens like dataentryscreen1,dataentryscreen2,dataentryscreen3.
When we select dataset1 in the dataentry module we should display these 3 screens for dataentry?
These 3 screens can have same dataelements or different dataelements?
On Wed, Dec 9, 2009 at 9:53 PM, Abyot Gizaw abyota@gmail.com wrote:
and also assigning multiple forms for datasets or programstages. For example a dataset might have dataelements having different categorycombos… for such cases it makes sense to group those datalements having the same categorycombo together then create a form… in the end such datasets will have a set of forms. So when displaying the dataentry screen we can display a list of forms and come up a nice looking dataentry screen.
Regarding DataEntryScreen, as we (Abyot & Me) discussed on his last visit we agreed to make CustomDataEntryScreen as Genric.
For that what we started is:
We changed DataEntryForm object to contain its id, name and html code only. (before it was bound with datasetid).
All the CustomScreen associations we kept in another table dataentryformassociation which contains associationtablename( represents which object DataSet/ProgramStage etc), associationid ( represents datasetid/programstage id), dataentryformid.
From the GUI:
If you goto dataset screen there will be icon to design custom screen for dataset like before. only change is before it was saving in dataentryform table. now it will save the design in dataentryform table and association will be saved in dataentryformassociation table.
Similarly for ProgramState also we can design screens.
Breifly that is what we are doing regarding CustomScreens. Is it OK or any changes?
I am sorry, the message of that commit was missing some information.
I had some problem while committing the source. It was locked, and I tried to use the break-lock command many times and finally it worked. That is why I did not write the full commit message.
So, that commit include
Custom DataEntryForm design form for ProgramStage ( this is what the FCK editor is for ).
I Added class DataEntryFormAssociation. And yes Abyot, when I was creating that class, I also wanted to create another package for all the dataentryform things. But that was like just a few days I touch the dhis code and I did not want to change so much. I just tried to follow the old work flow. Please do not worry, after read your email, I changed it immediately ( really happy to do that ) . You will see it in the next commit.
Custom data entry screen for case ( patient ) . This is not completely done, I am still working on it.
The reason for this commit is to merge my local changes with your trunk. Because I have been coding for about two weeks on this module with my local source code, not on any branch…so it really need to be merged as soon as possible.
And actually , I have not been in the dhis2-devs@lists.launchpad.net util Abyot send me and Bharath the email. I have just subscribed to the list yesterday. I did not know about that mailing list. When you added me to the core-dev team, I looked for mailing list of this team, but I could not.
About Abyot 's question :
The other thing, if you remember we even talked about having a set of forms for datasets/programstages. Because sometimes you might have multiple tables/sections. Do you think you can include these things?
Sorry, I do not know about this… Please explain it again if you have time.
Thank you,
Thanks Viet for your reply… Don’t misunderstand, we’re happy to see u contributing and I’m exited to see the new code/functionality. Its just important that Abyot understands whats going on and gets the change to review things as he is leading the patient system work… Now that you have enlisted for the dev list that’s will be easier:-) Tell me if you there is other stuff u are wondering about.
After some more testing of data set, data set section and data entry forms and Bob’s more relaxed attitude on sdmx vs DHIS models I am a bit more positive and have some suggestions (from an implementer’s perspective) on short and long term improvements to data sets and forms.
Some general thought on this: * For the time being the data set sections seem like a suitable object were we can group data elements that belong to the same catcombo, for use in data entry forms, and for data export purposes. They play the role of form sections.
I managed to create a dataset with mutliple catcombos, put these in separate sections, and then designed a custom form for data entry that included multiple tables/catcombos. I can even combine multiple combos into one monster table if I like, it’s all flexible down to the “data element + catoptioncombo” level. So for entering in custom forms and storing dimensional data the current model seem to work ok (not thinking of sdmx).
Some short term improvements (by early January latest to support new forms for next year):
Right now the data set sections are not supported in the default data entry forms, which means you have to design custom forms to make use of them. This should be fixed as it should be possible to have default forms with more than 1 catcombo/table. A simple version would be to list these tables 1 by 1, under each other with a table heading taken from the data set section label property (which already exists for some reason).
(Bob’s) Sections on a form should have the constraint that they only allow dataelements of similar dimensionality. (Ola’s): Right now there is no such constraint. If not, it will all be chaos in default tabular forms and messy for sdmx I assume.
When creating a new dataset with data elements from different catcombos the data set sections should automatically be created, one per catcombo and data elements allocated to these sections.
Same as above for data set reports (forms with data for any level and period), so that these static reports look exactly like the forms.
Long term (3-4 months maybe?):* Find a better name for data set section, possibly remodel a dataset, form, data value set model (to cater for data entry in default and custom mode, import/export dhis2dhis, and reporting via sdmx)
It should be possible to make minor adjustments to what the default data entry forms look like (not completely custom form). Default data entry form view properties should include things like adding headings to each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table). In a more advanced version (>3 months) even be able to position these tables on a screen grid e.g. 3x3 or 2x2, like the drag and drop portal stuff that’s around, e.g iGoogle.
Data set reports should make use of the view properties to default forms (the previous point) so that they look exactly like the forms
Even more flexible data entry forms with multiple orgunits or periods and possibility to pivot these around in a table
It should be possible to create forms that combine data for a facility, a household and a patient/client. Saptarshi had some ideas on how Xforms might support that.
There’s probably much more, but that’s what I’ve got from an implementer’s perspective at least. I’m sure you guys have comments and additions. Let’s get this discussion done quickly and into a blueprint and into action so that all the people working on creating new data sets and forms for 2010 can get their work done more easily.
I have brought up this discussion a couple of times before and good to see it moving now.
I like the idea of allowing multiple catcombos in one data entry form, and I think this should also go for default non-custom forms, as in many cases it would be a matter of listing multiple tables on a page, one table per catcombo.
We are facing exactly this challenge now that we are working on revising the Sierra Leone database for 2010. We want to use multidimensional data elements as much as possible and that typically means having 1-5 catcombos per paper form. At the same time we want this database to be compatible with the new SDMX-HD reporting so that it can easily work together with OpenMRS and other software. Then we need to make sure that our datasets only have 1 catcombo, if not the sdmx export will have trouble finding and structuring the data. Our problem then becomes how to create data entry forms that look exactly like the paper forms when such a form typically will have more then 1 catcombo? So we are under a bit of pressure here to be able to come up with such forms before they start entering data for 2010 in beginning of February next year.
I agree with Bob’s view that a data entry form is the combined thing, what you see and where you enter data, typically the same as the paper form. In order to support multiple catcombos (which to me means multiple datasets), then I think it is natural to be able to link a data entry form to n data sets. And a dataset can be used in multiple forms, that still needs to be supported. The datasets become the crucial database design part, where we define how data is modelled in the DHIS. The forms as I see it are simply views on top of that which can make data entry easier and be more like the paper experience.
After considerable reflection I have softened in some respects on this on hardened in others. I do think we need to have a notion of a set of reported data - whether that data has arrived through a data entry screen/form or sucked in from the ether. That is a particular set of datavalues rather than a set of dataelements. The Dataset has always been approximately (but not quite) this abstraction - I guess I am talking about a Dataset instance - a ReportedDataSet. I think this is in some ways similar to our completedDataSetRegistration.
But I have also come to understand the fact that SDMX-HD datasets are obliged to be of the same dimensionality as being a limitation of SDMX and not necessarily the shining example of best practice in our field. When we read in data (say an sdmx package from openmrs which might be a group of SDMX datasets) its not really that important that we have to match 1-1 those SDMX datasets onto DHIS datasets. In fact there is no loss of meaningful information if they all go into the same DHIS dataset. As long as we can pull back out values byCategoryCombo() then that is fine.
And I suspect the same thing is true in reverse - when we are reporting data rather than receiving reported data. That is a more complicated problem which I haven’t fully got my head around yet (its different because we would be dealing mostly with aggregated indicator values with no categorycombos) but again DHIS just needs to produce the reported data set. If that data has to be subdivided further to parcel into SDMX messages that is a downstream issue which shouldn’t necessarily affect constraints on our model.
So what about the forms/screens? Well I think we must cleanly separate model-view issues. The requirement to group dataentry elements in different sized tables is primarily a UI issue which should be taken care of at the UI layer. The UI might have screens with forms or screens with sections or what have you. And those sections might have a common dimensionality requirement but I don’t think this has necessarily got to do with how we model datasets. Currently I think sections are a property of the DataSet - the SectionManager has getSectionsByDataSet() for example. This I am now suspecting is the wrong approach. One data set for reporting instrument (whether we call that a screen or a form is immaterial as long as we agree on something) makes sense. And each section on that form should have a requirement that only dataelements from the dataset which share the same dimensionality are eligle for inclusion in that section. But these are sections on the form not anything structurally meaningful to the DataSet. I think this can be enforced without fragmenting our current model.
So in summary (my list):
discount my earlier concerns about sdmx datasets - they are not the same as dhis datasets and we can workaround the difference.
think about something to represent an actual set of reported (electronic or screen-entered) values. There is metadata regarding this from sdmx which we might currently have to discard. It might be a refinement of completedDataSetRegistartion.
sections on a form should have the constraint that they only allow dataelements of similar dimensionality.
If the UI (the web forms) requires a breakdown of datasets by categorycombo (or dimensionality) for the purpose of layout then keep that mechanism in the view layer. Sometimes the view needs its own micro-model …
Bob.
It seems we are thinking along the same lines, just use different names for the various parts.
I think a group of data elements sharing the same catcombo should be a dataset, as this also best matches the terminology of SDMX-HD where one dataset will contain data that share the same dimensions.
**Whatever the names we go for my main requirements are:**1) need for an artefact that groups together data elements that share the same category combo and also define the allowed periodicity (periodType) for these data elements
when entering data it should be possible to see multiple tables with different dimensions in the same form
these tables should be generated by default (like today for 1 catcombo) from catcombos without a need to design them manually
it should be able to make minor adjustments to what the default data entry form looks (not completely custom form) by e.g. adding headings for each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table)
Like today offer a custom form designer with full freedom to link to data elements+catoptioncombos from multiple catcombos
When exporting data we again need easily to be able to pick data elements that share the same category combo (“sdmx datasets”)
In report tables we need to filter by category combos and get options from these (already implemented)
It also seems like this improvement would be equally important to both routine and patient data.
Ola
I like the idea of separating forms (or screens) from data sets and to allow multiple data sets (or groups of data elements with the same catcombo) in one form.
I like your suggestion very much but want (again I know!) to suggest a slightly different slant. I tend to think of a form as being the composite thing (like a paper form which might even have a reference number - form ANC-B3). And that form might have different screens and/or sections. Rather than the screen having multiple forms. Just a matter of naming I know, but if we are going to refactor like this, we do need to think about what is the least confusing to users.
Otherwise I am fully behind the suggestion of subsetting by common categorycombos - it does make sense. Possibly we need to implement a getDataElementsByCategoryCombo() method on DataSet to make it easier to select appropriate dataelements.
What I wanted to say is - think of separating dataentry screens from dataentry forms they shouldn’t be the same. A dataentry screen contains dataentryforms. And there shouldn’t be any object called dataentryScreen but only dataentryForm. Dataentry screen is more of a presentation and something to be implemented using action classes at the web layer.
When you have a dataset having dataelements with different category combos then difficult to display them using a single table. So what I am suggesting is to group these dataelements (those having the same catcombo) and create them dataentryform… like this a dataset can have have multiple dataentry forms. DataEntry form is to have a group of dataelements all sharing same categorycombo. Finally the dataset is free to have a set of dataforms, but only one dataentryscreen. I have attached you a sample form … we need to handle such a requirement.
You are saying one dataset say dataset1 can have multiple dataentry screens like dataentryscreen1,dataentryscreen2,dataentryscreen3.
When we select dataset1 in the dataentry module we should display these 3 screens for dataentry?
These 3 screens can have same dataelements or different dataelements?
On Wed, Dec 9, 2009 at 9:53 PM, Abyot Gizaw abyota@gmail.com wrote:
and also assigning multiple forms for datasets or programstages. For example a dataset might have dataelements having different categorycombos… for such cases it makes sense to group those datalements having the same categorycombo together then create a form… in the end such datasets will have a set of forms. So when displaying the dataentry screen we can display a list of forms and come up a nice looking dataentry screen.
Regarding DataEntryScreen, as we (Abyot & Me) discussed on his last visit we agreed to make CustomDataEntryScreen as Genric.
For that what we started is:
We changed DataEntryForm object to contain its id, name and html code only. (before it was bound with datasetid).
All the CustomScreen associations we kept in another table dataentryformassociation which contains associationtablename( represents which object DataSet/ProgramStage etc), associationid ( represents datasetid/programstage id), dataentryformid.
From the GUI:
If you goto dataset screen there will be icon to design custom screen for dataset like before. only change is before it was saving in dataentryform table. now it will save the design in dataentryform table and association will be saved in dataentryformassociation table.
Similarly for ProgramState also we can design screens.
Breifly that is what we are doing regarding CustomScreens. Is it OK or any changes?
I am sorry, the message of that commit was missing some information.
I had some problem while committing the source. It was locked, and I tried to use the break-lock command many times and finally it worked. That is why I did not write the full commit message.
So, that commit include
Custom DataEntryForm design form for ProgramStage ( this is what the FCK editor is for ).
I Added class DataEntryFormAssociation. And yes Abyot, when I was creating that class, I also wanted to create another package for all the dataentryform things. But that was like just a few days I touch the dhis code and I did not want to change so much. I just tried to follow the old work flow. Please do not worry, after read your email, I changed it immediately ( really happy to do that ) . You will see it in the next commit.
Custom data entry screen for case ( patient ) . This is not completely done, I am still working on it.
The reason for this commit is to merge my local changes with your trunk. Because I have been coding for about two weeks on this module with my local source code, not on any branch…so it really need to be merged as soon as possible.
And actually , I have not been in the dhis2-devs@lists.launchpad.net util Abyot send me and Bharath the email. I have just subscribed to the list yesterday. I did not know about that mailing list. When you added me to the core-dev team, I looked for mailing list of this team, but I could not.
About Abyot 's question :
The other thing, if you remember we even talked about having a set of forms for datasets/programstages. Because sometimes you might have multiple tables/sections. Do you think you can include these things?
Sorry, I do not know about this… Please explain it again if you have time.
Thank you,
Thanks Viet for your reply… Don’t misunderstand, we’re happy to see u contributing and I’m exited to see the new code/functionality. Its just important that Abyot understands whats going on and gets the change to review things as he is leading the patient system work… Now that you have enlisted for the dev list that’s will be easier:-) Tell me if you there is other stuff u are wondering about.
After some more testing of data set, data set section and data entry forms and Bob’s more relaxed attitude on sdmx vs DHIS models I am a bit more positive and have some suggestions (from an implementer’s perspective) on short and long term improvements to data sets and forms.
Some general thought on this: * For the time being the data set sections seem like a suitable object were we can group data elements that belong to the same catcombo, for use in data entry forms, and for data export purposes. They play the role of form sections.
I managed to create a dataset with mutliple catcombos, put these in separate sections, and then designed a custom form for data entry that included multiple tables/catcombos. I can even combine multiple combos into one monster table if I like, it’s all flexible down to the “data element + catoptioncombo” level. So for entering in custom forms and storing dimensional data the current model seem to work ok (not thinking of sdmx).
Some short term improvements (by early January latest to support new forms for next year):
Right now the data set sections are not supported in the default data entry forms, which means you have to design custom forms to make use of them. This should be fixed as it should be possible to have default forms with more than 1 catcombo/table. A simple version would be to list these tables 1 by 1, under each other with a table heading taken from the data set section label property (which already exists for some reason).
(Bob’s) Sections on a form should have the constraint that they only allow dataelements of similar dimensionality. (Ola’s): Right now there is no such constraint. If not, it will all be chaos in default tabular forms and messy for sdmx I assume.
When creating a new dataset with data elements from different catcombos the data set sections should automatically be created, one per catcombo and data elements allocated to these sections.
Same as above for data set reports (forms with data for any level and period), so that these static reports look exactly like the forms.
Long term (3-4 months maybe?):* Find a better name for data set section, possibly remodel a dataset, form, data value set model (to cater for data entry in default and custom mode, import/export dhis2dhis, and reporting via sdmx)
It should be possible to make minor adjustments to what the default data entry forms look like (not completely custom form). Default data entry form view properties should include things like adding headings to each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table). In a more advanced version (>3 months) even be able to position these tables on a screen grid e.g. 3x3 or 2x2, like the drag and drop portal stuff that’s around, e.g iGoogle.
Data set reports should make use of the view properties to default forms (the previous point) so that they look exactly like the forms
Even more flexible data entry forms with multiple orgunits or periods and possibility to pivot these around in a table
It should be possible to create forms that combine data for a facility, a household and a patient/client. Saptarshi had some ideas on how Xforms might support that.
I didn’t get to that discussion with Saptarshi in Cape Town but an important point underlying this is that XForms has an instance model to represent the form data which can be more flexible and forms-oriented than the underlying persistence model.
There’s probably much more, but that’s what I’ve got from an implementer’s perspective at least. I’m sure you guys have comments and additions. Let’s get this discussion done quickly and into a blueprint and into action so that all the people working on creating new data sets and forms for 2010 can get their work done more easily.
I have brought up this discussion a couple of times before and good to see it moving now.
I like the idea of allowing multiple catcombos in one data entry form, and I think this should also go for default non-custom forms, as in many cases it would be a matter of listing multiple tables on a page, one table per catcombo.
We are facing exactly this challenge now that we are working on revising the Sierra Leone database for 2010. We want to use multidimensional data elements as much as possible and that typically means having 1-5 catcombos per paper form. At the same time we want this database to be compatible with the new SDMX-HD reporting so that it can easily work together with OpenMRS and other software. Then we need to make sure that our datasets only have 1 catcombo, if not the sdmx export will have trouble finding and structuring the data. Our problem then becomes how to create data entry forms that look exactly like the paper forms when such a form typically will have more then 1 catcombo? So we are under a bit of pressure here to be able to come up with such forms before they start entering data for 2010 in beginning of February next year.
I agree with Bob’s view that a data entry form is the combined thing, what you see and where you enter data, typically the same as the paper form. In order to support multiple catcombos (which to me means multiple datasets), then I think it is natural to be able to link a data entry form to n data sets. And a dataset can be used in multiple forms, that still needs to be supported. The datasets become the crucial database design part, where we define how data is modelled in the DHIS. The forms as I see it are simply views on top of that which can make data entry easier and be more like the paper experience.
After considerable reflection I have softened in some respects on this on hardened in others. I do think we need to have a notion of a set of reported data - whether that data has arrived through a data entry screen/form or sucked in from the ether. That is a particular set of datavalues rather than a set of dataelements. The Dataset has always been approximately (but not quite) this abstraction - I guess I am talking about a Dataset instance - a ReportedDataSet. I think this is in some ways similar to our completedDataSetRegistration.
But I have also come to understand the fact that SDMX-HD datasets are obliged to be of the same dimensionality as being a limitation of SDMX and not necessarily the shining example of best practice in our field. When we read in data (say an sdmx package from openmrs which might be a group of SDMX datasets) its not really that important that we have to match 1-1 those SDMX datasets onto DHIS datasets. In fact there is no loss of meaningful information if they all go into the same DHIS dataset. As long as we can pull back out values byCategoryCombo() then that is fine.
And I suspect the same thing is true in reverse - when we are reporting data rather than receiving reported data. That is a more complicated problem which I haven’t fully got my head around yet (its different because we would be dealing mostly with aggregated indicator values with no categorycombos) but again DHIS just needs to produce the reported data set. If that data has to be subdivided further to parcel into SDMX messages that is a downstream issue which shouldn’t necessarily affect constraints on our model.
So what about the forms/screens? Well I think we must cleanly separate model-view issues. The requirement to group dataentry elements in different sized tables is primarily a UI issue which should be taken care of at the UI layer. The UI might have screens with forms or screens with sections or what have you. And those sections might have a common dimensionality requirement but I don’t think this has necessarily got to do with how we model datasets. Currently I think sections are a property of the DataSet - the SectionManager has getSectionsByDataSet() for example. This I am now suspecting is the wrong approach. One data set for reporting instrument (whether we call that a screen or a form is immaterial as long as we agree on something) makes sense. And each section on that form should have a requirement that only dataelements from the dataset which share the same dimensionality are eligle for inclusion in that section. But these are sections on the form not anything structurally meaningful to the DataSet. I think this can be enforced without fragmenting our current model.
So in summary (my list):
discount my earlier concerns about sdmx datasets - they are not the same as dhis datasets and we can workaround the difference.
think about something to represent an actual set of reported (electronic or screen-entered) values. There is metadata regarding this from sdmx which we might currently have to discard. It might be a refinement of completedDataSetRegistartion.
sections on a form should have the constraint that they only allow dataelements of similar dimensionality.
If the UI (the web forms) requires a breakdown of datasets by categorycombo (or dimensionality) for the purpose of layout then keep that mechanism in the view layer. Sometimes the view needs its own micro-model …
Bob.
It seems we are thinking along the same lines, just use different names for the various parts.
I think a group of data elements sharing the same catcombo should be a dataset, as this also best matches the terminology of SDMX-HD where one dataset will contain data that share the same dimensions.
**Whatever the names we go for my main requirements are:**1) need for an artefact that groups together data elements that share the same category combo and also define the allowed periodicity (periodType) for these data elements
when entering data it should be possible to see multiple tables with different dimensions in the same form
these tables should be generated by default (like today for 1 catcombo) from catcombos without a need to design them manually
it should be able to make minor adjustments to what the default data entry form looks (not completely custom form) by e.g. adding headings for each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table)
Like today offer a custom form designer with full freedom to link to data elements+catoptioncombos from multiple catcombos
When exporting data we again need easily to be able to pick data elements that share the same category combo (“sdmx datasets”)
In report tables we need to filter by category combos and get options from these (already implemented)
It also seems like this improvement would be equally important to both routine and patient data.
Ola
I like the idea of separating forms (or screens) from data sets and to allow multiple data sets (or groups of data elements with the same catcombo) in one form.
I like your suggestion very much but want (again I know!) to suggest a slightly different slant. I tend to think of a form as being the composite thing (like a paper form which might even have a reference number - form ANC-B3). And that form might have different screens and/or sections. Rather than the screen having multiple forms. Just a matter of naming I know, but if we are going to refactor like this, we do need to think about what is the least confusing to users.
Otherwise I am fully behind the suggestion of subsetting by common categorycombos - it does make sense. Possibly we need to implement a getDataElementsByCategoryCombo() method on DataSet to make it easier to select appropriate dataelements.
What I wanted to say is - think of separating dataentry screens from dataentry forms they shouldn’t be the same. A dataentry screen contains dataentryforms. And there shouldn’t be any object called dataentryScreen but only dataentryForm. Dataentry screen is more of a presentation and something to be implemented using action classes at the web layer.
When you have a dataset having dataelements with different category combos then difficult to display them using a single table. So what I am suggesting is to group these dataelements (those having the same catcombo) and create them dataentryform… like this a dataset can have have multiple dataentry forms. DataEntry form is to have a group of dataelements all sharing same categorycombo. Finally the dataset is free to have a set of dataforms, but only one dataentryscreen. I have attached you a sample form … we need to handle such a requirement.
You are saying one dataset say dataset1 can have multiple dataentry screens like dataentryscreen1,dataentryscreen2,dataentryscreen3.
When we select dataset1 in the dataentry module we should display these 3 screens for dataentry?
These 3 screens can have same dataelements or different dataelements?
On Wed, Dec 9, 2009 at 9:53 PM, Abyot Gizaw abyota@gmail.com wrote:
and also assigning multiple forms for datasets or programstages. For example a dataset might have dataelements having different categorycombos… for such cases it makes sense to group those datalements having the same categorycombo together then create a form… in the end such datasets will have a set of forms. So when displaying the dataentry screen we can display a list of forms and come up a nice looking dataentry screen.
Regarding DataEntryScreen, as we (Abyot & Me) discussed on his last visit we agreed to make CustomDataEntryScreen as Genric.
For that what we started is:
We changed DataEntryForm object to contain its id, name and html code only. (before it was bound with datasetid).
All the CustomScreen associations we kept in another table dataentryformassociation which contains associationtablename( represents which object DataSet/ProgramStage etc), associationid ( represents datasetid/programstage id), dataentryformid.
From the GUI:
If you goto dataset screen there will be icon to design custom screen for dataset like before. only change is before it was saving in dataentryform table. now it will save the design in dataentryform table and association will be saved in dataentryformassociation table.
Similarly for ProgramState also we can design screens.
Breifly that is what we are doing regarding CustomScreens. Is it OK or any changes?
I am sorry, the message of that commit was missing some information.
I had some problem while committing the source. It was locked, and I tried to use the break-lock command many times and finally it worked. That is why I did not write the full commit message.
So, that commit include
Custom DataEntryForm design form for ProgramStage ( this is what the FCK editor is for ).
I Added class DataEntryFormAssociation. And yes Abyot, when I was creating that class, I also wanted to create another package for all the dataentryform things. But that was like just a few days I touch the dhis code and I did not want to change so much. I just tried to follow the old work flow. Please do not worry, after read your email, I changed it immediately ( really happy to do that ) . You will see it in the next commit.
Custom data entry screen for case ( patient ) . This is not completely done, I am still working on it.
The reason for this commit is to merge my local changes with your trunk. Because I have been coding for about two weeks on this module with my local source code, not on any branch…so it really need to be merged as soon as possible.
And actually , I have not been in the dhis2-devs@lists.launchpad.net util Abyot send me and Bharath the email. I have just subscribed to the list yesterday. I did not know about that mailing list. When you added me to the core-dev team, I looked for mailing list of this team, but I could not.
About Abyot 's question :
The other thing, if you remember we even talked about having a set of forms for datasets/programstages. Because sometimes you might have multiple tables/sections. Do you think you can include these things?
Sorry, I do not know about this… Please explain it again if you have time.
Thank you,
Thanks Viet for your reply… Don’t misunderstand, we’re happy to see u contributing and I’m exited to see the new code/functionality. Its just important that Abyot understands whats going on and gets the change to review things as he is leading the patient system work… Now that you have enlisted for the dev list that’s will be easier:-) Tell me if you there is other stuff u are wondering about.
After some more testing of data set, data set section and data entry forms and Bob’s more relaxed attitude on sdmx vs DHIS models I am a bit more positive and have some suggestions (from an implementer’s perspective) on short and long term improvements to data sets and forms.
Some general thought on this: * For the time being the data set sections seem like a suitable object were we can group data elements that belong to the same catcombo, for use in data entry forms, and for data export purposes. They play the role of form sections.
I managed to create a dataset with mutliple catcombos, put these in separate sections, and then designed a custom form for data entry that included multiple tables/catcombos. I can even combine multiple combos into one monster table if I like, it’s all flexible down to the “data element + catoptioncombo” level. So for entering in custom forms and storing dimensional data the current model seem to work ok (not thinking of sdmx).
Some short term improvements (by early January latest to support new forms for next year):
Right now the data set sections are not supported in the default data entry forms, which means you have to design custom forms to make use of them. This should be fixed as it should be possible to have default forms with more than 1 catcombo/table. A simple version would be to list these tables 1 by 1, under each other with a table heading taken from the data set section label property (which already exists for some reason).
(Bob’s) Sections on a form should have the constraint that they only allow dataelements of similar dimensionality. (Ola’s): Right now there is no such constraint. If not, it will all be chaos in default tabular forms and messy for sdmx I assume.
When creating a new dataset with data elements from different catcombos the data set sections should automatically be created, one per catcombo and data elements allocated to these sections.
Same as above for data set reports (forms with data for any level and period), so that these static reports look exactly like the forms.
Long term (3-4 months maybe?):* Find a better name for data set section, possibly remodel a dataset, form, data value set model (to cater for data entry in default and custom mode, import/export dhis2dhis, and reporting via sdmx)
It should be possible to make minor adjustments to what the default data entry forms look like (not completely custom form). Default data entry form view properties should include things like adding headings to each table (catcombo), selecting which fields to grey out (make inactive), freely select whether to put data element or catoptions on columns/rows (pivot the table). In a more advanced version (>3 months) even be able to position these tables on a screen grid e.g. 3x3 or 2x2, like the drag and drop portal stuff that’s around, e.g iGoogle.
Data set reports should make use of the view properties to default forms (the previous point) so that they look exactly like the forms
Even more flexible data entry forms with multiple orgunits or periods and possibility to pivot these around in a table
It should be possible to create forms that combine data for a facility, a household and a patient/client. Saptarshi had some ideas on how Xforms might support that.
I didn’t get to that discussion with Saptarshi in Cape Town but an important point underlying this is that XForms has an instance model to represent the form data which can be more flexible and forms-oriented than the underlying persistence model.
I hope the good work that Abyot and Saptarshi are currently doing on the community module is done with the whole system in mind - it seems natural to me that most of the data entry form functionality should be common. And ideally we could take lessons and libraries from what OpenMRS and OpenXData are doing with xforms (purcforms).
) . You will see it in the next commit.