Timestamp in DataValues

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…

···


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Morten,

I was talking about the DataValue in DataValueSet used in DataValuesSetController. We are trying to upload a series of dataValues with timestamp using the **DataValuesSetController.postJsonDataValueSet **endpoint. When we walked through the code, we saw that the timestamp field on DataValue was getting formatted to just Date (ignoring the time part) in DefaultDataValueSetService.saveDataValueSet (573).

However, when we save data values through the data entry screen in DHIS, we could see that the time part is also getting persisted.

If we see value in maintaining the same behaviour through the web api, we could submit a patch for the same.

Please do let us know your thoughts.

···

Thanks,
Aravind

On Tue, May 27, 2014 at 2:50 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hm, yes… I see that. I guess the idea is that time dimension doesn’t really matter for aggregate values.

I don’t see any harm in supporting multiple ISO formats, so if you could provide a patch, that would be welcome (but please also allow for the old format), maybe you could extend DateUtils to support multiple formats, something similar to what we have done in FacilityController (ohie-module):

private SimpleDateFormat[] simpleDateFormats = new SimpleDateFormat[]{

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ssZ” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ss” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH” ),

new SimpleDateFormat( “yyyy-MM-dd” ),

new SimpleDateFormat( “yyyy-MM” ),

new SimpleDateFormat( “yyyy” )

};

That would be enough at least as a interim solution. We are in the process of supporting multiple calendars in DHIS 2, so in the future we can’t really rely on SimpleDateFormat anymore, but would have to parse this information (possibly) using a regex (similar to what we have done in DateUnitType).

···


Morten

On Tue, May 27, 2014 at 12:49 PM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hi Morten,

I was talking about the DataValue in DataValueSet used in DataValuesSetController. We are trying to upload a series of dataValues with timestamp using the **DataValuesSetController.postJsonDataValueSet **endpoint. When we walked through the code, we saw that the timestamp field on DataValue was getting formatted to just Date (ignoring the time part) in DefaultDataValueSetService.saveDataValueSet (573).

However, when we save data values through the data entry screen in DHIS, we could see that the time part is also getting persisted.

If we see value in maintaining the same behaviour through the web api, we could submit a patch for the same.

Please do let us know your thoughts.

Thanks,
Aravind

On Tue, May 27, 2014 at 2:50 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi,
It actually does make sense I think, even for the aggregate data. We had asked for this some time back, because of the need to audit data. The date field was not precise enough for our needs, and thus, the need for a time stamp, and the change in the DB schema.

Regards,

Jason

···

On Tue, May 27, 2014 at 1:03 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hm, yes… I see that. I guess the idea is that time dimension doesn’t really matter for aggregate values.

I don’t see any harm in supporting multiple ISO formats, so if you could provide a patch, that would be welcome (but please also allow for the old format), maybe you could extend DateUtils to support multiple formats, something similar to what we have done in FacilityController (ohie-module):

private SimpleDateFormat[] simpleDateFormats = new SimpleDateFormat[]{

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ssZ” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ss” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH” ),

new SimpleDateFormat( “yyyy-MM-dd” ),

new SimpleDateFormat( “yyyy-MM” ),

new SimpleDateFormat( “yyyy” )

};

That would be enough at least as a interim solution. We are in the process of supporting multiple calendars in DHIS 2, so in the future we can’t really rely on SimpleDateFormat anymore, but would have to parse this information (possibly) using a regex (similar to what we have done in DateUnitType).


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

On Tue, May 27, 2014 at 12:49 PM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hi Morten,

I was talking about the DataValue in DataValueSet used in DataValuesSetController. We are trying to upload a series of dataValues with timestamp using the **DataValuesSetController.postJsonDataValueSet **endpoint. When we walked through the code, we saw that the timestamp field on DataValue was getting formatted to just Date (ignoring the time part) in DefaultDataValueSetService.saveDataValueSet (573).

However, when we save data values through the data entry screen in DHIS, we could see that the time part is also getting persisted.

If we see value in maintaining the same behaviour through the web api, we could submit a patch for the same.

Please do let us know your thoughts.

Thanks,
Aravind

On Tue, May 27, 2014 at 2:50 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Yes I remember that discussion.

It looks like Aravind is simply pointing out an anomaly in the behaviour of the data posted through the web api and data saved through data entry which should be made consistent. It doesn’t seem on the surface to be a matter of new behaviour or discussion - I am sure a patch to rectify this would be well received.

Unless there is something I am also missing. I guess ideally we’d like not to have two ways of saving datavaluesets and have dataentry screen act as a client of the datavaluest api.

···

On 27 May 2014 12:27, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi,
It actually does make sense I think, even for the aggregate data. We had asked for this some time back, because of the need to audit data. The date field was not precise enough for our needs, and thus, the need for a time stamp, and the change in the DB schema.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, May 27, 2014 at 1:03 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hm, yes… I see that. I guess the idea is that time dimension doesn’t really matter for aggregate values.

I don’t see any harm in supporting multiple ISO formats, so if you could provide a patch, that would be welcome (but please also allow for the old format), maybe you could extend DateUtils to support multiple formats, something similar to what we have done in FacilityController (ohie-module):

private SimpleDateFormat[] simpleDateFormats = new SimpleDateFormat[]{

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ssZ” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ss” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH” ),

new SimpleDateFormat( “yyyy-MM-dd” ),

new SimpleDateFormat( “yyyy-MM” ),

new SimpleDateFormat( “yyyy” )

};

That would be enough at least as a interim solution. We are in the process of supporting multiple calendars in DHIS 2, so in the future we can’t really rely on SimpleDateFormat anymore, but would have to parse this information (possibly) using a regex (similar to what we have done in DateUnitType).


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

On Tue, May 27, 2014 at 12:49 PM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hi Morten,

I was talking about the DataValue in DataValueSet used in DataValuesSetController. We are trying to upload a series of dataValues with timestamp using the **DataValuesSetController.postJsonDataValueSet **endpoint. When we walked through the code, we saw that the timestamp field on DataValue was getting formatted to just Date (ignoring the time part) in DefaultDataValueSetService.saveDataValueSet (573).

However, when we save data values through the data entry screen in DHIS, we could see that the time part is also getting persisted.

If we see value in maintaining the same behaviour through the web api, we could submit a patch for the same.

Please do let us know your thoughts.

Thanks,
Aravind

On Tue, May 27, 2014 at 2:50 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Yes, there is a discrepancy between the two APIs, a patch will be very welcome. We are using the API for data-entry, but we are using the API for dataValue, not dataValueSet (which takes multiple dataValues).

···


Morten

On Tue, May 27, 2014 at 1:54 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Yes I remember that discussion.

It looks like Aravind is simply pointing out an anomaly in the behaviour of the data posted through the web api and data saved through data entry which should be made consistent. It doesn’t seem on the surface to be a matter of new behaviour or discussion - I am sure a patch to rectify this would be well received.

Unless there is something I am also missing. I guess ideally we’d like not to have two ways of saving datavaluesets and have dataentry screen act as a client of the datavaluest api.

On 27 May 2014 12:27, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi,
It actually does make sense I think, even for the aggregate data. We had asked for this some time back, because of the need to audit data. The date field was not precise enough for our needs, and thus, the need for a time stamp, and the change in the DB schema.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, May 27, 2014 at 1:03 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hm, yes… I see that. I guess the idea is that time dimension doesn’t really matter for aggregate values.

I don’t see any harm in supporting multiple ISO formats, so if you could provide a patch, that would be welcome (but please also allow for the old format), maybe you could extend DateUtils to support multiple formats, something similar to what we have done in FacilityController (ohie-module):

private SimpleDateFormat[] simpleDateFormats = new SimpleDateFormat[]{

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ssZ” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm:ss” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH:mm” ),

new SimpleDateFormat( “yyyy-MM-dd’T’HH” ),

new SimpleDateFormat( “yyyy-MM-dd” ),

new SimpleDateFormat( “yyyy-MM” ),

new SimpleDateFormat( “yyyy” )

};

That would be enough at least as a interim solution. We are in the process of supporting multiple calendars in DHIS 2, so in the future we can’t really rely on SimpleDateFormat anymore, but would have to parse this information (possibly) using a regex (similar to what we have done in DateUnitType).


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

On Tue, May 27, 2014 at 12:49 PM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hi Morten,

I was talking about the DataValue in DataValueSet used in DataValuesSetController. We are trying to upload a series of dataValues with timestamp using the **DataValuesSetController.postJsonDataValueSet **endpoint. When we walked through the code, we saw that the timestamp field on DataValue was getting formatted to just Date (ignoring the time part) in DefaultDataValueSetService.saveDataValueSet (573).

However, when we save data values through the data entry screen in DHIS, we could see that the time part is also getting persisted.

If we see value in maintaining the same behaviour through the web api, we could submit a patch for the same.

Please do let us know your thoughts.

Thanks,
Aravind

On Tue, May 27, 2014 at 2:50 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Aravind,

I’m sorry, but could you give me some more details… is this event dataValues? and which timestamp field? is it the one on trackedentitydatavalue? in the even web-api, we are just doing a simple new Date() for data values

If you are talking about programstageinstance, yes, we currently only supports ISO Dates (yyyy-MM-dd), I agree that this should be improved, but it should also not break any existing clients…


Morten

On Tue, May 27, 2014 at 10:17 AM, Aravind Muralikrishnan aravindm@thoughtworks.com wrote:

Hello all,

We were looking at the API to save DataValues in DHIS. We saw that the timestamp field in DataValues only stores date value in timestamp field and ignores the time part completely before its saved.

We have a requirement in which we would want to sync the data values at regular intervals from DHIS through the web api, and we would want the time part also to be stored in DHIS.

Is there a specific reason why we are ignoring the time part when saving DataValues through the API?

Thanks,
Aravind


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp