OrgUnit Caching with Etag.

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.

Orgunitcaching.patch (2.66 KB)

Hi Sandesh

Would a simpler solution be to just enable springs Shallow ETag filter for the web-api? I thought it was already in use, but it seems to have been removed… we are using it in the ohie-module.

···


Morten

On Fri, Oct 10, 2014 at 5:32 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Morten ,

That approach might work too. Any approach is fine with us , provided an E-Tag is being returned and its value is dependent on the payload .

 Will you be picking this up ? .
···

On Mon, Oct 20, 2014 at 10:56 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Sandesh

Would a simpler solution be to just enable springs Shallow ETag filter for the web-api? I thought it was already in use, but it seems to have been removed… we are using it in the ohie-module.


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727


Morten

On Fri, Oct 10, 2014 at 5:32 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Ok, Yes, I will add the shallow E-tag filter today, which mean that the whole web-api should have a E-tag available.

···


Morten

On Mon, Oct 20, 2014 at 12:47 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

That approach might work too. Any approach is fine with us , provided an E-Tag is being returned and its value is dependent on the payload .

 Will you be picking this up ? .

On Mon, Oct 20, 2014 at 10:56 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Sandesh

Would a simpler solution be to just enable springs Shallow ETag filter for the web-api? I thought it was already in use, but it seems to have been removed… we are using it in the ohie-module.

Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727


Morten

On Fri, Oct 10, 2014 at 5:32 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

This is now available in rev 17175, please try it out.

···


Morten

On Mon, Oct 20, 2014 at 12:56 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Ok, Yes, I will add the shallow E-tag filter today, which mean that the whole web-api should have a E-tag available.


Morten

On Mon, Oct 20, 2014 at 12:47 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

That approach might work too. Any approach is fine with us , provided an E-Tag is being returned and its value is dependent on the payload .

 Will you be picking this up ? .

On Mon, Oct 20, 2014 at 10:56 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Sandesh

Would a simpler solution be to just enable springs Shallow ETag filter for the web-api? I thought it was already in use, but it seems to have been removed… we are using it in the ohie-module.

Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727


Morten

On Fri, Oct 10, 2014 at 5:32 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

That sounds great. Thanks Morten .

···

On Mon, Oct 20, 2014 at 11:26 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Ok, Yes, I will add the shallow E-tag filter today, which mean that the whole web-api should have a E-tag available.


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727


Morten

On Mon, Oct 20, 2014 at 12:47 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

That approach might work too. Any approach is fine with us , provided an E-Tag is being returned and its value is dependent on the payload .

 Will you be picking this up ? .

On Mon, Oct 20, 2014 at 10:56 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi Sandesh

Would a simpler solution be to just enable springs Shallow ETag filter for the web-api? I thought it was already in use, but it seems to have been removed… we are using it in the ohie-module.

Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727


Morten

On Fri, Oct 10, 2014 at 5:32 PM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hello,

We were using ‘organisationUnits’ api of DHIS to get all organisation units, which was very time consuming . To solve this problem , the approach that we came up is , to add the ‘Etag’ header to the response headers returned by the api , so that , the data is downloaded only if the Organisation unit structure has changed.

We took a look into the code , and saw that there is a generic way of implementing this by making changes in the ’ AbstractCrudController ’ . We have added a method called 'getEntityListWithEtag ’ , which is called upon passing the ‘name:like:addEtag’ filter. Any implementation of 'AbstractCrudController ’ can make use of this , by just passing the filter.

We committed the changes to tw-msf branch of DHIS (Revision no: 14720) . PFA the patch for the same.

Can you please review the changes and merge it to trunk.

Regards ,

Sandesh and Mansi.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Good. I think this will work equally well.

Lars

Hi Morten ,

Apologies for the delayed response. From the change you have made , we are getting the E-Tag header. However , since the response headers do not contain the “Cache-control” header , the browser depends on the client to set the “If-none-match” header. In which case , the client has to store the previous E-Tag value and send the “If-none-match” header.

In the change that we had made , along with the “E-Tag” header , we were also sending the “Cache-control: max-age=1” header. In this case , the browser itself takes the responsibility of setting the “If-none-match” header , instead of depending on the client.

Is it possible to send the “Cache-control: max-age=1” header along with the E-Tag header ?

···

On Mon, Oct 20, 2014 at 6:37 PM, Lars Helge Øverland larshelge@gmail.com wrote:


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

Good. I think this will work equally well.

Lars

Hi Sandesh,

are you sure…? I don’t think these are related. Either you do server validation (e-tag + lastmodified headers) or you expiry cache (expiry + cache control). It should not be necessary to also set cache-control, unless it is already set to no-cache…

Lars

···

On Wed, Oct 22, 2014 at 10:50 AM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

Apologies for the delayed response. From the change you have made , we are getting the E-Tag header. However , since the response headers do not contain the “Cache-control” header , the browser depends on the client to set the “If-none-match” header. In which case , the client has to store the previous E-Tag value and send the “If-none-match” header.

In the change that we had made , along with the “E-Tag” header , we were also sending the “Cache-control: max-age=1” header. In this case , the browser itself takes the responsibility of setting the “If-none-match” header , instead of depending on the client.

Is it possible to send the “Cache-control: max-age=1” header along with the E-Tag header ?

On Mon, Oct 20, 2014 at 6:37 PM, Lars Helge Øverland larshelge@gmail.com wrote:


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

Good. I think this will work equally well.

Lars

Hi lars ,

You are right. It works as expected.

···

On Wed, Oct 22, 2014 at 2:25 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Sandesh,

are you sure…? I don’t think these are related. Either you do server validation (e-tag + lastmodified headers) or you expiry cache (expiry + cache control). It should not be necessary to also set cache-control, unless it is already set to no-cache…

Lars


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

On Wed, Oct 22, 2014 at 10:50 AM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

Apologies for the delayed response. From the change you have made , we are getting the E-Tag header. However , since the response headers do not contain the “Cache-control” header , the browser depends on the client to set the “If-none-match” header. In which case , the client has to store the previous E-Tag value and send the “If-none-match” header.

In the change that we had made , along with the “E-Tag” header , we were also sending the “Cache-control: max-age=1” header. In this case , the browser itself takes the responsibility of setting the “If-none-match” header , instead of depending on the client.

Is it possible to send the “Cache-control: max-age=1” header along with the E-Tag header ?

On Mon, Oct 20, 2014 at 6:37 PM, Lars Helge Øverland larshelge@gmail.com wrote:


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

Good. I think this will work equally well.

Lars

Okay good.

···

On Wed, Oct 22, 2014 at 11:23 AM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi lars ,

You are right. It works as expected.

On Wed, Oct 22, 2014 at 2:25 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Sandesh,

are you sure…? I don’t think these are related. Either you do server validation (e-tag + lastmodified headers) or you expiry cache (expiry + cache control). It should not be necessary to also set cache-control, unless it is already set to no-cache…

Lars


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

On Wed, Oct 22, 2014 at 10:50 AM, Sandesh Sukumar Doolipeta sandeshd@thoughtworks.com wrote:

Hi Morten ,

Apologies for the delayed response. From the change you have made , we are getting the E-Tag header. However , since the response headers do not contain the “Cache-control” header , the browser depends on the client to set the “If-none-match” header. In which case , the client has to store the previous E-Tag value and send the “If-none-match” header.

In the change that we had made , along with the “E-Tag” header , we were also sending the “Cache-control: max-age=1” header. In this case , the browser itself takes the responsibility of setting the “If-none-match” header , instead of depending on the client.

Is it possible to send the “Cache-control: max-age=1” header along with the E-Tag header ?

On Mon, Oct 20, 2014 at 6:37 PM, Lars Helge Øverland larshelge@gmail.com wrote:


Sandesh DoolipetaApplication Developer
Email
sandeshd@thoughtworks.com

Telephone
9686062727

Good. I think this will work equally well.

Lars