DHIS Server - Setting up a scheduled event data check and update

Support - Assistance technique
1

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)

  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

2

hi james, you can call Postgres functions from sqlviews (from data administration) , and the functions can do updates as well as return data. the views can also be parameterized and called through the API .

···

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

3

so I suggest using cron with curl to call the sqlview

···

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

4

Hi James,

There are many different ways to do this, and it may or may not be necessary to access the database at all.

Really any scripting language can be used to automate tasks. Some of the ones which I know are being used are Bash, NodeJS, Python and R. Likely there are others out there. There is no requirement to have a Java executable by any means, but it of course could be done this way, if you were so inclined. Anything which can interact with the WebAPI using HTTP should be fine.

Here is a good example in NodeJS-https://github.com/dhis2/dhis2-utils/blob/master/resources/nodejs/integrity/integrity.js of a task which checks some special Integrity SQL views, and then sends out a notification if any of them fail.

The basic pattern though is to access some enpoint through the API, process what you need in your script, and possibly post something back to the server. The other way, as Pierre points out, would be to call an SQL view with a cron job through the API, but It would seem not to be really what you want to do, if you do not want to use PgAgent, which could do the same thing.

Regards,

Jason

···

On Sun, Nov 29, 2015 at 8:59 AM, Pierre Dane pierre@jembi.org wrote:

so I suggest using cron with curl to call the sqlview

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

5

Thanks, Pierre.

Interesting that we can call db functions from sqlviews. It’s definitely an option I can try if I go with sql statement.

As management point, it would be easier to manage than putting the sql statement in shell command ( psql — ).

···

On Sat, Nov 28, 2015 at 11:59 PM, Pierre Dane pierre@jembi.org wrote:

so I suggest using cron with curl to call the sqlview

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

6

Thanks, Jason.

Looked at the ‘integrity.js’. Quite good to find that I can use NodeJS to make it run via cron job.

Having it in java takes care of handling the JSON data quite easily.

Will have to install related libraries, but that’s a great option. Much better than doing it via Java.

Thanks for the file.

···

On Sun, Nov 29, 2015 at 4:46 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi James,

There are many different ways to do this, and it may or may not be necessary to access the database at all.

Really any scripting language can be used to automate tasks. Some of the ones which I know are being used are Bash, NodeJS, Python and R. Likely there are others out there. There is no requirement to have a Java executable by any means, but it of course could be done this way, if you were so inclined. Anything which can interact with the WebAPI using HTTP should be fine.

Here is a good example in NodeJS-https://github.com/dhis2/dhis2-utils/blob/master/resources/nodejs/integrity/integrity.js of a task which checks some special Integrity SQL views, and then sends out a notification if any of them fail.

The basic pattern though is to access some enpoint through the API, process what you need in your script, and possibly post something back to the server. The other way, as Pierre points out, would be to call an SQL view with a cron job through the API, but It would seem not to be really what you want to do, if you do not want to use PgAgent, which could do the same thing.

Regards,

Jason

On Sun, Nov 29, 2015 at 8:59 AM, Pierre Dane pierre@jembi.org wrote:

so I suggest using cron with curl to call the sqlview

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

7

Hi James,

Another example you might want to have a look at is here

http://rpubs.com/jason_p_pickering/124722

This shows how an external script can interact with both the API as well as the database. In certain cases, it may be much more efficient to interact/query the database directly as opposed to going through the API, for instance, with data intensive tasks better performed at the database level, or for things which are perhaps not so convenient through the API (unwinding category option combos is one example). This script (in R) partially uses the WebAPI, as well as the database, to perform “validation rule analytics”, outside of the system.

I guess similar libraries to access the database directly with NodeJS are available (https://www.npmjs.com/package/pg) to query the database directly with Node.

So my point is, interacting directly with the API has advantages, as the script can be run from anywhere, but accessing the database directly, may be more efficient, or in certain cases, the only way, but you are then obviously limited to running the script on a machine that actually has access to the DHIS2 database directly (which of course, does not need to be the same machine as is running the application).

Regards,

Jason

···

On Mon, Nov 30, 2015 at 7:00 AM, James Chang jamesbchang@gmail.com wrote:

Thanks, Jason.

Looked at the ‘integrity.js’. Quite good to find that I can use NodeJS to make it run via cron job.

Having it in java takes care of handling the JSON data quite easily.

Will have to install related libraries, but that’s a great option. Much better than doing it via Java.

Thanks for the file.

–

On Sun, Nov 29, 2015 at 4:46 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi James,

There are many different ways to do this, and it may or may not be necessary to access the database at all.

Really any scripting language can be used to automate tasks. Some of the ones which I know are being used are Bash, NodeJS, Python and R. Likely there are others out there. There is no requirement to have a Java executable by any means, but it of course could be done this way, if you were so inclined. Anything which can interact with the WebAPI using HTTP should be fine.

Here is a good example in NodeJS-https://github.com/dhis2/dhis2-utils/blob/master/resources/nodejs/integrity/integrity.js of a task which checks some special Integrity SQL views, and then sends out a notification if any of them fail.

The basic pattern though is to access some enpoint through the API, process what you need in your script, and possibly post something back to the server. The other way, as Pierre points out, would be to call an SQL view with a cron job through the API, but It would seem not to be really what you want to do, if you do not want to use PgAgent, which could do the same thing.

Regards,

Jason

On Sun, Nov 29, 2015 at 8:59 AM, Pierre Dane pierre@jembi.org wrote:

so I suggest using cron with curl to call the sqlview

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

8

Good to know. Thanks, Jason.

···

On Mon, Nov 30, 2015 at 12:26 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi James,

Another example you might want to have a look at is here

http://rpubs.com/jason_p_pickering/124722

This shows how an external script can interact with both the API as well as the database. In certain cases, it may be much more efficient to interact/query the database directly as opposed to going through the API, for instance, with data intensive tasks better performed at the database level, or for things which are perhaps not so convenient through the API (unwinding category option combos is one example). This script (in R) partially uses the WebAPI, as well as the database, to perform “validation rule analytics”, outside of the system.

I guess similar libraries to access the database directly with NodeJS are available (https://www.npmjs.com/package/pg) to query the database directly with Node.

So my point is, interacting directly with the API has advantages, as the script can be run from anywhere, but accessing the database directly, may be more efficient, or in certain cases, the only way, but you are then obviously limited to running the script on a machine that actually has access to the DHIS2 database directly (which of course, does not need to be the same machine as is running the application).

Regards,

Jason

On Mon, Nov 30, 2015 at 7:00 AM, James Chang jamesbchang@gmail.com wrote:

Thanks, Jason.

Looked at the ‘integrity.js’. Quite good to find that I can use NodeJS to make it run via cron job.

Having it in java takes care of handling the JSON data quite easily.

Will have to install related libraries, but that’s a great option. Much better than doing it via Java.

Thanks for the file.

–
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On Sun, Nov 29, 2015 at 4:46 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi James,

There are many different ways to do this, and it may or may not be necessary to access the database at all.

Really any scripting language can be used to automate tasks. Some of the ones which I know are being used are Bash, NodeJS, Python and R. Likely there are others out there. There is no requirement to have a Java executable by any means, but it of course could be done this way, if you were so inclined. Anything which can interact with the WebAPI using HTTP should be fine.

Here is a good example in NodeJS-https://github.com/dhis2/dhis2-utils/blob/master/resources/nodejs/integrity/integrity.js of a task which checks some special Integrity SQL views, and then sends out a notification if any of them fail.

The basic pattern though is to access some enpoint through the API, process what you need in your script, and possibly post something back to the server. The other way, as Pierre points out, would be to call an SQL view with a cron job through the API, but It would seem not to be really what you want to do, if you do not want to use PgAgent, which could do the same thing.

Regards,

Jason

On Sun, Nov 29, 2015 at 8:59 AM, Pierre Dane pierre@jembi.org wrote:

so I suggest using cron with curl to call the sqlview

On Sunday, 29 November 2015, James Chang jamesbchang@gmail.com wrote:

Hi,

In DHIS Server, we are trying to create a scheduled task that checks events and update some data in events.

We like to avoid using database scheduler (like pgadmin III PgAgent)

That leaves us (We think) with:

  • cron job with shell script using database sql comments (in file)
  • cron job with java executable (and Web API calls within Java)

We prefer to do this using Web API.

Does that mean ‘cron job with java executable’ is the only choice we could have for our operation?

Any suggestion on other methods that we could use to set up a ‘Web API data checking/updating’ scheduled task? (If there is, could you also provide us with example?)

Thank you.

–
Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

9

2 posts were split to a new topic: DHIS Server - Setting up a scheduled event data check and update