We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
I would like to see Etags supported with our web API, if removing camel
The point is that then we can listen to the changes using ETags and check modification for data when new ETags are generated for the same request.
Even shallow ETags would be a good first pass, if camel is removed.
Regards,
Saptarshi PURKAYASTHA
Date: Tue, 1 Oct 2013 11:26:55 +0100
From: bobjolliffe@gmail.com
To: dhis2-devs@lists.launchpad.net
Subject: [Dhis2-devs] Removing camel integration module
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help
Agree etags would be good for tracking changes - in some ways regardless of integrated camel, but monitoring changes in resources from the outside is even more challenging.
On 1 October 2013 14:48, Saptarshi Purkayastha sunbiz@gmail.com wrote:
I would like to see Etags supported with our web API, if removing camel
The point is that then we can listen to the changes using ETags and check modification for data when new ETags are generated for the same request.Even shallow ETags would be a good first pass, if camel is removed.
Regards,
Saptarshi PURKAYASTHA
Date: Tue, 1 Oct 2013 11:26:55 +0100
From: bobjolliffe@gmail.comTo: dhis2-devs@lists.launchpad.net
Subject: [Dhis2-devs] Removing camel integration moduleWe have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi Rodolfo, we are not removing the functionality, rather just making it an independent module which can be set up alongside DHIS. Bob will know the details.
Lars
Nothing against Camel, but not sure what the issue with executing a task on the database might be. Just wrap it up as a shell script and schedule it with cron.
Maybe I am missing some security issue there, but we use this approach extensively for this type of task.
Regards,
jason
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Well by taking it out of the jvm we do remove some functionality … mostly access to all the dhis2 beans and whatever is not immediately available through the api.
But it is quite straightforward to run a standalone camel instance which talks to dhis2 via the api.
Unfortunately there are always some times when you have to resort to the database 
If you only want to schedule some sql script to run then I agree with Jason that a cron job which pipes the script to psql (not pgadmin) can often suffice.
If you find yourself (in addition to just scheduling), also doing more complex data transformations and the like then it can make sense to have an integration engine like camel - which also has a jdbc endpoint. But again, if the only thing your camel context is doing is running the sql then it really doesn’t need to be in the dhis.war. Thats really why we have removed it. You can do the same thing with camel (and/or other tools) without having it loaded in the virtual machine.
On 1 October 2013 15:04, Jason Pickering jason.p.pickering@gmail.com wrote:
Nothing against Camel, but not sure what the issue with executing a task on the database might be. Just wrap it up as a shell script and schedule it with cron.
Maybe I am missing some security issue there, but we use this approach extensively for this type of task.
Regards,
jason
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Note that the danger of using sql directly is not so much securing a cron executed script (that can be done reasonably well), but more that it is quite easy to mess up the metadata integrity.
On 1 October 2013 15:19, Bob Jolliffe bobjolliffe@gmail.com wrote:
Well by taking it out of the jvm we do remove some functionality … mostly access to all the dhis2 beans and whatever is not immediately available through the api.
But it is quite straightforward to run a standalone camel instance which talks to dhis2 via the api.
Unfortunately there are always some times when you have to resort to the database
If you find yourself (in addition to just scheduling), also doing more complex data transformations and the like then it can make sense to have an integration engine like camel - which also has a jdbc endpoint. But again, if the only thing your camel context is doing is running the sql then it really doesn’t need to be in the dhis.war. Thats really why we have removed it. You can do the same thing with camel (and/or other tools) without having it loaded in the virtual machine.
On 1 October 2013 15:04, Jason Pickering jason.p.pickering@gmail.com wrote:
Nothing against Camel, but not sure what the issue with executing a task on the database might be. Just wrap it up as a shell script and schedule it with cron.
Maybe I am missing some security issue there, but we use this approach extensively for this type of task.
Regards,
jason
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
For me the issue here is choosing the right tool. I do not know much about Camel, but it seems like a good tool. There are others which perform similar things like Pentaho and Kettle. Personally (as you have probably noted from my previous posts on this list) I tend to personally prefer a combination of R and SQL. We run some small tasks, like trimming data element and orgunit names on a regular basis. In my mind, there is no need for a heavy hammer for this. It is a single SQL statement, and can be easily scheduled on a regular basis to run with cron (or pgagent). Other tasks are more suited to things that R is good at, such as more complex data manipulation Since R can work easily with both SQL as well as the web API (and can be automated to execute from the shell) I have never seen a need for Camel, but again, I think it is a matter of which tools you prefer to get the job done and are most familiar with.
Personally, I think it makes sense to remove this from the core build. It makes sense to continue to document how it Camel can be used with DHIS2, but in the long run, I think the loose coupling of the two probably makes a lot of sense.
Regards,
Jason
On Tue, Oct 1, 2013 at 4:24 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:
Note that the danger of using sql directly is not so much securing a cron executed script (that can be done reasonably well), but more that it is quite easy to mess up the metadata integrity.
On 1 October 2013 15:19, Bob Jolliffe bobjolliffe@gmail.com wrote:
Well by taking it out of the jvm we do remove some functionality … mostly access to all the dhis2 beans and whatever is not immediately available through the api.
But it is quite straightforward to run a standalone camel instance which talks to dhis2 via the api.
Unfortunately there are always some times when you have to resort to the database
If you find yourself (in addition to just scheduling), also doing more complex data transformations and the like then it can make sense to have an integration engine like camel - which also has a jdbc endpoint. But again, if the only thing your camel context is doing is running the sql then it really doesn’t need to be in the dhis.war. Thats really why we have removed it. You can do the same thing with camel (and/or other tools) without having it loaded in the virtual machine.
On 1 October 2013 15:04, Jason Pickering jason.p.pickering@gmail.com wrote:
Nothing against Camel, but not sure what the issue with executing a task on the database might be. Just wrap it up as a shell script and schedule it with cron.
Maybe I am missing some security issue there, but we use this approach extensively for this type of task.
Regards,
jason
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
I actually meant pgAgent. Sorry for the confusion. It looks like we will be looking at cron jobs, while dhis apps gain some sort of scheduling capabilities.
Rodolfo Meliá
+44 777 576 4090 | +1 708 872-7636
Please ignore any typos on this email sent from my mobile, probably written while I was at the park with the kids, at the airport walking to the gate, or sleepless (although exhausted). Grammatical faults? I’m sorry- I do my best
On 1 October 2013 15:04, Jason Pickering jason.p.pickering@gmail.com wrote:
Nothing against Camel, but not sure what the issue with executing a task on the database might be. Just wrap it up as a shell script and schedule it with cron.
Maybe I am missing some security issue there, but we use this approach extensively for this type of task.
Regards,
jason
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi Jason
To complement what Rodolfo mentioned, the issue we had with pgAgent is the requirement to set the password in ~/.pgpass We felt this was insecure because it allows access to the database just by using the correct username.
Best regards,
JM
On Tue, Oct 1, 2013 at 3:59 PM, Rodolfo Melia rmelia@knowming.com wrote:
Hi - we were thinking about using Camel for automating certain jobs that we want to run every 24 hours (update some Data Elements based on some custom calculations that cannot be run as indicators). We ruled out pgAdmin because we feel is too insecure. Now that camel is out, how can we schedule task to run every day? Do you plan to have a scheduler for an app, so certain routines can run on the basis of a schedule?
Rodolfo
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Rodolfo Meliá
Principal
+1 708 872 7636
+44 777 576 4090
Skype: rod.melia
On Tue, Oct 1, 2013 at 11:26 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:
We have decided to remove the camel integration module from the dhis2 war file in the next release.
As the web api has matured it is now more feasible to run the likes of apache camel or mule or similar software as external clients of the api.
Regards
Bob
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi Juan
If your script is running on the same machine as the postgres server then
my preference is to connect using unix domain socket (that would be local
in pg_hba.conf) and peer authentication so no password.
Typically I would have a system user (say dhis) which runs the tomcat
container for an instance (using the 'standardised' install from
dhis2-tools). That same user would have access to the database. So in the
crontab for that user you can include various sql scripts which you execute
at prescribed times using psql.
Bob