Thanks everyone, I very much appreciate the help. I will give both of these a shot! Node.js can be damn confusing at times.
···
On Tue, Aug 25, 2015 at 1:57 AM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi Tim,
you can use promises to handle the async requests, or simply use synchronous requests. No definite answers here but I find using synchronous requests easier to work with and user interface responsiveness is not an issue for server side ad-hoc scripts like this.
The urllib-sync library will wrap urllib and provide sync behavior:
https://www.npmjs.com/package/urllib-sync
Install with npm:
npm install urllib-sync
Import:
var urlsync = require(‘urllib-sync’);
Then do synchronous requests/responses like this:
var response = urlsync.request(url, postOptions);
Lars
Timothy Harding
RPCV Vanuatu
Skype: hardingt@gmail.com
+1 (541) 632-6623
On Tue, Aug 25, 2015 at 10:35 AM, Mark Polak mark@thedutchies.com wrote:
Hey Timothy,
Is there a reason why you would not want to wait for the response? You have figured out you’re sort of flooding the server with a bunch of requests.
You can still send them in batches of say 5, wait till those are done then send the next 5, etc.
Node now supports Promises so you could transform your requests into promises and handle them in batches by using Promise.all().
The code below i just wrote up quickly without running it so you might need to see if it works and adjust it, but it was more to give you an idea.
Regards,
Mark
function doRequests(dataElementsToUpdate) {
//Map over the items
return dataElementsToUpdate.map(function (dataElementToUpdate) {
//Create a promise for each item and call resolve when that request is done.
return new Promise(function (resolve, reject) {
var postData = JSON.stringify(dataElementsToUpdate[i]);
serverConfigPOST.dhis2server.path = "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].[object.id](http://object.id);
serverConfigPOST.dhis2server.headers = {
'Content-Type': 'application/json',
'Content-Length': postData.length
};
console.log(serverConfigPOST);
var req = https.request(serverConfigPOST.dhis2server, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
res.on('end', function () {
resolve({item: dataElementsToUpdate[i].object, success: true);
});
});
req.on('error', function(e) {
//console.log('problem with request: ' + e.message);
resolve({item: dataElementsToUpdate[i].object, success: false, message: e.message);
});
});
});
}
function updateSharingInBatches() {
//Pick 5 elements to process next
var itemsToProcess = dataElementsToUpdate.splice(0, 5);
//Only continue if there are more elements
if (itemsToProcess.length) {
//Wait til the passed batch is done and excute the next after
return Promise.all(doRequests(itemsToProcess))
.then(function (responses) {
updateSharingInBatches(dataElementsToUpdate);
})
//If you want to
.catch(function (responses) {
console.log('One or more did not work');
//If you wish to continue with the batches you can just call
//updateSharingInBatches(dataElementsToUpdate);
//Or you can try run the failed one again. :)
})
}
}
//Run the whole thing
updateSharingInBatches(dataElementsToUpdate);
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org
–
On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding hardingt@gmail.com wrote:
Thanks Pierre,
Not sure how I would set up the recursive loop in javascript, I had heard about async from stack overflow as well, I was thinking of trying that, but I kind of like how quickly this works, not needing to wait for the response from one request before sending the next. The DHIS2 hardware I am working with just doesn’t seem to keep up after a certain number of requests however, hence my attempt to slow them down.
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Timothy Harding
RPCV Vanuatu
Skype: hardingt@gmail.com
+1 (541) 632-6623
On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane pierre@jembi.org wrote:
might be best to try a recursive loop using setInterval or setTimeout with a configurable value (I.e. a method calling itself) . or you could look at the async library to make things synchronous .
On Monday, August 24, 2015, Timothy Harding hardingt@gmail.com wrote:
Hello DHIS2 Devs,
I’ve got a script that works pretty well for updating a bunch of sharing settings for various objects using node.js. The only problem is, I am getting timeouts when the object update counts get too high (500+).
I am a bit new to the asynchronous nature of node.js, and I like that I can bundle a lot of updates into one without needing to worry that I need a response for one before sending the request for the next. The downside is, I have no idea where to put the sleep timer to slow down the volume of requests. Everywhere I’ve put it so far increases the timeouts, rather than decreases them. I know there are some node.js experts out there on the DHIS2 dev list so, perhaps someone has run into something similar:
var serverConfigPOST = serverConfig;
serverConfigPOST.dhis2server.method = "POST";
var i=0;
for(i=0; i < dataElementsToUpdate.length; i++){
Tried here
var postData = JSON.stringify(dataElementsToUpdate[i]);
serverConfigPOST.dhis2server.path = "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i].[object.id](http://object.id);
serverConfigPOST.dhis2server.headers = {
'Content-Type': 'application/json',
'Content-Length': postData.length
};
console.log(serverConfigPOST);
var req = https.request(serverConfigPOST.dhis2server, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
});
req.on('error', function(e) {
//console.log('problem with request: ' + e.message);
});
// write data to request body
req.write(postData);
req.end();
Tried here
// if(i==20){
// break;
// }
}
Not quite sure how to slow this one down, I may just have to run/rerun it with different starting and stopping i values in the mean time.
Timothy Harding
RPCV Vanuatu
Skype: hardingt@gmail.com
+1 (541) 632-6623
Pierre Dane
Development Manager
Jembi Health Systems
–