Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Hi Morten,
We are using metaData import frequently when transferring data back and forth in running instances. Dry Run feature is almost always used just to check any incompatibility before actual transfer. So it’s quite useful.
On Tue, Feb 2, 2016 at 1:45 PM, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Regards,
Pamod Amarakoon
Confidentiality Notice: the information contained in this email and any attachments may be legally privileged and confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachments for any purpose, nor disclose all or any part of the contents to any other person.
Since there is no rollback feature we rely on the dry run to test imports before actually importing. If you will replace with a comparable feature then great otherwise please let it live through the next season.
Thanks.
Busoye
On Tue, Feb 2, 2016 at 1:45 PM, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Regards,
Pamod Amarakoon
Confidentiality Notice: the information contained in this email and any attachments may be legally privileged and confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachments for any purpose, nor disclose all or any part of the contents to any other person.
Morten,
Dropping the dry run is OK if, and I stress if, the import process itself becomes more transparent and step-wise. Basically this means
Display new records coming in, with the ability to remove all or some from import
Display updated records coming in and the equivalent existing records, with the ability to remove all or some from import/updating.
The fundamental problem with DHIS2 imports are that they are “black boxes”, you cannot see what’s happening until after it’s done. Without even the dry run, your only “Safety net” is to back up the whole database just before importing - which is tricky to do in a production environment when manager A imports data while user B,C,D, and E are capturing data directly.
Regards
Calle
On 2 February 2016 at 10:15, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
Hi Calle
Yeah, I think its obvious from the quick emails we are getting that we should not remove it. I would be great to have a proper rollback feature, but right now this won’t be that easy to add.
(I am talking about metadata only though, you seem to also mention aggregate/event data? those importers are not scheduled to be rewritten, and will behave as today)
On Tue, Feb 2, 2016 at 3:26 PM, Calle Hedberg calle.hedberg@gmail.com wrote:
Morten,
Dropping the dry run is OK if, and I stress if, the import process itself becomes more transparent and step-wise. Basically this means
Display new records coming in, with the ability to remove all or some from import
Display updated records coming in and the equivalent existing records, with the ability to remove all or some from import/updating.
The fundamental problem with DHIS2 imports are that they are “black boxes”, you cannot see what’s happening until after it’s done. Without even the dry run, your only “Safety net” is to back up the whole database just before importing - which is tricky to do in a production environment when manager A imports data while user B,C,D, and E are capturing data directly.
Regards
Calle
–
Morten
On 2 February 2016 at 10:15, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
Morten,
I was talking about meta-data, and I’m not suggesting a roll-back feature - I’m suggesting a more step-wise import process where the user can see and control what is being imported/updated:
what meta-data appear as new (it might not actually be new meta-data, but rather just appear as new due to minor discrepancies in UIDs, names or codes)
what meta-data appear as updates (some might be wanted, some not).
I’m frequently addressing these challenges by comparing external meta-data to DHIS2 meta-data using either PostgreSQL views or ODBC connections via MS Access, but that’s really cumbersome. Still better than importing what turns out to be duplicates etc, and which later has to be cleaned up manually.
A more step-wise process will also have major advantages over the current “import summary” in the sense that you can see exactly WHAT fields are updated (and presumably also filter out updates where e.g. the LastUpdated value is older).
Regards
Calle
On 2 February 2016 at 10:29, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi Calle
Yeah, I think its obvious from the quick emails we are getting that we should not remove it. I would be great to have a proper rollback feature, but right now this won’t be that easy to add.
(I am talking about metadata only though, you seem to also mention aggregate/event data? those importers are not scheduled to be rewritten, and will behave as today)
–
–
Morten
On Tue, Feb 2, 2016 at 3:26 PM, Calle Hedberg calle.hedberg@gmail.com wrote:
Morten,
Dropping the dry run is OK if, and I stress if, the import process itself becomes more transparent and step-wise. Basically this means
Display new records coming in, with the ability to remove all or some from import
Display updated records coming in and the equivalent existing records, with the ability to remove all or some from import/updating.
The fundamental problem with DHIS2 imports are that they are “black boxes”, you cannot see what’s happening until after it’s done. Without even the dry run, your only “Safety net” is to back up the whole database just before importing - which is tricky to do in a production environment when manager A imports data while user B,C,D, and E are capturing data directly.
Regards
Calle
On 2 February 2016 at 10:15, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
From a dev perspective dryRun is pretty useful for debugging any features built on top of it.
2016-02-02 9:15 GMT+01:00 Morten Olav Hansen mortenoh@gmail.com:
Hi everyone
I’m currently in the process of re-writing our DXF2 metadata importer, and was wondering if anyone is actually using the dryRun option? its adds quite a bit of complexity to the code, so it will be added only to the new importer if people are actually using it for imports
–
Morten
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp
Calle
Here's another take on the problem.
Imagine you had a separate "metadata" instance (could variously be
called facility registry, data dictionary, indicator registry etc).
Basically a dhis2 instance with no data. And a staging instance which
contains just a copy of the metadata instance. Then you would always
apply your metadata imports to the staging instance.
Now what we don't have, but it seems like a very feasable app, is
something that can compare the contents of 2 instances. Making very
clear to the user that the staging instance has more orgunits, less
orgunits or some changed orgunits for example. And maybe some
rudimentary actions (import this, don't import that ..).
This approach might allow (i) easy user visualization of diffs, and
(ii) rollbacks, all without the need for maintaining dryRun code
within dhis2. Keeping the dryRun logic out of dhis2 core means
keeping the metadata model relatively simple. I can understand why
Morten would like to see it dropped. I suspect this could be managed
by implementing the differencing logic outside of the core logic as
well. Basically doing the dryRun into a separate instance. In fact,
for something like facility registry functionality, this might be the
only way to sensibly do it.
Just my 2c
Bob
On 2 February 2016 at 11:08, Calle Hedberg <calle.hedberg@gmail.com> wrote:
Morten,
I was talking about meta-data, and I'm not suggesting a roll-back feature -
I'm suggesting a more step-wise import process where the user can see and
control what is being imported/updated:
- what meta-data appear as new (it might not actually be new meta-data, but
rather just appear as new due to minor discrepancies in UIDs, names or
codes)
- what meta-data appear as updates (some might be wanted, some not).I'm frequently addressing these challenges by comparing external meta-data
to DHIS2 meta-data using either PostgreSQL views or ODBC connections via MS
Access, but that's really cumbersome. Still better than importing what turns
out to be duplicates etc, and which later has to be cleaned up manually.A more step-wise process will also have major advantages over the current
"import summary" in the sense that you can see exactly WHAT fields are
updated (and presumably also filter out updates where e.g. the LastUpdated
value is older).Regards
CalleOn 2 February 2016 at 10:29, Morten Olav Hansen <mortenoh@gmail.com> wrote:
Hi Calle
Yeah, I think its obvious from the quick emails we are getting that we
should not remove it. I would be great to have a proper rollback feature,
but right now this won't be that easy to add.(I am talking about metadata only though, you seem to also mention
aggregate/event data? those importers are not scheduled to be rewritten, and
will behave as today)--
MortenOn Tue, Feb 2, 2016 at 3:26 PM, Calle Hedberg <calle.hedberg@gmail.com> >> wrote:
Morten,
Dropping the dry run is OK if, and I stress if, the import process itself
becomes more transparent and step-wise. Basically this means
1.
Display new records coming in, with the ability to remove all or some
from import
2.
Display updated records coming in and the equivalent existing records,
with the ability to remove all or some from import/updating.The fundamental problem with DHIS2 imports are that they are "black
boxes", you cannot see what's happening until after it's done. Without even
the dry run, your only "Safety net" is to back up the whole database just
before importing - which is tricky to do in a production environment when
manager A imports data while user B,C,D, and E are capturing data directly.Regards
CalleOn 2 February 2016 at 10:15, Morten Olav Hansen <mortenoh@gmail.com> >>> wrote:
Hi everyone
I'm currently in the process of re-writing our DXF2 metadata importer,
and was wondering if anyone is actually using the dryRun option? its adds
quite a bit of complexity to the code, so it will be added only to the new
importer if people are actually using it for imports--
Morten_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help--
*******************************************
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
*******************************************
--
*******************************************
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
*******************************************
_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help