From an architectural view of DHIS2 as a platform, it is quite interesting to see suggestions coming up about third party solutions such as QGIS when there is already an attempt well underway to build on our own app framework to enable this functionality within DHIS2. Especially when this goes to the heart of configuring the platform, namely bootstrapping the OU hierarchy, which forms the backbone of an implementation.
There are several pieces to the puzzle: Reprojection, Simplification/Generalisation, Linking to existing OU hierarchy. To me it seems natural to want to pull this away from a series of command line steps and external tools, but it is interesting to hear arguments in other directions, also for Sushil’s thesis. Feedback from the community is important. Of course, we used to have such an external module with the DataMart, but that was on the output side (not configuration)
Knut
···
On Tue, Mar 31, 2015 at 12:34 PM, Knut Staring knutst@gmail.com wrote:
My point is that mapshaper already reads the shapefile, and Sushil’s app built around it already exists.
You are right that some shapefiles are really heavy (thus the need for mapshaper).
To me it makes little sense to be quite honest. QGIS and other tools can do all of this, remarkably well. Yes, it is maybe not the smoothest workflow, but on the other hand, does it really make sense to invest time and money into developing a custom tool for transformation, manipulation and export of geographical data?
What about another example… a tool to import Excel sheets, instead of CSV. Metadata import via CSV is supported, but why should we not direct import of Excel files? I am sure an app could be developed for this, but since Excel can do this so trivially, it would seem to really make little sense to do so.
This is my feeling as well with the geographical data. There are so many other things which we do not cover, such as the linkage between an existing hierarchy and a shape file, resolution of broken topology, cleansing of donuts holes which often result from generalization, etc. Will these be covered as well? Also, in my experience, we do always derive the hierarchy itself from the shape file (although in certain cases this may happen). How would the proposed tool address all of these use cases, and would it not be better to create detailed guidance on how to do these sorts of tasks with more general tools instead? After all, there are only so many countries.
I do not want to put a damper on the development of this app, as I think it could be useful to some, but feel it will be a lot of trouble to keep it up and maintained as the platform itself changes.
My two cents.
Best regards,
Jason
···
On Tue, Mar 31, 2015 at 1:02 PM, Knut Staring knutst@gmail.com wrote:
From an architectural view of DHIS2 as a platform, it is quite interesting to see suggestions coming up about third party solutions such as QGIS when there is already an attempt well underway to build on our own app framework to enable this functionality within DHIS2. Especially when this goes to the heart of configuring the platform, namely bootstrapping the OU hierarchy, which forms the backbone of an implementation.
There are several pieces to the puzzle: Reprojection, Simplification/Generalisation, Linking to existing OU hierarchy. To me it seems natural to want to pull this away from a series of command line steps and external tools, but it is interesting to hear arguments in other directions, also for Sushil’s thesis. Feedback from the community is important. Of course, we used to have such an external module with the DataMart, but that was on the output side (not configuration)
Knut, the nub of your argument seems to be to to implement shapefile
import as an augmentation to dhis - something which I also saw Calle
and others suggest in earlier thread.
Its not necessarily a bad idea, but my point is that I don't think
this would be easy to do as an "app". I might of course be wrong about
this. But I can't see the processing of shapefiles (which are
misleadingly called files) being easily done in javascript in the
browser. So I think you are going to have some additional java code
in the core - not necessarily a lot, but some.
Someone like Jan might know better.
Regarding a high level architecture view, I suppose the tension here
is between simple tools which do what they do well but are not always
easy to put together when you don't know what you are doing (the unix
philosophy, and ogr approach) vs limitless growth to the web
application and pumping-up-the-jam There are forces which drive
both rationales.
···
On 31 March 2015 at 12:02, Knut Staring <knutst@gmail.com> wrote:
From an architectural view of DHIS2 as a platform, it is quite interesting
to see suggestions coming up about third party solutions such as QGIS when
there is already an attempt well underway to build on our own app framework
to enable this functionality within DHIS2. Especially when this goes to the
heart of configuring the platform, namely bootstrapping the OU hierarchy,
which forms the backbone of an implementation.
There are several pieces to the puzzle: Reprojection,
Simplification/Generalisation, Linking to existing OU hierarchy. To me it
seems natural to want to pull this away from a series of command line steps
and external tools, but it is interesting to hear arguments in other
directions, also for Sushil's thesis. Feedback from the community is
important. Of course, we used to have such an external module with the
DataMart, but that was on the output side (not configuration)
Knut
On Tue, Mar 31, 2015 at 12:34 PM, Knut Staring <knutst@gmail.com> wrote:
My point is that mapshaper already reads the shapefile, and Sushil's app
built around it already exists.
You are right that some shapefiles are really heavy (thus the need for
mapshaper).
On Tue, Mar 31, 2015 at 11:50 AM, Bob Jolliffe <bobjolliffe@gmail.com> >> wrote:
That is a handy enough library for dealing with projection all right.
But doesn't really help in reading the shapefile, which *might* be a
bit of a heavy lift for an app dealing with dbf and shp files. I'm
not convinced this would be a really good use of developer time.
QGis already does all the grunt work of reading in the shapefile and
dealing with the various oddnesses. Getting it to dump its output as
dhis orgunits seems like a relatively straightforward proposition.
But you can't assume everyone would want to use qgis.
Maybe another approach would be to create a simple less geeky ui on
top of ogr2ogr. dunno really. I'm not seeing a silver bullet here.
On 31 March 2015 at 10:12, Knut Staring <knutst@gmail.com> wrote:
> Much better use of developer time would be to include
> http://proj4js.org/ in
> Sushil's app, so that everything needed comes in one sweet package.
>
>
> On Tue, Mar 31, 2015 at 11:10 AM, Knut Staring <knutst@gmail.com> >>> > wrote:
>>
>> shp2gml is taken care of by ogr2ogr, as described in our manual.
>>
>> Yes, someone could write a QGis plugin, but why should they, since we
>> are
>> now working on a web app to do this. Granted, it does not (yet)
>> include
>> reprojection, but it does include the other crucial step of
>> generalisation
>> using http://www.mapshaper.org/\.
>>
>> On Tue, Mar 31, 2015 at 9:53 AM, Bob Jolliffe <bobjolliffe@gmail.com> >>> >> wrote:
>>>
>>> Of course shp2gml tools already exist - they could be described as a
>>> bit geeky using the command line. Though there is need for some of
>>> those geeky options for dealing with projection systems and the like.
>>>
>>> I think someone with some python know how could write a neat little
>>> QGis plugin to export to dhis2 format which *might* make it easier
>>> for
>>> users.
>>>
>>> On 30 March 2015 at 18:59, Lars Helge Øverland <larshelge@gmail.com> >>> >>> wrote:
>>> > Shape file import sounds like an ideal task for an external
>>> > developer,
>>> > as it
>>> > could transform the shapefile content and pipe it into the GML
>>> > importer
>>> > and
>>> > hence does not require lots of DHIS 2 knowledge.
>>> >
>>> > Lars
>>> >
>>> >
>>> > _______________________________________________
>>> > Mailing list: DHIS 2 developers in Launchpad
>>> > Post to : dhis2-devs@lists.launchpad.net
>>> > Unsubscribe : DHIS 2 developers in Launchpad
>>> > More help : ListHelp - Launchpad Help
>>> >
>>>
>>> _______________________________________________
>>> Mailing list: DHIS 2 developers in Launchpad
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : DHIS 2 developers in Launchpad
>>> More help : ListHelp - Launchpad Help
>>
>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> Norway: +4791880522
>> Skype: knutstar
>> http://dhis2.org
>
>
>
>
> --
> Knut Staring
> Dept. of Informatics, University of Oslo
> Norway: +4791880522
> Skype: knutstar
> http://dhis2.org
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar http://dhis2.org
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar http://dhis2.org
Knut, the nub of your argument seems to be to to implement shapefile
import as an augmentation to dhis - something which I also saw Calle
and others suggest in earlier thread.
Its not necessarily a bad idea,
It is not just an idea, it has already been done, cf. Sushil's post.
but my point is that I don't think
this would be easy to do as an "app". I might of course be wrong about
this. But I can't see the processing of shapefiles (which are
misleadingly called files) being easily done in javascript in the
browser.
So I think you are going to have some additional java code
in the core - not necessarily a lot, but some.
The only challenge will be size. I think the limit of the old, Flash
implementation was 80 MB. No such limit is displayed for the new one: http://www.mapshaper.org/offline.html
Someone like Jan might know better.
Regarding a high level architecture view, I suppose the tension here
is between simple tools which do what they do well but are not always
easy to put together when you don't know what you are doing (the unix
philosophy, and ogr approach) vs limitless growth to the web
application and pumping-up-the-jam There are forces which drive
both rationales.
Certainly. But in this case, we already have the Unix approach available,
and it's well documented, but still quite daunting for some users. What
Sushil is building is the "point-and-click" version, but better than that,
because we also leverage the DHIS Web API to link with existing OU (e.g
with alternative spellings)
Knut
···
On Tue, Mar 31, 2015 at 1:23 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
On 31 March 2015 at 12:02, Knut Staring <knutst@gmail.com> wrote:
> From an architectural view of DHIS2 as a platform, it is quite
interesting
> to see suggestions coming up about third party solutions such as QGIS
when
> there is already an attempt well underway to build on our own app
framework
> to enable this functionality within DHIS2. Especially when this goes to
the
> heart of configuring the platform, namely bootstrapping the OU hierarchy,
> which forms the backbone of an implementation.
>
> There are several pieces to the puzzle: Reprojection,
> Simplification/Generalisation, Linking to existing OU hierarchy. To me it
> seems natural to want to pull this away from a series of command line
steps
> and external tools, but it is interesting to hear arguments in other
> directions, also for Sushil's thesis. Feedback from the community is
> important. Of course, we used to have such an external module with the
> DataMart, but that was on the output side (not configuration)
>
> Knut
>
> On Tue, Mar 31, 2015 at 12:34 PM, Knut Staring <knutst@gmail.com> wrote:
>>
>> My point is that mapshaper already reads the shapefile, and Sushil's app
>> built around it already exists.
>>
>> You are right that some shapefiles are really heavy (thus the need for
>> mapshaper).
>>
>> On Tue, Mar 31, 2015 at 11:50 AM, Bob Jolliffe <bobjolliffe@gmail.com> > >> wrote:
>>>
>>> That is a handy enough library for dealing with projection all right.
>>> But doesn't really help in reading the shapefile, which *might* be a
>>> bit of a heavy lift for an app dealing with dbf and shp files. I'm
>>> not convinced this would be a really good use of developer time.
>>>
>>> QGis already does all the grunt work of reading in the shapefile and
>>> dealing with the various oddnesses. Getting it to dump its output as
>>> dhis orgunits seems like a relatively straightforward proposition.
>>> But you can't assume everyone would want to use qgis.
>>>
>>> Maybe another approach would be to create a simple less geeky ui on
>>> top of ogr2ogr. dunno really. I'm not seeing a silver bullet here.
>>>
>>> On 31 March 2015 at 10:12, Knut Staring <knutst@gmail.com> wrote:
>>> > Much better use of developer time would be to include
>>> > http://proj4js.org/ in
>>> > Sushil's app, so that everything needed comes in one sweet package.
>>> >
>>> >
>>> > On Tue, Mar 31, 2015 at 11:10 AM, Knut Staring <knutst@gmail.com> > >>> > wrote:
>>> >>
>>> >> shp2gml is taken care of by ogr2ogr, as described in our manual.
>>> >>
>>> >> Yes, someone could write a QGis plugin, but why should they, since
we
>>> >> are
>>> >> now working on a web app to do this. Granted, it does not (yet)
>>> >> include
>>> >> reprojection, but it does include the other crucial step of
>>> >> generalisation
>>> >> using http://www.mapshaper.org/\.
>>> >>
>>> >> On Tue, Mar 31, 2015 at 9:53 AM, Bob Jolliffe < > bobjolliffe@gmail.com> > >>> >> wrote:
>>> >>>
>>> >>> Of course shp2gml tools already exist - they could be described as
a
>>> >>> bit geeky using the command line. Though there is need for some of
>>> >>> those geeky options for dealing with projection systems and the
like.
>>> >>>
>>> >>> I think someone with some python know how could write a neat little
>>> >>> QGis plugin to export to dhis2 format which *might* make it easier
>>> >>> for
>>> >>> users.
>>> >>>
>>> >>> On 30 March 2015 at 18:59, Lars Helge Øverland < > larshelge@gmail.com> > >>> >>> wrote:
>>> >>> > Shape file import sounds like an ideal task for an external
>>> >>> > developer,
>>> >>> > as it
>>> >>> > could transform the shapefile content and pipe it into the GML
>>> >>> > importer
>>> >>> > and
>>> >>> > hence does not require lots of DHIS 2 knowledge.
>>> >>> >
>>> >>> > Lars
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > Mailing list: DHIS 2 developers in Launchpad
>>> >>> > Post to : dhis2-devs@lists.launchpad.net
>>> >>> > Unsubscribe : DHIS 2 developers in Launchpad
>>> >>> > More help : ListHelp - Launchpad Help
>>> >>> >
>>> >>>
>>> >>> _______________________________________________
>>> >>> Mailing list: DHIS 2 developers in Launchpad
>>> >>> Post to : dhis2-devs@lists.launchpad.net
>>> >>> Unsubscribe : DHIS 2 developers in Launchpad
>>> >>> More help : ListHelp - Launchpad Help
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Knut Staring
>>> >> Dept. of Informatics, University of Oslo
>>> >> Norway: +4791880522
>>> >> Skype: knutstar
>>> >> http://dhis2.org
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Knut Staring
>>> > Dept. of Informatics, University of Oslo
>>> > Norway: +4791880522
>>> > Skype: knutstar
>>> > http://dhis2.org
>>
>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> Norway: +4791880522
>> Skype: knutstar
>> http://dhis2.org
>
>
>
>
> --
> Knut Staring
> Dept. of Informatics, University of Oslo
> Norway: +4791880522
> Skype: knutstar
> http://dhis2.org
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar
To me it makes little sense to be quite honest. QGIS and other tools can
do all of this, remarkably well.
Unfortunately, I have found this to NOT be the case when it comes to the
simplification/generalisation that Mapshaper does so nicely. I have yet to
find similar smooth functionality with either QGIS, OpenJump or PostGIS.
Probably ArcGIS has it, but most people don't have access to that. And
since this step is crucial, the idea was to leverage Mapshaper in its
Javascript reincarnation.
Yes, it is maybe not the smoothest workflow, but on the other hand, does
it really make sense to invest time and money into developing a custom tool
for transformation, manipulation and export of geographical data?
Perhaps not, but this is an area that is pretty alien to most people, even
techies. But you are right that what usually happens is that someone like
you, Jan or I do it for them.
What about another example... a tool to import Excel sheets, instead of
CSV. Metadata import via CSV is supported, but why should we not direct
import of Excel files? I am sure an app could be developed for this, but
since Excel can do this so trivially, it would seem to really make little
sense to do so.
Saving a sheet as CSV is trivial. As you probably knows better than anyone,
there are a lot of non-trivial tasks surrounding import of metadata and
legacy data. My hunch is that though these processes are quite resistant to
full autmation without some very serious AI built in, projects like Tableau
tell me that quite a lot can be done to ease the overall bootstrapping
process. But I do agree that guidance on best practices is probably going
to be even more useful.
This is my feeling as well with the geographical data. There are so many
other things which we do not cover, such as the linkage between an existing
hierarchy and a shape file, resolution of broken topology, cleansing of
donuts holes which often result from generalization, etc. Will these be
covered as well?
Yes, the app does actually aim to cover the linkage, and Mapshaper does a
much better job at the generalization (I've never felt a need to do any
post-cleaning).
Also, in my experience, we do always derive the hierarchy itself from the
shape file (although in certain cases this may happen). How would the
proposed tool address all of these use cases, and would it not be better to
create detailed guidance on how to do these sorts of tasks with more
general tools instead? After all, there are only so many countries.
That is a good point - there are no more than 200+ countries, so we could
actually prepare a set of DHIS2 OU import files for all countries a la http://gadm.org/, maybe in collaboration with them. The same argument of
course goes for any other tool we would develop.
I do not want to put a damper on the development of this app, as I think
it could be useful to some, but feel it will be a lot of trouble to keep it
up and maintained as the platform itself changes.
I would expect the hierarchy to be fairly stable, but of course it could be
built out to handle historical changes, and also extend the GIS module to
be more flexible, in which case the API might undergo changes also.
My two cents.
Thanks for your input!
Knut
···
On Tue, Mar 31, 2015 at 1:16 PM, Jason Pickering < jason.p.pickering@gmail.com> wrote:
Best regards,
Jason
On Tue, Mar 31, 2015 at 1:02 PM, Knut Staring <knutst@gmail.com> wrote:
From an architectural view of DHIS2 as a platform, it is quite
interesting to see suggestions coming up about third party solutions such
as QGIS when there is already an attempt well underway to build on our own
app framework to enable this functionality within DHIS2. Especially when
this goes to the heart of configuring the platform, namely bootstrapping
the OU hierarchy, which forms the backbone of an implementation.
There are several pieces to the puzzle: Reprojection,
Simplification/Generalisation, Linking to existing OU hierarchy. To me it
seems natural to want to pull this away from a series of command line steps
and external tools, but it is interesting to hear arguments in other
directions, also for Sushil's thesis. Feedback from the community is
important. Of course, we used to have such an external module with the
DataMart, but that was on the output side (not configuration)
Knut
On Tue, Mar 31, 2015 at 12:34 PM, Knut Staring <knutst@gmail.com> wrote:
My point is that mapshaper already reads the shapefile, and Sushil's app
built around it already exists.
You are right that some shapefiles are really heavy (thus the need for
mapshaper).
On Tue, Mar 31, 2015 at 11:50 AM, Bob Jolliffe <bobjolliffe@gmail.com> >>> wrote:
That is a handy enough library for dealing with projection all right.
But doesn't really help in reading the shapefile, which *might* be a
bit of a heavy lift for an app dealing with dbf and shp files. I'm
not convinced this would be a really good use of developer time.
QGis already does all the grunt work of reading in the shapefile and
dealing with the various oddnesses. Getting it to dump its output as
dhis orgunits seems like a relatively straightforward proposition.
But you can't assume everyone would want to use qgis.
Maybe another approach would be to create a simple less geeky ui on
top of ogr2ogr. dunno really. I'm not seeing a silver bullet here.
On 31 March 2015 at 10:12, Knut Staring <knutst@gmail.com> wrote:
> Much better use of developer time would be to include http://proj4js.org/ in
> Sushil's app, so that everything needed comes in one sweet package.
>
>
> On Tue, Mar 31, 2015 at 11:10 AM, Knut Staring <knutst@gmail.com> >>>> wrote:
>>
>> shp2gml is taken care of by ogr2ogr, as described in our manual.
>>
>> Yes, someone could write a QGis plugin, but why should they, since
we are
>> now working on a web app to do this. Granted, it does not (yet)
include
>> reprojection, but it does include the other crucial step of
generalisation
>> using http://www.mapshaper.org/\.
>>
>> On Tue, Mar 31, 2015 at 9:53 AM, Bob Jolliffe <bobjolliffe@gmail.com >>>> > >>>> >> wrote:
>>>
>>> Of course shp2gml tools already exist - they could be described as a
>>> bit geeky using the command line. Though there is need for some of
>>> those geeky options for dealing with projection systems and the
like.
>>>
>>> I think someone with some python know how could write a neat little
>>> QGis plugin to export to dhis2 format which *might* make it easier
for
>>> users.
>>>
>>> On 30 March 2015 at 18:59, Lars Helge Øverland <larshelge@gmail.com >>>> > >>>> >>> wrote:
>>> > Shape file import sounds like an ideal task for an external
developer,
>>> > as it
>>> > could transform the shapefile content and pipe it into the GML
importer
>>> > and
>>> > hence does not require lots of DHIS 2 knowledge.
>>> >
>>> > Lars
>>> >
>>> >
>>> > _______________________________________________
>>> > Mailing list: DHIS 2 developers in Launchpad
>>> > Post to : dhis2-devs@lists.launchpad.net
>>> > Unsubscribe : DHIS 2 developers in Launchpad
>>> > More help : ListHelp - Launchpad Help
>>> >
>>>
>>> _______________________________________________
>>> Mailing list: DHIS 2 developers in Launchpad
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : DHIS 2 developers in Launchpad
>>> More help : ListHelp - Launchpad Help
>>
>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> Norway: +4791880522
>> Skype: knutstar
>> http://dhis2.org
>
>
>
>
> --
> Knut Staring
> Dept. of Informatics, University of Oslo
> Norway: +4791880522
> Skype: knutstar
> http://dhis2.org
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar http://dhis2.org
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar http://dhis2.org
There are forces which drive