Update of gml2dxf.xsl

Moving this discussion to the mailing list.

Hi Lars, could you please update

dhis-2/dhis-services/dhis-service-importexport/src/main/resources/transform?

I thought the conclusion of Jason's decimal saga was that we were not going
to cut the decimals?

Nope. Let me set out the issues in detail:

1) One concern is that DHIS2 should be able to work as a repository of
OrgUnit info, including shapes, and potentially also offer this data
through web services to clients other than the OpenHealthMapper (OHM).
Therefore, we should not reduce the precision of the orginal GIS data.
For this, we may need to store the original files separately from the
OrgUnit table. However, I am not fully convinced we need this for now
- it could be a blueprint for 2.0.6.

2) The most important concern is the need to supply the OHM client
with GeoJSON files that are not too large. GeoJSON files are 25-70%
larger than shapefiles. We cannot ask people with slow connections to
download 15 MB just to see a map - and many browsers will choke under
the burden of processing such large files.

3) Furthermore, the shape-to-GML conversion algorithm used by ogr2ogr
(which is a very good tool), ALWAYS results in each coordinate having
15 decimals, regardless of the precision of the original file, whereas
the shape-to-GeoJSON converter does not do that. I illustrated that
clearly several times in this discussion. Our method of importing from
shapefiles triples the size of what we store, transmit and process.
Let me illustrate:

I have a shapefile with all the countries in the world. This is 6 MB.
Converting to GeoJSON makes it 9.95 MB. Converting to GML makes it
14.7 MB.

Here are the values for the first coordinate using ogr2ogr in three
different ways:
Shape-to-GeoJSON -69.882233
GeoJSON-to GML -69.882232999999999
Shape-to-GML -69.882232666015625

Rounding to 4 decimals would preserve precision down to 10 m, which
is sufficient for our purposes (we are not using the files as input
for building construction), but 5 or 6 should also work.

Knut

···

2010/8/23 Lars Helge Øverland <larshelge@gmail.com>:

On Mon, Aug 23, 2010 at 5:11 PM, Knut Staring <knutst@gmail.com> wrote:

Knut

On Tue, Aug 17, 2010 at 11:59 AM, Knut Staring <knutst@gmail.com> wrote:
> Hi guys,
>
> In line with our earlier discussions, I have updated the gml2dxf
> transformation to round to 4 decimals (corresponding to an accuracy of
> 10 meters), which will reduce the geojson for polygons significantly.
> We may want to keep one more decimal in the case of points, should not
> be too hard.
>
> However, I am very rusty in terms of committing now with branches and
> merging and stuff - could one of you pls help out?
>
> The file is in
> dhis-2\dhis-services\dhis-service-importexport\target\transform
>
> Thanks,
> Knut
>

--
Cheers,
Knut Staring

--
Cheers,
Knut Staring

I am fine with this for presentation purposes, whatever works, speeds
up the map, and does not unduly degrade any presentation. 10 m
precision seems more than enough for most purposes as Knut mentions.

<RANT>
The issue of a repository is something entirely different. I would
suggest this be an entirely separate discussion, and a whole big set
of blueprints. DHIS2 does not store enough metadata (or do it in a
flexible enough fashion) for it to serve as a repository. Although we
have made some progress in arriving at a more comprehensive view of a
data model of a health facility, and what metadata elements actually
describe it, there is still yet no agreement or adoption by the
broader community who deal with this sort of stuff on what should be
there. We will have a consultative meeting in Geneva in September to
discuss this in more detail. The upshot is however, that DHIS2 stores
a portion of the metadata about a facility, including the geographic
coordinates, in some type of truncated form for its own purpose.

I have mentioned many times, and will mention it again, that not all
entities use WGS84 (Geographic lat/long) for storing of their
geographic data, and there is no reason that a so-called repository
should force this on them. Although we should be capable of dealing
with other projection systems in DHIS2, it is of course work to make
it happen. UTM for example does not use decimals, even for meter level
precision, so this discussion simply applies to one coordinate system
that we have chosen to implement. Thinking about other applications,
such as humanitarian, not all coordinate are necessarily numeric even,
they may be in an arbitrary (e.g. military grid) reference system. We
clearly should not think about reproducing the geometric
transformations that something like Geoserver (actually Geotools)
handles, but rather be able to consume data from something that can,
and transform it into coordinates that the application actually
understands.
</RANT>

Regards,
Jason

···

On Tue, Aug 24, 2010 at 9:31 AM, Knut Staring <knutst@gmail.com> wrote:

Moving this discussion to the mailing list.

2010/8/23 Lars Helge Øverland <larshelge@gmail.com>:

On Mon, Aug 23, 2010 at 5:11 PM, Knut Staring <knutst@gmail.com> wrote:

Hi Lars, could you please update

dhis-2/dhis-services/dhis-service-importexport/src/main/resources/transform?

I thought the conclusion of Jason's decimal saga was that we were not going
to cut the decimals?

Nope. Let me set out the issues in detail:

1) One concern is that DHIS2 should be able to work as a repository of
OrgUnit info, including shapes, and potentially also offer this data
through web services to clients other than the OpenHealthMapper (OHM).
Therefore, we should not reduce the precision of the orginal GIS data.
For this, we may need to store the original files separately from the
OrgUnit table. However, I am not fully convinced we need this for now
- it could be a blueprint for 2.0.6.

2) The most important concern is the need to supply the OHM client
with GeoJSON files that are not too large. GeoJSON files are 25-70%
larger than shapefiles. We cannot ask people with slow connections to
download 15 MB just to see a map - and many browsers will choke under
the burden of processing such large files.

3) Furthermore, the shape-to-GML conversion algorithm used by ogr2ogr
(which is a very good tool), ALWAYS results in each coordinate having
15 decimals, regardless of the precision of the original file, whereas
the shape-to-GeoJSON converter does not do that. I illustrated that
clearly several times in this discussion. Our method of importing from
shapefiles triples the size of what we store, transmit and process.
Let me illustrate:

I have a shapefile with all the countries in the world. This is 6 MB.
Converting to GeoJSON makes it 9.95 MB. Converting to GML makes it
14.7 MB.

Here are the values for the first coordinate using ogr2ogr in three
different ways:
Shape-to-GeoJSON -69.882233
GeoJSON-to GML -69.882232999999999
Shape-to-GML -69.882232666015625

Rounding to 4 decimals would preserve precision down to 10 m, which
is sufficient for our purposes (we are not using the files as input
for building construction), but 5 or 6 should also work.

Knut

Knut

On Tue, Aug 17, 2010 at 11:59 AM, Knut Staring <knutst@gmail.com> wrote:
> Hi guys,
>
> In line with our earlier discussions, I have updated the gml2dxf
> transformation to round to 4 decimals (corresponding to an accuracy of
> 10 meters), which will reduce the geojson for polygons significantly.
> We may want to keep one more decimal in the case of points, should not
> be too hard.
>
> However, I am very rusty in terms of committing now with branches and
> merging and stuff - could one of you pls help out?
>
> The file is in
> dhis-2\dhis-services\dhis-service-importexport\target\transform
>
> Thanks,
> Knut
>

--
Cheers,
Knut Staring

--
Cheers,
Knut Staring

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+17069260025

Thanks, Jason,

I fully agree that we should aspire to allow DHIS2 to work as a
national repository for metadata. On the GIS side, I see this most
realistically done through the integration of PostGIS and Geoserver
(and GeoNetwork?). An added benefit of such integration would be the
ability to complement the GeoJSON with WMS images, which in many cases
would mean smaller downloads. So we do need a number of new blueprints
- and also for versioning of metadata, as Bob has raised repeatedly.

One very crucial issue that is linked to this is the use of
identifiers. I am currently about to embark on linking the OHM to
external applications - hopefully using SDMX and some way to
automatically trigger its generation and import. However, I would
prefer to use ISO country codes rather than country names. Suggestions
welcome.

Knut

···

On Tue, Aug 24, 2010 at 9:45 AM, Jason Pickering <jason.p.pickering@gmail.com> wrote:

I am fine with this for presentation purposes, whatever works, speeds
up the map, and does not unduly degrade any presentation. 10 m
precision seems more than enough for most purposes as Knut mentions.

<RANT>
The issue of a repository is something entirely different. I would
suggest this be an entirely separate discussion, and a whole big set
of blueprints. DHIS2 does not store enough metadata (or do it in a
flexible enough fashion) for it to serve as a repository. Although we
have made some progress in arriving at a more comprehensive view of a
data model of a health facility, and what metadata elements actually
describe it, there is still yet no agreement or adoption by the
broader community who deal with this sort of stuff on what should be
there. We will have a consultative meeting in Geneva in September to
discuss this in more detail. The upshot is however, that DHIS2 stores
a portion of the metadata about a facility, including the geographic
coordinates, in some type of truncated form for its own purpose.

I have mentioned many times, and will mention it again, that not all
entities use WGS84 (Geographic lat/long) for storing of their
geographic data, and there is no reason that a so-called repository
should force this on them. Although we should be capable of dealing
with other projection systems in DHIS2, it is of course work to make
it happen. UTM for example does not use decimals, even for meter level
precision, so this discussion simply applies to one coordinate system
that we have chosen to implement. Thinking about other applications,
such as humanitarian, not all coordinate are necessarily numeric even,
they may be in an arbitrary (e.g. military grid) reference system. We
clearly should not think about reproducing the geometric
transformations that something like Geoserver (actually Geotools)
handles, but rather be able to consume data from something that can,
and transform it into coordinates that the application actually
understands.
</RANT>

Regards,
Jason

On Tue, Aug 24, 2010 at 9:31 AM, Knut Staring <knutst@gmail.com> wrote:

Moving this discussion to the mailing list.

2010/8/23 Lars Helge Øverland <larshelge@gmail.com>:

On Mon, Aug 23, 2010 at 5:11 PM, Knut Staring <knutst@gmail.com> wrote:

Hi Lars, could you please update

dhis-2/dhis-services/dhis-service-importexport/src/main/resources/transform?

I thought the conclusion of Jason's decimal saga was that we were not going
to cut the decimals?

Nope. Let me set out the issues in detail:

1) One concern is that DHIS2 should be able to work as a repository of
OrgUnit info, including shapes, and potentially also offer this data
through web services to clients other than the OpenHealthMapper (OHM).
Therefore, we should not reduce the precision of the orginal GIS data.
For this, we may need to store the original files separately from the
OrgUnit table. However, I am not fully convinced we need this for now
- it could be a blueprint for 2.0.6.

2) The most important concern is the need to supply the OHM client
with GeoJSON files that are not too large. GeoJSON files are 25-70%
larger than shapefiles. We cannot ask people with slow connections to
download 15 MB just to see a map - and many browsers will choke under
the burden of processing such large files.

3) Furthermore, the shape-to-GML conversion algorithm used by ogr2ogr
(which is a very good tool), ALWAYS results in each coordinate having
15 decimals, regardless of the precision of the original file, whereas
the shape-to-GeoJSON converter does not do that. I illustrated that
clearly several times in this discussion. Our method of importing from
shapefiles triples the size of what we store, transmit and process.
Let me illustrate:

I have a shapefile with all the countries in the world. This is 6 MB.
Converting to GeoJSON makes it 9.95 MB. Converting to GML makes it
14.7 MB.

Here are the values for the first coordinate using ogr2ogr in three
different ways:
Shape-to-GeoJSON -69.882233
GeoJSON-to GML -69.882232999999999
Shape-to-GML -69.882232666015625

Rounding to 4 decimals would preserve precision down to 10 m, which
is sufficient for our purposes (we are not using the files as input
for building construction), but 5 or 6 should also work.

Knut

Knut

On Tue, Aug 17, 2010 at 11:59 AM, Knut Staring <knutst@gmail.com> wrote:
> Hi guys,
>
> In line with our earlier discussions, I have updated the gml2dxf
> transformation to round to 4 decimals (corresponding to an accuracy of
> 10 meters), which will reduce the geojson for polygons significantly.
> We may want to keep one more decimal in the case of points, should not
> be too hard.
>
> However, I am very rusty in terms of committing now with branches and
> merging and stuff - could one of you pls help out?
>
> The file is in
> dhis-2\dhis-services\dhis-service-importexport\target\transform
>
> Thanks,
> Knut
>

--
Cheers,
Knut Staring

--
Cheers,
Knut Staring

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+17069260025

--
Cheers,
Knut Staring

One more issue regarding the GML import:

In order to really be able to import a full hierarchy, it would be
great if there was also a way to specify the parent for each orgunit.
I.e. in addition to ogr:Name being obligatory, I think we should allow
for (but not require) ogr:Parent. Comments on the feasibility of this?

Knut

···

On Tue, Aug 24, 2010 at 10:05 AM, Knut Staring <knutst@gmail.com> wrote:

Thanks, Jason,

I fully agree that we should aspire to allow DHIS2 to work as a
national repository for metadata. On the GIS side, I see this most
realistically done through the integration of PostGIS and Geoserver
(and GeoNetwork?). An added benefit of such integration would be the
ability to complement the GeoJSON with WMS images, which in many cases
would mean smaller downloads. So we do need a number of new blueprints
- and also for versioning of metadata, as Bob has raised repeatedly.

One very crucial issue that is linked to this is the use of
identifiers. I am currently about to embark on linking the OHM to
external applications - hopefully using SDMX and some way to
automatically trigger its generation and import. However, I would
prefer to use ISO country codes rather than country names. Suggestions
welcome.

Knut

On Tue, Aug 24, 2010 at 9:45 AM, Jason Pickering > <jason.p.pickering@gmail.com> wrote:

I am fine with this for presentation purposes, whatever works, speeds
up the map, and does not unduly degrade any presentation. 10 m
precision seems more than enough for most purposes as Knut mentions.

<RANT>
The issue of a repository is something entirely different. I would
suggest this be an entirely separate discussion, and a whole big set
of blueprints. DHIS2 does not store enough metadata (or do it in a
flexible enough fashion) for it to serve as a repository. Although we
have made some progress in arriving at a more comprehensive view of a
data model of a health facility, and what metadata elements actually
describe it, there is still yet no agreement or adoption by the
broader community who deal with this sort of stuff on what should be
there. We will have a consultative meeting in Geneva in September to
discuss this in more detail. The upshot is however, that DHIS2 stores
a portion of the metadata about a facility, including the geographic
coordinates, in some type of truncated form for its own purpose.

I have mentioned many times, and will mention it again, that not all
entities use WGS84 (Geographic lat/long) for storing of their
geographic data, and there is no reason that a so-called repository
should force this on them. Although we should be capable of dealing
with other projection systems in DHIS2, it is of course work to make
it happen. UTM for example does not use decimals, even for meter level
precision, so this discussion simply applies to one coordinate system
that we have chosen to implement. Thinking about other applications,
such as humanitarian, not all coordinate are necessarily numeric even,
they may be in an arbitrary (e.g. military grid) reference system. We
clearly should not think about reproducing the geometric
transformations that something like Geoserver (actually Geotools)
handles, but rather be able to consume data from something that can,
and transform it into coordinates that the application actually
understands.
</RANT>

Regards,
Jason

On Tue, Aug 24, 2010 at 9:31 AM, Knut Staring <knutst@gmail.com> wrote:

Moving this discussion to the mailing list.

2010/8/23 Lars Helge Øverland <larshelge@gmail.com>:

On Mon, Aug 23, 2010 at 5:11 PM, Knut Staring <knutst@gmail.com> wrote:

Hi Lars, could you please update

dhis-2/dhis-services/dhis-service-importexport/src/main/resources/transform?

I thought the conclusion of Jason's decimal saga was that we were not going
to cut the decimals?

Nope. Let me set out the issues in detail:

1) One concern is that DHIS2 should be able to work as a repository of
OrgUnit info, including shapes, and potentially also offer this data
through web services to clients other than the OpenHealthMapper (OHM).
Therefore, we should not reduce the precision of the orginal GIS data.
For this, we may need to store the original files separately from the
OrgUnit table. However, I am not fully convinced we need this for now
- it could be a blueprint for 2.0.6.

2) The most important concern is the need to supply the OHM client
with GeoJSON files that are not too large. GeoJSON files are 25-70%
larger than shapefiles. We cannot ask people with slow connections to
download 15 MB just to see a map - and many browsers will choke under
the burden of processing such large files.

3) Furthermore, the shape-to-GML conversion algorithm used by ogr2ogr
(which is a very good tool), ALWAYS results in each coordinate having
15 decimals, regardless of the precision of the original file, whereas
the shape-to-GeoJSON converter does not do that. I illustrated that
clearly several times in this discussion. Our method of importing from
shapefiles triples the size of what we store, transmit and process.
Let me illustrate:

I have a shapefile with all the countries in the world. This is 6 MB.
Converting to GeoJSON makes it 9.95 MB. Converting to GML makes it
14.7 MB.

Here are the values for the first coordinate using ogr2ogr in three
different ways:
Shape-to-GeoJSON -69.882233
GeoJSON-to GML -69.882232999999999
Shape-to-GML -69.882232666015625

Rounding to 4 decimals would preserve precision down to 10 m, which
is sufficient for our purposes (we are not using the files as input
for building construction), but 5 or 6 should also work.

Knut

Knut

On Tue, Aug 17, 2010 at 11:59 AM, Knut Staring <knutst@gmail.com> wrote:
> Hi guys,
>
> In line with our earlier discussions, I have updated the gml2dxf
> transformation to round to 4 decimals (corresponding to an accuracy of
> 10 meters), which will reduce the geojson for polygons significantly.
> We may want to keep one more decimal in the case of points, should not
> be too hard.
>
> However, I am very rusty in terms of committing now with branches and
> merging and stuff - could one of you pls help out?
>
> The file is in
> dhis-2\dhis-services\dhis-service-importexport\target\transform
>
> Thanks,
> Knut
>

--
Cheers,
Knut Staring

--
Cheers,
Knut Staring

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+17069260025

--
Cheers,
Knut Staring

--
Cheers,
Knut Staring