[Branch ~dhis2-devs-core/dhis2/trunk] Rev 19296: WIP proper error reporting on failed GML parsing during import

DHIS_2_Users · 5 June 2015 12:57

Calle_Hedberg · 25 June 2015 15:54

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

···

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Halvdan_Grelland · 25 June 2015 16:16

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

···

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Halvdan_Grelland · 25 June 2015 23:52

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

···

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

morten · 26 June 2015 02:09

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

···

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Morten

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle_Hedberg · 26 June 2015 08:12

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

···

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle_Hedberg · 26 June 2015 09:17

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])
INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])
INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])
INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])
INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])
INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BitClass.<init>(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.<init>(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

···

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Dan · 26 June 2015 12:17

I’ve seen this error in the past on import of GML the cause was an Org Unit had a lot of coordinates (as in greater than 200 it was every turn of Lake Victoria) and this caused the importer to hang.

···

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296
committer: Halvdan Hoem Grelland halvdanhg@gmail.com
branch nick: dhis2
timestamp: Fri 2015-06-05 14:56:34 +0200
message:
WIP proper error reporting on failed GML parsing during import
added:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
modified:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–
lp:dhis2
https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.
To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000
@@ -32,6 +32,8 @@
import com.google.common.base.Strings;
import com.google.common.collect.Iterators;
import com.google.common.collect.Maps;
+import org.apache.commons.io.IOUtils;
+import org.hibernate.Hibernate;
import org.hisp.dhis.common.IdentifiableObjectManager;
import org.hisp.dhis.common.IdentifiableProperty;
import org.hisp.dhis.common.MergeStrategy;
@@ -88,13 +90,91 @@
// GmlImportService implementation
// -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}

@Override
public MetaData fromGml( InputStream inputStream )
throws IOException, TransformerException
{
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }

   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );
}

@Transactional
@Override
public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )
{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000
@@ -62,6 +62,24 @@
throws IOException, TransformerException;
 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000
@@ -0,0 +1,70 @@
+package org.hisp.dhis.dxf2.gml;
+
+import org.hisp.dhis.dxf2.metadata.MetaData;
+
+import java.io.InputStream;
+
+/**
Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/
+public final class GmlPreProcessingResult
+{

private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}
=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000
@@ -117,4 +117,6 @@
assertEquals( 1, units.get( “Blindern”).getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
assertEquals( 76, units.get( “Forskningsparken” ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
}
+

// TODO Add test for GmlImportService#preProcessGml(InputStream)
}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’
— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000
+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000
@@ -77,7 +77,7 @@
{
taskExecutor.execute( task );
}

@Override
public boolean scheduleTask( String key, Runnable task, String cronExpr )
{

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000
@@ -187,7 +187,7 @@
}
else if ( “gml”.equals( importFormat ) )
{
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );
   }
   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )
   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000
@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

+import org.apache.commons.lang.exception.ExceptionUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.hisp.dhis.dxf2.common.ImportOptions;
import org.hisp.dhis.dxf2.gml.GmlImportService;
+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
import org.hisp.dhis.scheduling.TaskId;
+import org.hisp.dhis.system.notification.NotificationLevel;
+import org.hisp.dhis.system.notification.Notifier;
+import org.springframework.web.util.HtmlUtils;
+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;
-import java.io.IOException;
import java.io.InputStream;

/**
@@ -50,12 +54,15 @@
 private String userUid;
// -------------------------------------------------------------------------
// Dependencies
// -------------------------------------------------------------------------

private GmlImportService gmlImportService;

private Notifier notifier;

private ImportOptions importOptions;

private InputStream inputStream;
@@ -64,11 +71,12 @@
// Constructors
// -------------------------------------------------------------------------

public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;
this.gmlImportService = gmlImportService;
   this.notifier = notifier;
   this.importOptions = importOptions;
   this.inputStream = inputStream;
   this.taskId = taskId;
@@ -83,15 +91,37 @@
{
importOptions.setImportStrategy( “update” ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’
— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000
@@ -300,6 +300,7 @@
ignored=Ignored
conflicts=Conflicts
no_conflicts_found=No conflicts found
+no_import_summary_available=No summary available
type=Type
count=Count
export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’
— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000
@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )
+ +#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)
$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@
#else

$i18n.getString( "no_conflicts_found" )
#end +#else + $i18n.getString( "no_summary_available" ) +#end \ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Knut_Staring · 26 June 2015 12:48

I would recommend running your shapefile through Mapshaper to simplify it before conversion to GML

···

On Fri, Jun 26, 2015 at 2:17 PM, Dan Cocos dan@dancocos.com wrote:

I’ve seen this error in the past on import of GML the cause was an Org Unit had a lot of coordinates (as in greater than 200 it was every turn of Lake Victoria) and this caused the importer to hang.

On Jun 26, 2015, at 5:17 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BitClass.<init>(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.<init>(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296
committer: Halvdan Hoem Grelland halvdanhg@gmail.com
branch nick: dhis2
timestamp: Fri 2015-06-05 14:56:34 +0200
message:
WIP proper error reporting on failed GML parsing during import
added:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
modified:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–
lp:dhis2
https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.
To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000
@@ -32,6 +32,8 @@
import com.google.common.base.Strings;
import com.google.common.collect.Iterators;
import com.google.common.collect.Maps;
+import org.apache.commons.io.IOUtils;
+import org.hibernate.Hibernate;
import org.hisp.dhis.common.IdentifiableObjectManager;
import org.hisp.dhis.common.IdentifiableProperty;
import org.hisp.dhis.common.MergeStrategy;
@@ -88,13 +90,91 @@
// GmlImportService implementation
// -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}

@Override
public MetaData fromGml( InputStream inputStream )
throws IOException, TransformerException
{
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }

   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );
}

@Transactional
@Override
public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )
{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000
@@ -62,6 +62,24 @@
throws IOException, TransformerException;
 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000
@@ -0,0 +1,70 @@
+package org.hisp.dhis.dxf2.gml;
+
+import org.hisp.dhis.dxf2.metadata.MetaData;
+
+import java.io.InputStream;
+
+/**
Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/
+public final class GmlPreProcessingResult
+{

private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}
=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000
@@ -117,4 +117,6 @@
assertEquals( 1, units.get( “Blindern”).getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
assertEquals( 76, units.get( “Forskningsparken” ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
}
+

// TODO Add test for GmlImportService#preProcessGml(InputStream)
}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’
— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000
+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000
@@ -77,7 +77,7 @@
{
taskExecutor.execute( task );
}

@Override
public boolean scheduleTask( String key, Runnable task, String cronExpr )
{

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000
@@ -187,7 +187,7 @@
}
else if ( “gml”.equals( importFormat ) )
{
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );
   }
   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )
   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000
@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

+import org.apache.commons.lang.exception.ExceptionUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.hisp.dhis.dxf2.common.ImportOptions;
import org.hisp.dhis.dxf2.gml.GmlImportService;
+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
import org.hisp.dhis.scheduling.TaskId;
+import org.hisp.dhis.system.notification.NotificationLevel;
+import org.hisp.dhis.system.notification.Notifier;
+import org.springframework.web.util.HtmlUtils;
+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;
-import java.io.IOException;
import java.io.InputStream;

/**
@@ -50,12 +54,15 @@
 private String userUid;
// -------------------------------------------------------------------------
// Dependencies
// -------------------------------------------------------------------------

private GmlImportService gmlImportService;

private Notifier notifier;

private ImportOptions importOptions;

private InputStream inputStream;
@@ -64,11 +71,12 @@
// Constructors
// -------------------------------------------------------------------------

public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;
this.gmlImportService = gmlImportService;
   this.notifier = notifier;
   this.importOptions = importOptions;
   this.inputStream = inputStream;
   this.taskId = taskId;
@@ -83,15 +91,37 @@
{
importOptions.setImportStrategy( “update” ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’
— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000
@@ -300,6 +300,7 @@
ignored=Ignored
conflicts=Conflicts
no_conflicts_found=No conflicts found
+no_import_summary_available=No summary available
type=Type
count=Count
export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’
— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000
@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )
+ +#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)
$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@
#else

$i18n.getString( "no_conflicts_found" )
#end +#else + $i18n.getString( "no_summary_available" ) +#end \ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Knut_Staring · 26 June 2015 12:48

Alternatively, try out Sushil’s app

···

On Fri, Jun 26, 2015 at 2:48 PM, Knut Staring knutst@gmail.com wrote:

I would recommend running your shapefile through Mapshaper to simplify it before conversion to GML

–

On Fri, Jun 26, 2015 at 2:17 PM, Dan Cocos dan@dancocos.com wrote:

I’ve seen this error in the past on import of GML the cause was an Org Unit had a lot of coordinates (as in greater than 200 it was every turn of Lake Victoria) and this caused the importer to hang.

On Jun 26, 2015, at 5:17 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)

at java.util.regex.Pattern$BitClass.<init>(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.<init>(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296
committer: Halvdan Hoem Grelland halvdanhg@gmail.com
branch nick: dhis2
timestamp: Fri 2015-06-05 14:56:34 +0200
message:
WIP proper error reporting on failed GML parsing during import
added:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
modified:
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–
lp:dhis2
https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.
To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000
@@ -32,6 +32,8 @@
import com.google.common.base.Strings;
import com.google.common.collect.Iterators;
import com.google.common.collect.Maps;
+import org.apache.commons.io.IOUtils;
+import org.hibernate.Hibernate;
import org.hisp.dhis.common.IdentifiableObjectManager;
import org.hisp.dhis.common.IdentifiableProperty;
import org.hisp.dhis.common.MergeStrategy;
@@ -88,13 +90,91 @@
// GmlImportService implementation
// -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}

@Override
public MetaData fromGml( InputStream inputStream )
throws IOException, TransformerException
{
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }

   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );
}

@Transactional
@Override
public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )
{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000
@@ -62,6 +62,24 @@
throws IOException, TransformerException;
 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000
@@ -0,0 +1,70 @@
+package org.hisp.dhis.dxf2.gml;
+
+import org.hisp.dhis.dxf2.metadata.MetaData;
+
+import java.io.InputStream;
+
+/**
Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/
+public final class GmlPreProcessingResult
+{

private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}
=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’
— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000
+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000
@@ -117,4 +117,6 @@
assertEquals( 1, units.get( “Blindern”).getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
assertEquals( 76, units.get( “Forskningsparken” ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
}
+

// TODO Add test for GmlImportService#preProcessGml(InputStream)
}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’
— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000
+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000
@@ -77,7 +77,7 @@
{
taskExecutor.execute( task );
}

@Override
public boolean scheduleTask( String key, Runnable task, String cronExpr )
{

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000
@@ -187,7 +187,7 @@
}
else if ( “gml”.equals( importFormat ) )
{
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );
   }
   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )
   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’
— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000
@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

+import org.apache.commons.lang.exception.ExceptionUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.hisp.dhis.dxf2.common.ImportOptions;
import org.hisp.dhis.dxf2.gml.GmlImportService;
+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
import org.hisp.dhis.scheduling.TaskId;
+import org.hisp.dhis.system.notification.NotificationLevel;
+import org.hisp.dhis.system.notification.Notifier;
+import org.springframework.web.util.HtmlUtils;
+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;
-import java.io.IOException;
import java.io.InputStream;

/**
@@ -50,12 +54,15 @@
 private String userUid;
// -------------------------------------------------------------------------
// Dependencies
// -------------------------------------------------------------------------

private GmlImportService gmlImportService;

private Notifier notifier;

private ImportOptions importOptions;

private InputStream inputStream;
@@ -64,11 +71,12 @@
// Constructors
// -------------------------------------------------------------------------

public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;
this.gmlImportService = gmlImportService;
   this.notifier = notifier;
   this.importOptions = importOptions;
   this.inputStream = inputStream;
   this.taskId = taskId;
@@ -83,15 +91,37 @@
{
importOptions.setImportStrategy( “update” ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’
— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000
@@ -300,6 +300,7 @@
ignored=Ignored
conflicts=Conflicts
no_conflicts_found=No conflicts found
+no_import_summary_available=No summary available
type=Type
count=Count
export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’
— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000
+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000
@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )
+ +#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)
$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@
#else

$i18n.getString( "no_conflicts_found" )
#end +#else + $i18n.getString( "no_summary_available" ) +#end \ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Halvdan_Grelland · 26 June 2015 18:26

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

···

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Halvdan_Grelland · 26 June 2015 23:03

Ok, so I had a closer look.

The issue is not a performance one like I first indicated, but rather a combination of a few factors:

First, the GML does a pre-merge with the database before sending the orgunit updates on to the meta data importer. Since your DB has AttributeValues for the orgunit in question (three different attributevalues) these are included in this pre-merge and are imported along with the geospatial updates. Removing them (from the intermediate dxf2 stream) causes the import to work again, but the attributevalues end up being overwritten (i.e. removed from the DB), which we obviously don’t want.

Second: The core of the issue seems to be that the meta data importer can’t handle the attributevalues when they are present. This is also what’s reflected in the stacktraces given (which I was also able to reproduce on the ZA database).

Could you please look into this, Morten? I’ve attached the generated dxf-payload which is given to the metadata importer with this email. The DB is given in an earlier email by Calle.

Halvdan

dxfstream.xml (264 KB)

···

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle_Hedberg · 27 June 2015 16:50

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

···

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Halvdan_Grelland · 27 June 2015 22:16

So. The stack size thing was just a qualified guess derived from the stack trace. It’s pretty obviously not the cause. I’ll wait on feedback from Morten before going any further on this.

···

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

morten · 1 July 2015 04:57

Hi guys

I’ve been trying to reproduce your bug this morning, but I’m not able to. I restored the db, and imported the xml without any issues, I tested on both 2.19 and 2.20, so I will need more information to reproduce.

This was the command I used to import:

curl -X POST -d @dxfstream.xml -H “Content-Type: application/xml” -v -u admin:district http://localhost:8080/api/metadata

···

–
Morten

On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

So. The stack size thing was just a qualified guess derived from the stack trace. It’s pretty obviously not the cause. I’ll wait on feedback from Morten before going any further on this.

On Jun 27, 2015 6:50 PM, “Calle Hedberg” calle.hedberg@gmail.com wrote:

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle_Hedberg · 1 July 2015 07:23

Morten,

I’m not using curl to import GML files - I’m using the normal user interface (apps → import/export → import GML file

Regards

Calle

···

On 1 July 2015 at 06:57, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi guys

I’ve been trying to reproduce your bug this morning, but I’m not able to. I restored the db, and imported the xml without any issues, I tested on both 2.19 and 2.20, so I will need more information to reproduce.

This was the command I used to import:

curl -X POST -d @dxfstream.xml -H “Content-Type: application/xml” -v -u admin:district http://localhost:8080/api/metadata

–

–
Morten

On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

So. The stack size thing was just a qualified guess derived from the stack trace. It’s pretty obviously not the cause. I’ll wait on feedback from Morten before going any further on this.

On Jun 27, 2015 6:50 PM, “Calle Hedberg” calle.hedberg@gmail.com wrote:

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

morten · 1 July 2015 07:28

Yes, but according to Halvdan he was able to reproduce with the XML file (which is the transformed GML import file) he gave me.

@Halvdan: maybe you can add some details?

···

–
Morten

On Wed, Jul 1, 2015 at 2:23 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Morten,

I’m not using curl to import GML files - I’m using the normal user interface (apps → import/export → import GML file

Regards

Calle

On 1 July 2015 at 06:57, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi guys

I’ve been trying to reproduce your bug this morning, but I’m not able to. I restored the db, and imported the xml without any issues, I tested on both 2.19 and 2.20, so I will need more information to reproduce.

This was the command I used to import:

curl -X POST -d @dxfstream.xml -H “Content-Type: application/xml” -v -u admin:district http://localhost:8080/api/metadata

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

So. The stack size thing was just a qualified guess derived from the stack trace. It’s pretty obviously not the cause. I’ll wait on feedback from Morten before going any further on this.

On Jun 27, 2015 6:50 PM, “Calle Hedberg” calle.hedberg@gmail.com wrote:

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle_Hedberg · 1 July 2015 07:28

Hi

sorry - early morning for me, wasn’t thinking. You will have to talk to Halvdan about that dxfstream.xml file - I’m not sure how/where he got that.

Otherwise, just try to import the GML file directly using the UI (i.e. follow the steps Halvdan took to replicate the error). Use the same 2.19 that I used over - just in case the root cause of this is a bug that’s been fixed during the last few days.

Regards

Calle

···

On 1 July 2015 at 09:23, Calle Hedberg calle.hedberg@gmail.com wrote:

Morten,

I’m not using curl to import GML files - I’m using the normal user interface (apps → import/export → import GML file

Regards

Calle

–

On 1 July 2015 at 06:57, Morten Olav Hansen mortenoh@gmail.com wrote:

Hi guys

I’ve been trying to reproduce your bug this morning, but I’m not able to. I restored the db, and imported the xml without any issues, I tested on both 2.19 and 2.20, so I will need more information to reproduce.

This was the command I used to import:

curl -X POST -d @dxfstream.xml -H “Content-Type: application/xml” -v -u admin:district http://localhost:8080/api/metadata

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

So. The stack size thing was just a qualified guess derived from the stack trace. It’s pretty obviously not the cause. I’ll wait on feedback from Morten before going any further on this.

On Jun 27, 2015 6:50 PM, “Calle Hedberg” calle.hedberg@gmail.com wrote:

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

bobj · 1 July 2015 09:12

Hi Calle

Getting stack overflow messages is almost always a symptom of infinite (or too much) recursion, Increasing the stack size just delays the symptoms presenting themselves (and makes the problem bigger). This is probably also what you have found with your experiments. Also it starts to cost in general as you increase the number of threads. So I think in general we find that the java default is adequate and I don’t think there is much need to further optimize it.

Having said that, this stack overflow business might carry some clue. @Morten you mention elsewhere in this thread “It seems to be looping in Attribute.toString()”. Are you sure it is looping or is there something recursive going on here? (note that xslt with very large nodesets could indeed also cause recursion problems)

Bob

···

On 27 June 2015 at 17:50, Calle Hedberg calle.hedberg@gmail.com wrote:

Halvdan,

I understand from your last email that the root cause of the problem is a 2.19 software issue - the meta data importer in 2.19 is not able to handle OrgUnit attribute values when present. That makes sense, and it is likely that the root cause of this bug / design flaw is a 2.19 change (since the same import worked fine in 2.18). Morten and/or Lars will hopefully know what change(s) caused this.

With regard to JVM running out of heap space, and in relation to the (outdated - very!) chapter 8.2.8 you referenced:

Firstly, with regard to the 150+ instances we are hosting on our data centre servers, we do have some constraints at the moment: Those servers have 32GB RAM only, and most instances are in reality running on 1-3GB. That will only change from next week, when we start moving to new servers with (initially) 128GB RAM. That should on average provide 6-8GB per instance (varying between 2 and 32GB).

Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 bits) memory allocations on my Windows notebook (32GB RAM) has been a nightmare - the people writing up java and tomcat documentation are useless, they seem unable to give any quantified advice at all where any normal person can find it. (In comparison, chapter 8.2.8 although outdated is super-clear). I am currently using the following setting for Java:

JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m

For some reason I/we do not understand, it seems Tomcat 8.23 does not use CATALINA_OPTS settings, so I ended up using the “Configure Tomcat” and then specify the following under the “Java” tab:

Java options:

-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0

-Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 8.0\endorsed

-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.0\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.0\conf\logging.properties

-XX:MaxPermSize=4096m

(NOTE: the -XX value here is different from what I currently have in the JAVA_OPTS environmental variable - I cannot find any info on which of them takes precedence

Also under that same “Java” tab on “Configure Tomcat”:

Initial memory pool: 4096MB

Maximum memory pool: 8192MB

Thread stack size: 4096KB (4MB)

Based on your initial advice to increase the stack size, I tried to increase it to 8MB (GML import still crashing), 64MB (GML import still crashing), and 132MB (GML import went completely haywire). Which is understandable, given your subsequent finding that the root cause of the problem was not related to heap/stack settings.

So I’ve set the thread stack size back to 4096KB (4MB) - even if I don’t know whether that’s correct or not (seems high to my layman eyes, but maybe DHIS2 require it?)

If anybody actually KNOWS what memory/stack sizes that I optimal for a 32GB Windows machine, I would appreciate the advice…

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 26 June 2015 at 20:26, Halvdan Grelland halvdanhg@gmail.com wrote:

Ok, I could have a look, but am preoccupied at the moment. Will get back to you.

In the meantime, it seems that you’re facing a stack overflow error, which could be caused by the JVM running out of heap space. Could you please check the allocated heap for tomcat? If it is not being set (in the CATALINA_OPTS env variable), please set it to something quite large and try again.

Under chapter 8.2.8 at the following link is a brief guide https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html

Please let me know how this works out for you.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-26 11:44 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

Here’s a stripped down version of the Free State instance (no records in the data value table). Restore it as za_fs_s (psql -U dhis -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)

You have got that single-record country GML file already.

I would appreciate any help in figuring out what’s happening. Just note that I’m using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.

Best regards

Calle

On 26 June 2015 at 11:17, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I ran it again (revision 19247) and pulled out the first part of the log error:

INFO 2015-06-26 11:04:18,338 User ‘Calle_Hedberg’ started import at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 seconds. (DefaultObjectBridge.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])

INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])

ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.StackOverflowError

at java.util.regex.Pattern$BmpCharProperty.(Unknown Source)

at java.util.regex.Pattern$BitClass.(Unknown Source)

at java.util.regex.Pattern.clazz(Unknown Source)

at java.util.regex.Pattern.sequence(Unknown Source)

at java.util.regex.Pattern.expr(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.util.regex.Pattern.(Unknown Source)

at java.util.regex.Pattern.compile(Unknown Source)

at java.lang.String.replaceAll(Unknown Source)

at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)

at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)

at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)

at org.hisp.dhis.attribute.Attribute_$$jvst804_10.toString(Attribute$$_jvst804_10.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

Any ideas?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 10:12, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

Firstly, yes we are using a custom revision - although it is SUPPOSED to be based on a 2.19 version from a few days ago. I will check that.

BUT I have tried the import using revision 19247 too, and the problem/behaviour is exactly the same - it hangs. Whereas if I import into an older version of the same provincial instance that still runs on 2.18, it imports fine.

The fact that the same GML file imports in the Sierra Leone demo - I cannot explain that, and the tomcat log only shows that Attribute.toString() looping…

Can I create a stripped down version of the instance and upload it for you to test?

Regards

Calle

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

On 26 June 2015 at 04:09, Morten Olav Hansen mortenoh@gmail.com wrote:

It seems to be looping in Attribute.toString(), I remember this was an issue at some point… but that was a long long time ago… so maybe you need a revision bump

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

–
Morten

On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland halvdanhg@gmail.com wrote:

I tried your example GML on both our demo servers (dev on 2.20 and demo on 2.19, same rev as your server) and it works just fine.

If you’re experiencing it hanging and not finishing you might be experiencing some kind of performance and/or memory issue. The coordinate set is fairly large (though it works well on our demos, but does take some time to finish). You can verify by trying yourself to create the orgunit (with the exact name) and import the GML on the Sierra Leone demo.

The fact that the dry run seems to “work” for you (as well as the stack trace you provided) also points at the issue not residing in the GML processing but rather in the actual import stage (the GML importer uses the metadata importer as it’s backend).

Another thing: The instance you linked to is using a revision of 16783 and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you using a custom build? Does it contain customizations to the core in any way? That would also be a possible source of issues, obviously.

You could try deploying this war https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war and replicate the issue using the za South Africa orgunit.

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

2015-06-26 0:35 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Halvdan,

The GML files I’m trying to import using 2.19 is the same GML files I’ve been using successfully for 2.17, 2.18 (so it’s using NAME). Something in the import has changed:

I have attached the small GML file I’m trying to import - it has the international boundary of SA, so there is only ONE GIS record (top node in all our OrgHierarchies). It imports with no problems in 2.18.

In 2.19, if I choose YES for dry run, it will complete with usual import summary etc.

If I choose NO for dry run, I get

2015-06-25 22:18:15
Importing 1 OrganisationUnits
2015-06-25 22:18:11
Importing meta-data
and there it hangs. Nothing gets imported.

The online instance you can try to import into is http://fs.dhis.dhmis.org/staging/

When I try to import into a local instance, it seems to enter some kind of endless loop based on the tomcat log:

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)

at org.hisp.dhis.attribute.Attribute_$$javassist_16.toString(Attribute$$_javassist_16.java)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at java.util.AbstractCollection.toString(Unknown Source)

at org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)

at java.lang.String.valueOf(Unknown Source)

at java.lang.StringBuilder.append(Unknown Source)

at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)

at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)

at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

On 25 June 2015 at 18:16, Halvdan Grelland halvdanhg@gmail.com wrote:

Hi Calle,

The commit you’ve replied to implements (but not fully, therefore ‘WIP’, there’s another commit following it pretty soon after) better error reporting feedback to the user through the interface. I.e. not having to see the stacktrace on the server to determine if and why something went wrong. I believe you have been among the people requesting this too

Also, as you’re referencing, we’ve made some other changes to GML import. In particular the importer will now match on uid, code or name, as oppsed to the former mode of matching only on name. This is not reflected in the UI as it is a backend enhancement only, and does not change the import options at all. It IS, however, documented and ready for you to read about. Short version: it matches on a prioritized list of uid → code → name. Long version: https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html

Be adviced that the commit you’ve referenced is towards development trunk, and is not backported to 2.19, the match on multiple identifiers, however, was introduced in 2.19 and should work in current 2.19 stable. If you’re having problems with it you can send me a full stack trace and preferably the GML input data.

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2015-06-25 17:54 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:

Hi,

I’m unable to import GML files using 2.19 (the same files import fine in 2.18) - the import simply hangs (endless rolling progress bar icon).

Will the above bug-fix sort out that problem - I do not understand what the above actually means, or if the GML format has changed in some way.

Note also that the final 2.19 release was supposed to include a more diversified GML import - but I cannot see WHERE to do that (I still see the same two buttons: choose file and import, either dry run or not)

Regards

Calle

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On 5 June 2015 at 14:57, noreply@launchpad.net wrote:

revno: 19296

committer: Halvdan Hoem Grelland halvdanhg@gmail.com

branch nick: dhis2

timestamp: Fri 2015-06-05 14:56:34 +0200

message:

WIP proper error reporting on failed GML parsing during import

added:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java

modified:

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java

dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java

dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java

dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java

dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties

dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java 2015-06-05 12:56:34 +0000

@@ -32,6 +32,8 @@

import com.google.common.base.Strings;

import com.google.common.collect.Iterators;

import com.google.common.collect.Maps;

+import org.apache.commons.io.IOUtils;

+import org.hibernate.Hibernate;

import org.hisp.dhis.common.IdentifiableObjectManager;

import org.hisp.dhis.common.IdentifiableProperty;

import org.hisp.dhis.common.MergeStrategy;

@@ -88,13 +90,91 @@
 // GmlImportService implementation

 // -------------------------------------------------------------------------
@Transactional

@Override

public GmlPreProcessingResult preProcessGml( InputStream inputStream )

{
   InputStream dxfStream = null;
   MetaData metaData = null;
   try
   {
       dxfStream = transformGml( inputStream );
       metaData = renderService.fromXml( dxfStream, MetaData.class );
   }
   catch ( IOException | TransformerException e )
   {
       return GmlPreProcessingResult.failure( e );
   }
   finally
   {
       IOUtils.closeQuietly( dxfStream );
   }
   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
   matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
   Map<String, OrganisationUnit> persistedUidMap  = getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
   Map<String, OrganisationUnit> persistedCodeMap = getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
   Map<String, OrganisationUnit> persistedNameMap = getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
   Iterator<OrganisationUnit> persistedIterator = Iterators.concat( persistedUidMap.values().iterator(),
       persistedCodeMap.values().iterator(), persistedNameMap.values().iterator() );
   while ( persistedIterator.hasNext() )
   {
       OrganisationUnit persisted = persistedIterator.next(), imported = null;
       if ( !Strings.isNullOrEmpty( persisted.getUid() ) && uidMap.containsKey( persisted.getUid() ) )
       {
           imported = uidMap.get( persisted.getUid() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
       {
           imported = codeMap.get( persisted.getCode() );
       }
       else if ( !Strings.isNullOrEmpty( persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
       {
           imported = nameMap.get( persisted.getName() );
       }
       if ( imported == null || imported.getCoordinates() == null || imported.getFeatureType() == null )
       {
           continue; // Failed to dereference a persisted entity for this org unit or geo data incomplete/missing, therefore ignore
       }
       mergeNonGeoData( persisted, imported );
   }
   return GmlPreProcessingResult.success( metaData );
}
 @Override

 public MetaData fromGml( InputStream inputStream )

     throws IOException, TransformerException

 {
   InputStream dxfStream = transformGml( inputStream );
   MetaData metaData = renderService.fromXml( dxfStream, MetaData.class );
   dxfStream.close();
   InputStream dxfStream;
   MetaData metaData;
   try
   {
       dxfStream = transformGml( inputStream );
   }
   catch (Exception e)
   {
       dxfStream = null;
   }
   if(dxfStream != null)
   {
       metaData = renderService.fromXml( dxfStream, MetaData.class );
       dxfStream.close();
   }
   else
   {
       return null;
   }



   Map<String, OrganisationUnit> uidMap  = Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
@@ -143,6 +223,7 @@
     importService.importMetaData( userUid, fromGml( inputStream ), importOptions, taskId );

 }
@Transactional

@Override

public void importGml( MetaData metaData, String userUid, ImportOptions importOptions, TaskId taskId )

{

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-03 15:00:19 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java 2015-06-05 12:56:34 +0000

@@ -62,6 +62,24 @@
     throws IOException, TransformerException;



 /**
* Pre-process a GML document. The process, in short, entails the following:
* <ol>
*     <li>Parse the GML payload and transform it into DXF2 format</li>
*     <li>Get the given identifiers (uid, code or name) from the parsed payload and fetch
*     the corresponding entities from the DB</li>
*     <li>Merge the geospatial data given in the input GML into DB entities and return</li>
* </ol>
*
* The result of this process in returned in a {@link GmlPreProcessingResult} which
* encapsulates the returned {@link MetaData} object or the exception in cause of parse
* failure due to IO errors or malformed input.
*
* @param gmlInputStream the InputStream providing the GML input.
* @return a GmlPreProcessingResult representing the end result of the process.
*/
GmlPreProcessingResult preProcessGml( InputStream gmlInputStream );

/**

Imports GML data and merges the geospatial data updates into the database.

See {@link #fromGml(InputStream)} for details on the underlying process.
=== added file ‘dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 1970-01-01 00:00:00 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java 2015-06-05 12:56:34 +0000

@@ -0,0 +1,70 @@

+package org.hisp.dhis.dxf2.gml;

+import org.hisp.dhis.dxf2.metadata.MetaData;

+import java.io.InputStream;

+/**

Wraps the result of {@link GmlImportService#preProcessGml(InputStream)}.

This is necessary when performing GML import on a context where exceptions

due to malformed input cannot be caught, thus leaving the user uninformed of

the error. This class will wrap the failure and provide the Throwable to the

consuming class on error or the resulting MetaData object on success.

@author Halvdan Hoem grelland halvdanhg@gmail.com

*/

+public final class GmlPreProcessingResult

+{
private boolean isSuccess;

private MetaData resultMetaData;

private Throwable throwable;

private GmlPreProcessingResult(){}

public static GmlPreProcessingResult success( MetaData resultMetaData ) {
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setResultMetaData( resultMetaData );
   result.setSuccess( true );
   return result;
}

public static GmlPreProcessingResult failure( Throwable throwable )

{
   GmlPreProcessingResult result = new GmlPreProcessingResult();
   result.setThrowable( throwable );
   result.setSuccess( false );
   return result;
}

private void setSuccess( boolean isSuccess )

{
   this.isSuccess = isSuccess;
}

public boolean isSuccess()

{
   return isSuccess;
}

private void setResultMetaData( MetaData resultMetaData )

{
   this.resultMetaData = resultMetaData;
}

public MetaData getResultMetaData()

{
   return resultMetaData;
}

private void setThrowable( Throwable throwable )

{
   this.throwable = throwable;
}

public Throwable getThrowable()

{
   return throwable;
}
+}

=== modified file ‘dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java’

— dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-01-17 07:41:26 +0000

+++ dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java 2015-06-05 12:56:34 +0000

@@ -117,4 +117,6 @@
     assertEquals( 1, units.get( "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );

     assertEquals( 76, units.get( "Forskningsparken" ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );

 }
// TODO Add test for GmlImportService#preProcessGml(InputStream)

}

=== modified file ‘dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java’

— dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-05-30 13:36:07 +0000

+++ dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java 2015-06-05 12:56:34 +0000

@@ -77,7 +77,7 @@
 {

     taskExecutor.execute( task );

 }
 @Override

 public boolean scheduleTask( String key, Runnable task, String cronExpr )

 {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-05-28 16:10:07 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java 2015-06-05 12:56:34 +0000

@@ -187,7 +187,7 @@
     }

     else if ( "gml".equals( importFormat ) )

     {
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, importOptions, in, taskId ) );
       scheduler.executeTask( new ImportMetaDataGmlTask( userId, gmlImportService, notifier, importOptions, in, taskId ) );

   }

   else if ( "json".equals( importFormat ) || "xml".equals( importFormat ) )

   {
=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java’

— dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-04-11 14:06:51 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java 2015-06-05 12:56:34 +0000

@@ -28,14 +28,18 @@

SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

+import org.apache.commons.lang.exception.ExceptionUtils;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.hisp.dhis.dxf2.common.ImportOptions;

import org.hisp.dhis.dxf2.gml.GmlImportService;

+import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;

import org.hisp.dhis.scheduling.TaskId;

+import org.hisp.dhis.system.notification.NotificationLevel;

+import org.hisp.dhis.system.notification.Notifier;

+import org.springframework.web.util.HtmlUtils;

+import org.xml.sax.SAXParseException;

-import javax.xml.transform.TransformerException;

-import java.io.IOException;

import java.io.InputStream;

/**

@@ -50,12 +54,15 @@
 private String userUid;
 // -------------------------------------------------------------------------

 // Dependencies

 // -------------------------------------------------------------------------



 private GmlImportService gmlImportService;
private Notifier notifier;
 private ImportOptions importOptions;



 private InputStream inputStream;
@@ -64,11 +71,12 @@
 // Constructors

 // -------------------------------------------------------------------------
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService,
public ImportMetaDataGmlTask( String userUid, GmlImportService gmlImportService, Notifier notifier,
ImportOptions importOptions, InputStream inputStream, TaskId taskId )
{
this.userUid = userUid;

this.gmlImportService = gmlImportService;
   this.notifier = notifier;

   this.importOptions = importOptions;

   this.inputStream = inputStream;

   this.taskId = taskId;
@@ -83,15 +91,37 @@
 {

     importOptions.setImportStrategy( "update" ); // Force update only for GML import
   try
   {
       gmlImportService.importGml( inputStream, userUid, importOptions, taskId );
   }
   catch ( IOException | TransformerException e )
   {
       log.error( "Unable to read GML data from input stream", e );
       throw new RuntimeException( "Failed to parse GML input stream", e );
   }
   GmlPreProcessingResult gmlPreProcessingResult = gmlImportService.preProcessGml( inputStream );
   if ( !gmlPreProcessingResult.isSuccess() )
   {
       Throwable throwable = gmlPreProcessingResult.getThrowable();
       String message = createErrorMessage( throwable );
       notifier.notify( taskId, NotificationLevel.ERROR, message, false );
       log.error( "GML import failed: " + message, throwable );
       return;
   }
   gmlImportService.importGml( gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId );
}

private String createErrorMessage( Throwable throwable )

{
   String message = "";
   Throwable rootThrowable = ExceptionUtils.getRootCause( throwable );
   if ( rootThrowable instanceof SAXParseException )
   {
       SAXParseException e = (SAXParseException) rootThrowable;
       message += "Syntax error on line " + e.getLineNumber() + ". " + e.getMessage();
   }
   else
   {
       message += rootThrowable.getMessage();
   }
   return HtmlUtils.htmlEscape( message );
}
}

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties’

— dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-02-20 11:17:31 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties 2015-06-05 12:56:34 +0000

@@ -300,6 +300,7 @@

ignored=Ignored

conflicts=Conflicts

no_conflicts_found=No conflicts found

+no_import_summary_available=No summary available

type=Type

count=Count

export_as_xml=Export as XML

=== modified file ‘dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm’

— dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2013-03-21 09:15:24 +0000

+++ dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm 2015-06-05 12:56:34 +0000

@@ -1,4 +1,6 @@

$i18n.getString( "import_summary" )

+#if( $summary ) ## ImportSummary can be null on failed GML import (pre-processing fails)

$i18n.getString( "import_count" )

$summary.importCount.imported Imported

@@ -54,3 +56,6 @@

#else

$i18n.getString( "no_conflicts_found" )

#end

+#else

$i18n.getString( “no_summary_available” )

+#end

\ No newline at end of file

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

morten · 1 July 2015 09:16

Well, it reminded me of an earlier issue we had, but that have since been
fixed.

As I can import the sample XML file from Halvdan without any issues in 2.19
/ 2.20 (but he could not...) I might not be reproducing it properly, so I'm
waiting for input from him.

···

On Wed, Jul 1, 2015 at 4:12 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Having said that, this stack overflow business might carry some clue.
@Morten you mention elsewhere in this thread "It seems to be looping in
Attribute.toString()". Are you sure it is looping or is there something
recursive going on here? (note that xslt with very large nodesets could
indeed also cause recursion problems)

--
Morten