Halvdan,
Firstly, I have just done a test import of the za_CTRY.gml file using the latest 2.20 trunk - and IT WORKS. I just hope using 2.20 for importing those files and then reverting back to our customised 2.19 won’t mess up anything (NOTE: for the time being, we have to use a customised version of 2.19 due to the inclusion of 1.4 conversion code, the new database Synchronisation Manager enabling synching between many different DHIS2 instances, and the small piece of code enabling vertical cursor movement when capturing daily data in the custom “pivot” data entry form).
Secondly, I do get some warning messages when importing - not sure if they are relevant/expected (see tomcat log extract below).
I really appreciate the effort you’ve put in on this, especially during those hot mid-summer days in Oslo 
Regards
Calle
Tomcat log:
04-Jul-2015 01:33:26.270 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 78200 ms
- WARN 2015-07-04 01:33:39,997 Authentication event AuthenticationSuccessEvent: Calle_Hedberg; details: org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8: RemoteIpAddress: 0
0-exec-5])
- WARN 2015-07-04 01:33:39,998 Authentication event SessionFixationProtectionEvent: Calle_Hedberg; details: org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8: RemoteIpAddres
-8080-exec-5])
- WARN 2015-07-04 01:33:39,999 Authentication event InteractiveAuthenticationSuccessEvent: Calle_Hedberg; details: org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8: RemoteI
ttp-nio-8080-exec-5])
Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property ‘http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit’ is not recognized.
Compiler warnings:
WARNING: ‘org.apache.xerces.jaxp.SAXParserImpl: Property ‘http://javax.xml.XMLConstants/property/accessExternalDTD’ is not recognized.’
Warning: org.apache.xerces.parsers.SAXParser: Feature ‘http://javax.xml.XMLConstants/feature/secure-processing’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://javax.xml.XMLConstants/property/accessExternalDTD’ is not recognized.
Warning: org.apache.xerces.parsers.SAXParser: Property ‘http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit’ is not recognized.
-
INFO 2015-07-04 01:34:39,988 User ‘Calle_Hedberg’ started import at Sat Jul 04 01:34:39 CAT 2015 (DefaultImportService.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:39,988 [Level: INFO, category: METADATA_IMPORT, time: Sat Jul 04 01:34:39 CAT 2015, message: Importing meta-data] (InMemoryNotifier.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:39,998 Building object-bridge maps (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,235 Building object-bridge maps took 3.24 seconds. (DefaultObjectBridge.java [taskScheduler-1])
-
WARN 2015-07-04 01:34:43,235 Can not find getter for ‘oAuth2Clients’. (DefaultImportService.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,235 [Level: INFO, category: METADATA_IMPORT, time: Sat Jul 04 01:34:43 CAT 2015, message: Importing 1 OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,245 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,245 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,245 Deleted objects associated with object of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
-
WARN 2015-07-04 01:34:43,265 Can not find getter for ‘programRuleVariables’. (DefaultImportService.java [taskScheduler-1])
-
WARN 2015-07-04 01:34:43,265 Can not find getter for ‘programRules’. (DefaultImportService.java [taskScheduler-1])
-
WARN 2015-07-04 01:34:43,265 Can not find getter for ‘programRuleActions’. (DefaultImportService.java [taskScheduler-1])
-
INFO 2015-07-04 01:34:43,275 [Level: INFO, category: METADATA_IMPORT, time: Sat Jul 04 01:34:43 CAT 2015, message: Import done. Completed in 3.28 seconds.] (InMemoryNotifier.java [taskScheduler-1])
···
On 4 July 2015 at 01:12, Halvdan Grelland halvdanhg@gmail.com wrote:
Oh, and for the record: the stack overflow was caused by cirular calls to toString between AttributeValue and Attribute, respectively.
–
2015-07-04 0:56 GMT+02:00 Halvdan Grelland halvdanhg@gmail.com:
Yes, that is the plan, ultimately.
It’s a pretty major change though, making it non-trivial to port the fix (it’s more of a refactor). I’ll look into it after the weekend (or during the weekend if the sun stops shining in Oslo).
I did make (and backport) some changes which fix the stack overflow issue, however. In practice this means that GML import should work for you on the latest 2.19, but any attribute values will unfortunately be deleted in the process (!), or you might even experience the importer halting if there are attribute values for the orgunits. Not ideal, in other words, and something you should be aware of.
Halvdan
2015-07-03 18:55 GMT+02:00 Calle Hedberg calle.hedberg@gmail.com:
Halvdan,
I see you have committed a significant fix to 2.20 trunk - which I hope will also fix my GML import issue.
Will you port this fix to 2.19 also, please?
Regards
Calle
On 1 July 2015 at 15:01, Halvdan Grelland halvdanhg@gmail.com wrote:
Ok, so I’ve looked further into this. The only issue I am able to consistently reproduce happens only in the GML import process. The stack trace can be found here
Note that this might be unrelated to the looping/recursion issue.,
This does not happen if I pre-stage the meta data payload (ie write it to a file and import using the metadata importer). My suspicion that the conflict occurs because of the clear/re-add of attributevalues in the orgunit.mergeWith() method (which I’m using to merge geodata updates into the orgunit), which happens within a transactional scope. I’m gonna have to experiment further to find a solution. Any suggestions are much appreciated.
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19274
Email: calle.hedberg@gmail.com
Skype: calle_hedberg
2015-07-01 14:39 GMT+02:00 Bob Jolliffe bobjolliffe@gmail.com:
Morten this is a long shot, but do you think there is any possibility
the removal of xerces exclusions in 2.19 might be causing these
strange unpredictable problems? Weird StackOverflowErrors and Xerces
problems seem to go hand in hand - google the two terms 
I guess
the nub is that it becomes unpredictable which xercesimpl the
classloader uses.
The original reason for adding the xercesImpl exclusions was that it
seemed to make the java xml parsing more stable. I am not sure how it
easy it would be test this as you’d have to build without openid4java
… which is the library which caused us to remove the exclusions.
On 1 July 2015 at 10:16, Morten Olav Hansen mortenoh@gmail.com wrote:
On Wed, Jul 1, 2015 at 4:12 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:
Having said that, this stack overflow business might carry some clue.
@Morten you mention elsewhere in this thread "It seems to be looping in
Attribute.toString()". Are you sure it is looping or is there something
recursive going on here? (note that xslt with very large nodesets could
indeed also cause recursion problems)
Well, it reminded me of an earlier issue we had, but that have since been
fixed.
As I can import the sample XML file from Halvdan without any issues in 2.19
/ 2.20 (but he could not…) I might not be reproducing it properly, so I’m
waiting for input from him.
–
Morten
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19274
Email: calle.hedberg@gmail.com
Skype: calle_hedberg