Thanks a lot.
···
On Fri, Oct 23, 2015 at 9:55 AM, Morten Olav Hansen mortenoh@gmail.com wrote:
Ok, if you are still having issues (and can share the org units) feel free to email me the file and I will have a look.
–
–
Morten
On Fri, Oct 23, 2015 at 3:53 PM, Nicolas Germeau ngermeau@bluesquare.org wrote:
Hello Morten,
Thanks for the advice. For what I know the server looks big enough (4gb mem). I might try to scale it to 8gb change some vm parameters and see how it works. But knowing that I have the backup solution to split it in level is always good.
Regards,
Nicolas
On Fri, Oct 23, 2015 at 3:01 AM, Morten Olav Hansen mortenoh@gmail.com wrote:
It’s also fine splitting it up in many small pieces. As you say, you must ensure that the parents exists, so doing it one level at the time is the approach. If not, you must do import multiple times, so it can update the parents (not really a proper solution).
Does the server have sufficient memory? because 37k OUs (8mb) should not really be a problem… I have imported 50-60mb on my local machine before without any issues (although that was more than just OUs)
–
Nicolas Germeau
Software Developer
GSM: +32 487 129820
Tel: +32 2 737 65 16
Skype: nicolas.germeau
Follow us on Twitter
Join us on Facebook
Visit our new website
–
Morten
On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau ngermeau@bluesquare.org wrote:
Jason,
I guess I’m going on the XSLT solution and split it per level, this is the fastest line for me.
Thanks a lot for the advices. Enjoy you mushrooms wanderings.
See you
Nicolas
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering jason.p.pickering@gmail.com wrote:
Hi Nicolas
Not sure there is a tool per se, but there are approaches. Not sure where your data is coming from, but if you are transforming to JSON/XML, it may be just as easy to transform to SQL insert statements.
The trick is to do it level by level and be sure you get the parent ID correct. Otherwise, set the parent level and be sure you reset the hibernate sequence of the database to avoid collisions with future inserts.
Otherwise, maybe try what you have but in batches of a few thousand at a time since you seem to have it already in an importable format.
Thanks. I need to do something other than dhis2 and looking for mushrooms helps to clear the mind.
Regards,
Jason
On Thu, Oct 22, 2015, 18:57 Nicolas Germeau ngermeau@bluesquare.org wrote:
Hello Jason,
Nope, no stacktrace. I suppose the transaction is too big and blow the whole thing up. Yep sure, I don’t really care about the method as long as it doesn’t take to much time to setup. I’m thinking about creating a quick XSLT to extract org unit per level but if you have any tool/idea which allow direct SQL I’m in.
Nice mushroom picture by the way.
Regards.
Nicolas
–
Nicolas Germeau
Software Developer
GSM: +32 487 129820
Tel: +32 2 737 65 16
Skype: nicolas.germeau
Follow us on Twitter
Join us on Facebook
Visit our new website
On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering jason.p.pickering@gmail.com wrote:
Hi Nicolas,
Hard to say without a stack trace. It just bombs with no explanation?
Could you consider using direct SQL injection for such a large number of orgunits?
Regards,
Jason
On Thu, Oct 22, 2015, 18:14 Nicolas Germeau ngermeau@bluesquare.org wrote:
Hello,
I’m trying to make a rather large import of orgunit. The size of the file is around 8megs zipped and it contains more than 37000 orgunits.
When I’m launching the import it takes a little bit more than an hour. I can see in the log file (dhis.log) outputs telling me that org unit are being imported. At the end I’m receiving the following message in catalina.out
- INFO 2015-10-22 15:26:50,878 [Level: INFO, category: METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done. Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
and then the server crash without any message on what when wrong. Nothing is imported.
I’m wondering how the import works, can I split the file arbitrary in multiple smaller parts (will it handle the fk parent id correctly) ? Or do I have to split the file per level to ensure parent finding ?
Thanks
Regards
Nicolas
–
Nicolas Germeau
Software Developer
GSM: +32 487 129820
Tel: +32 2 737 65 16
Skype: nicolas.germeau
Follow us on Twitter
Join us on Facebook
Visit our new website
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
–
Nicolas Germeau
Software Developer
GSM: +32 487 129820
Tel: +32 2 737 65 16
Skype: nicolas.germeau
Follow us on Twitter
Join us on Facebook
Visit our new website
Nicolas Germeau
Software Developer
GSM: +32 487 129820
Tel: +32 2 737 65 16
Skype: nicolas.germeau
Follow us on Twitter
Join us on Facebook
Visit our new website