I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
Moses
···
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Hi Moses,
No, it is not normal. We regularly import data around this size, and it only takes a few minutes, on a much smaller virtual machine than what you mention.
What type of importer are you using? XML or CSV?
Regards,
Jason
···
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Thanks Jason. I have tried both options. XML, and CSV. Same result. I have also tried running data integrity check and it took forever. Never completed the check. Could there be a connection?
···
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Sounds to me like something is hogging your system resources. How much memory have you allocated to both Tomcat and Postgres? How many rows of data do you have in your datavalue table? Which database are you using? What do the logs say? What is you current CPU usage before you start the import?
···
On Fri, Apr 4, 2014 at 10:52 AM, Moses Marimo marimo@ict.co.zw wrote:
Thanks Jason. I have tried both options. XML, and CSV. Same result. I have also tried running data integrity check and it took forever. Never completed the check. Could there be a connection?
Hi Moses,
No, it is not normal. We regularly import data around this size, and it only takes a few minutes, on a much smaller virtual machine than what you mention.
What type of importer are you using? XML or CSV?
Regards,
Jason
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Thanks Jason. Problem Solved. “How much memory have you allocated to both Tomcat and Postgres” Well, this jogged my memory. Tomcat+Postgres memory was more than available RAM. So I guess one of the two was boxing out the other. I reduced the size of both memories so that there was at least 2GB Free for the OS and it did the trick.
Sounds to me like something is hogging your system resources. How much memory have you allocated to both Tomcat and Postgres? How many rows of data do you have in your datavalue table? Which database are you using? What do the logs say? What is you current CPU usage before you start the import?
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
7 Chelmsford Belgravia, Harare
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
On Fri, Apr 4, 2014 at 10:52 AM, Moses Marimo marimo@ict.co.zw wrote:
Thanks Jason. I have tried both options. XML, and CSV. Same result. I have also tried running data integrity check and it took forever. Never completed the check. Could there be a connection?
Hi Moses,
No, it is not normal. We regularly import data around this size, and it only takes a few minutes, on a much smaller virtual machine than what you mention.
What type of importer are you using? XML or CSV?
Regards,
Jason
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
2G seems like a lot for the OS, but agree that over allocating (and the subsequent swapping and thrashing) would certainly be harmful. Probably 1G for the OS would be sufficient.
Thanks Jason. Problem Solved. “How much memory have you allocated to both Tomcat and Postgres” Well, this jogged my memory. Tomcat+Postgres memory was more than available RAM. So I guess one of the two was boxing out the other. I reduced the size of both memories so that there was at least 2GB Free for the OS and it did the trick.
Sounds to me like something is hogging your system resources. How much memory have you allocated to both Tomcat and Postgres? How many rows of data do you have in your datavalue table? Which database are you using? What do the logs say? What is you current CPU usage before you start the import?
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
7 Chelmsford Belgravia, Harare
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
On Fri, Apr 4, 2014 at 10:52 AM, Moses Marimo marimo@ict.co.zw wrote:
Thanks Jason. I have tried both options. XML, and CSV. Same result. I have also tried running data integrity check and it took forever. Never completed the check. Could there be a connection?
Hi Moses,
No, it is not normal. We regularly import data around this size, and it only takes a few minutes, on a much smaller virtual machine than what you mention.
What type of importer are you using? XML or CSV?
Regards,
Jason
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Ok. Thanks Bob. One other thing. Jason asked me the number of records in my datavalue table. I just checked, its now over 3.6 million records. Is there a size that would start to cause performance issues and if yes, what is that size.
2G seems like a lot for the OS, but agree that over allocating (and the subsequent swapping and thrashing) would certainly be harmful. Probably 1G for the OS would be sufficient.
Bob
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
7 Chelmsford Belgravia, Harare
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
Thanks Jason. Problem Solved. “How much memory have you allocated to both Tomcat and Postgres” Well, this jogged my memory. Tomcat+Postgres memory was more than available RAM. So I guess one of the two was boxing out the other. I reduced the size of both memories so that there was at least 2GB Free for the OS and it did the trick.
Sounds to me like something is hogging your system resources. How much memory have you allocated to both Tomcat and Postgres? How many rows of data do you have in your datavalue table? Which database are you using? What do the logs say? What is you current CPU usage before you start the import?
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
7 Chelmsford Belgravia, Harare
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.
On Fri, Apr 4, 2014 at 10:52 AM, Moses Marimo marimo@ict.co.zw wrote:
Thanks Jason. I have tried both options. XML, and CSV. Same result. I have also tried running data integrity check and it took forever. Never completed the check. Could there be a connection?
Hi Moses,
No, it is not normal. We regularly import data around this size, and it only takes a few minutes, on a much smaller virtual machine than what you mention.
What type of importer are you using? XML or CSV?
Regards,
Jason
On Fri, Apr 4, 2014 at 9:41 AM, Moses Marimo marimo@ict.co.zw wrote:
Dear Colleagues,
I am trying to benchmark the time it takes for my users to import data into DHIS2.14, Build 13793. I know this import time also depends on the resources available on my server, BUT a little comparison would not hurt. My server has a Xeon Processor Quad-Core and the RAM is 12GB.
On average, compressed import files of 764KB (uncompressing this file gives about 15MB) in size take as much as 2-3 hours to import. Is this normal? What are the experience of others out there with the time it takes to import files. 3hrs just appears too much and I may be missing something.
Regards,
–
ICT Warehouse (Pvt) Ltd - “The Stockist of e-nnovation”
Innovative solutions in Web Design and E-Commerce, Software Engineering, Mobile and Bulk SMS Solutions, Graphic Design and Printing, General ICT Solutions and Consultancy.