Failure with Analytics tables update

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !

image

···

Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

Hi

I think you need to have a look at your server logs, there will be full stack trace etc which we can use in debugging your issues

···

On Fri, Jun 17, 2016 at 1:45 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !


Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi Morten,

Please find attached an extraction of the log file containing the process of the table analytics update. This kind of error is always repeating “Caused by: org.postgresql.util.PSQLException: ERROR: could not extend file “base/16387/820574”: No space left on device”. The memory allocated to Tomcat and postgreSQL are already tuned according to HISP recommendations. What are the other parameters that should be corrected in order to troubleshoot those issues?

Thank you for your support.

20160617_DHIS2_JDBC_ERROR.txt (493 KB)

···

Tantely Raminosoa
Data Officer
Management Sciences for Health
Antananarivo Madagascar
Mobile: 0321180040
E-mail: traminosoa@mikolo.org
Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

On Fri, Jun 17, 2016 at 10:32 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi

I think you need to have a look at your server logs, there will be full stack trace etc which we can use in debugging your issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Jun 17, 2016 at 1:45 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !


Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Oh, this have nothing to do with DHIS 2, it means you harddrive is out of space… maybe you should have a look at your tomcat/logs directory, see if its full (we have had some issues with logs filling up drives), if not, you need to somehow free some disk space on your drive.

···

On Fri, Jun 17, 2016 at 4:03 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi Morten,

Please find attached an extraction of the log file containing the process of the table analytics update. This kind of error is always repeating “Caused by: org.postgresql.util.PSQLException: ERROR: could not extend file “base/16387/820574”: No space left on device”. The memory allocated to Tomcat and postgreSQL are already tuned according to HISP recommendations. What are the other parameters that should be corrected in order to troubleshoot those issues?

Thank you for your support.


Tantely Raminosoa
Data Officer
Management Sciences for Health
Antananarivo Madagascar
Mobile: 0321180040
E-mail: traminosoa@mikolo.org
Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Jun 17, 2016 at 10:32 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi

I think you need to have a look at your server logs, there will be full stack trace etc which we can use in debugging your issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Jun 17, 2016 at 1:45 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !


Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Raminosoa,

it seems you are running out of disk space on your computer.

regards,

Lars

	em.util.ConcurrentUtils.waitForCompletion(ConcurrentUtils.java:51) ... 13 more Caused by: org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [create index "in_sixmonthlyapril_ax_2016_RpwHm" on analytics_temp_2016 ("sixmonthlyapril")]; ERROR: could not write block 3842 of temporary file: No space left on device Hint: Perhaps out of disk space?; nested exception is org.postgresql.util.PSQLException: ERROR: could not write block 3842 of temporary file: No space left on device Hint: Perhaps out of disk space?

···

On Fri, Jun 17, 2016 at 11:03 AM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi Morten,

Please find attached an extraction of the log file containing the process of the table analytics update. This kind of error is always repeating “Caused by: org.postgresql.util.PSQLException: ERROR: could not extend file “base/16387/820574”: No space left on device”. The memory allocated to Tomcat and postgreSQL are already tuned according to HISP recommendations. What are the other parameters that should be corrected in order to troubleshoot those issues?

Thank you for your support.


Tantely Raminosoa
Data Officer
Management Sciences for Health
Antananarivo Madagascar
Mobile: 0321180040
E-mail: traminosoa@mikolo.org
Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Jun 17, 2016 at 10:32 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi

I think you need to have a look at your server logs, there will be full stack trace etc which we can use in debugging your issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Jun 17, 2016 at 1:45 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !


Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi Raminosoa,

check your disk space by this command if u are on linux: df -h

That happens to me a lot of time when i do work on smaller Virtual Machines.

regards,

···

On Fri, Jun 17, 2016 at 11:13 AM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Raminosoa,

it seems you are running out of disk space on your computer.

regards,

Lars

	em.util.ConcurrentUtils.waitForCompletion(ConcurrentUtils.java:51) ... 13 more Caused by: org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [create index "in_sixmonthlyapril_ax_2016_RpwHm" on analytics_temp_2016 ("sixmonthlyapril")]; ERROR: could not write block 3842 of temporary file: No space left on device Hint: Perhaps out of disk space?; nested exception is org.postgresql.util.PSQLException: ERROR: could not write block 3842 of temporary file: No space left on device Hint: Perhaps out of disk space?

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Fri, Jun 17, 2016 at 11:03 AM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi Morten,

Please find attached an extraction of the log file containing the process of the table analytics update. This kind of error is always repeating “Caused by: org.postgresql.util.PSQLException: ERROR: could not extend file “base/16387/820574”: No space left on device”. The memory allocated to Tomcat and postgreSQL are already tuned according to HISP recommendations. What are the other parameters that should be corrected in order to troubleshoot those issues?

Thank you for your support.


Tantely Raminosoa
Data Officer
Management Sciences for Health
Antananarivo Madagascar
Mobile: 0321180040
E-mail: traminosoa@mikolo.org
Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Fri, Jun 17, 2016 at 10:32 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi

I think you need to have a look at your server logs, there will be full stack trace etc which we can use in debugging your issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Jun 17, 2016 at 1:45 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hello,

Does anyone have some idea on how to troubleshoot the table analytics failure process as indicated by the image below? The process starts fine but it is stuck after a while. Then, the data called from the crosstab visualization could not be updated according to data entered in the database. Before, it has been already working fine and scheduled for every day.

Using DHIS2 2.19.

Thank you !


Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi All,

Thank you for your points of view. I am just wondering on this disk capacity issue. We are using a cloud Amazon Web services to host the DHIS2 application. There are 80 gigabyte disk space used only by DHIS2. There are 30 gigabyte of RAM where 9 GB are for JVM and 9 GB for postgreSQL. My question is that which parameters should I correct to troubleshoot this disk issue as physically there is still large available disk space on the server, or is there other parameters to allocate the physical disk space (in addition to the RAM) managed by JVM?

Thank you.

Tantely.

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.

This is an issue with Postgres specifically and is therefore not related to the JVM. Most likely it is failing to temporarily store data during a big query due to low capacity.

If you have plentiful physical capacity on the disk you could have a look at https://www.postgresql.org/docs/9.5/static/runtime-config-resource.html

Specifically see section 18.4.2 which addresses the temp_file_limit.

Also try to do a VACUUM FULL

···

On Fri, Jun 17, 2016 at 3:26 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi All,

Thank you for your points of view. I am just wondering on this disk capacity issue. We are using a cloud Amazon Web services to host the DHIS2 application. There are 80 gigabyte disk space used only by DHIS2. There are 30 gigabyte of RAM where 9 GB are for JVM and 9 GB for postgreSQL. My question is that which parameters should I correct to troubleshoot this disk issue as physically there is still large available disk space on the server, or is there other parameters to allocate the physical disk space (in addition to the RAM) managed by JVM?

Thank you.

Tantely.

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi There,

We have experienced similar issues earlier where we were having plenty of disk space but most of it was not mounted. Could you please check the mount points and then check the disk space available for postgres and other stuff. You can use the command ‘df’ and understand how is the disk organized.

Regards,

Sultan Ahamar.

···

On Fri, Jun 17, 2016 at 7:13 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This is an issue with Postgres specifically and is therefore not related to the JVM. Most likely it is failing to temporarily store data during a big query due to low capacity.

If you have plentiful physical capacity on the disk you could have a look at https://www.postgresql.org/docs/9.5/static/runtime-config-resource.html

Specifically see section 18.4.2 which addresses the temp_file_limit.

Also try to do a VACUUM FULL


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Fri, Jun 17, 2016 at 3:26 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi All,

Thank you for your points of view. I am just wondering on this disk capacity issue. We are using a cloud Amazon Web services to host the DHIS2 application. There are 80 gigabyte disk space used only by DHIS2. There are 30 gigabyte of RAM where 9 GB are for JVM and 9 GB for postgreSQL. My question is that which parameters should I correct to troubleshoot this disk issue as physically there is still large available disk space on the server, or is there other parameters to allocate the physical disk space (in addition to the RAM) managed by JVM?

Thank you.

Tantely.

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi All,

Thank you for your suggestions. I am on the way to verify the server and I will be back with informations.

Thank you.

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.

···

On Mon, Jun 20, 2016 at 6:56 AM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:

Hi There,

We have experienced similar issues earlier where we were having plenty of disk space but most of it was not mounted. Could you please check the mount points and then check the disk space available for postgres and other stuff. You can use the command ‘df’ and understand how is the disk organized.

Regards,

Sultan Ahamar.

On Fri, Jun 17, 2016 at 7:13 PM, Halvdan Hoem Grelland halvdan@dhis2.org wrote:

This is an issue with Postgres specifically and is therefore not related to the JVM. Most likely it is failing to temporarily store data during a big query due to low capacity.

If you have plentiful physical capacity on the disk you could have a look at https://www.postgresql.org/docs/9.5/static/runtime-config-resource.html

Specifically see section 18.4.2 which addresses the temp_file_limit.

Also try to do a VACUUM FULL


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Fri, Jun 17, 2016 at 3:26 PM, Raminosoa Rabemanantsoa, Tantely traminosoa@mikolo.org wrote:

Hi All,

Thank you for your points of view. I am just wondering on this disk capacity issue. We are using a cloud Amazon Web services to host the DHIS2 application. There are 80 gigabyte disk space used only by DHIS2. There are 30 gigabyte of RAM where 9 GB are for JVM and 9 GB for postgreSQL. My question is that which parameters should I correct to troubleshoot this disk issue as physically there is still large available disk space on the server, or is there other parameters to allocate the physical disk space (in addition to the RAM) managed by JVM?

Thank you.

Tantely.

This message and its attachments are confidential and solely for the intended recipients. If received in error, please delete them and notify the sender via reply e-mail immediately.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Halvdan Hoem Grelland

Software developer, DHIS 2

University of Oslo

http://www.dhis2.org

Tantely Raminosoa

Data Officer
Management Sciences for Health
Antananarivo Madagascar

Mobile: 0321180040

E-mail: traminosoa@mikolo.org

Skype: traminosoa

Stronger health systems. Greater health impact.

www.msh.org