need solution for running out of disk space generating analytics tables

Hi

I am experimenting with a non-production instance of dhis2 (version 2.15).

After I imported data from a production instance (using web api), the postgres user’s home directory takes 7.5 GB

I have 99GB of available disk space

When I try to generate the analytics tables, I run out of disk space (does this make any sense?)

My dhis2 instance has some historical data going back 10+ years.

Perhaps if analytics tables were generated for only the last couple of years, it would fit in the available disk space?

  1. I don’t mind throwing away data before 2010. Is there an easy way to purge all older data?

  2. Is there a way to tell dhis2 to generate analytics data for only 2010 and onward?

Best regards,

Lavi

Hi Lavi,

interesting - that sounds like a lot. Approximately how many records do you have in datavalue table? How many dimensions do you see under /api/dimensions ? Normally the datavalue - analytics size ratio should be less.

You can have more control over analytics through the web api. Docs here. By using curl you can issue a command like this (replace 4 with no of years):

curl “http:///api/resourceTables/analytics?lastYears=4” -X POST -u user:passwrd -v

Curl comes preinstalled on linux and mac and can be installed through gow on win.

This should remove the need to purge the historical data. If you want to do that still, you can use an SQL command like:

delete from datavalue dv using period p where dv.periodid=p.periodid and p.startdate <= ‘2009-12-31’;

best regards,

Lars

···

On Wed, Jul 23, 2014 at 8:43 AM, Lavi Shpigelman shpigi@gmail.com wrote:

Hi

I am experimenting with a non-production instance of dhis2 (version 2.15).

After I imported data from a production instance (using web api), the postgres user’s home directory takes 7.5 GB

I have 99GB of available disk space

When I try to generate the analytics tables, I run out of disk space (does this make any sense?)

My dhis2 instance has some historical data going back 10+ years.

Perhaps if analytics tables were generated for only the last couple of years, it would fit in the available disk space?

  1. I don’t mind throwing away data before 2010. Is there an easy way to purge all older data?
  1. Is there a way to tell dhis2 to generate analytics data for only 2010 and onward?

Best regards,

Lavi


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Thanks !

Running analytics generation for last 4 years was enough to avoid disk space limit.

The postgresql DB directory grew from 7.5GB to 44GB

The datavalue table holds 53M records.
The dimensions resource lists 13 dimensions.

best regards,

Lavi

Hi Lavi,

Thanks !

Running analytics generation for last 4 years was enough to avoid disk
space limit.

that's great.

The postgresql DB directory grew from 7.5GB to 44GB

The datavalue table holds 53M records.
The dimensions resource lists 13 dimensions.

Okay. That's a fair number of dimensions. I guess the only option is
getting a bigger disk;)

One could omit certain dimensions to become columns and indexes in
analytics tables (org unit group sets, data element group sets, categories)
by setting "data dimension" to no through the UI, but that would of course
defeat the purpose of setting up those dimensions in the first place.

regards,

Lars

···

On Thu, Jul 24, 2014 at 9:22 AM, Lavi Shpigelman <shpigi@gmail.com> wrote: