it seems like the analytics table update does not remove data/tables of years,
that are no longer valid.
Since dataValues can be deleted via api in batch, it can happen that dataValues
for a whole year get deleted. Of course these years should not show up in the
analytics (pivot, chart etc...) any more. However, those obsolete tables seem to
remain in the database after running the analytics update. Of course they could
be remove manually, but this is not applicable in a nightly load scenario.
I did empty the cache and checked catalina.out, but I can't find an error. In
the database the dataValues table remains with only the years, that were not
deleted, so the deletion obviously worked. But the obsolete analytics tables
(e.g. analytics_2012) are still there and populated. You can verify in the
attached excerpt from catalina.out: only years >= 2016 were (correctly) updated,
deleted years <= 2015 remained untouched.
This is quite serious as it results into wrong statistics, if not detected. I
just discovered this because of a major clean-up activity - if there are only
very few years affected, users might not even notice that they are reporting
deleted data.
Currently running on 2.22, build 21765, Standard Ubuntu 14.04 & OracleJava 8.
Hi, I can only support/confirm what Uwe is saying here - I just experieced the same thing this morning: deleted data for one year, ran analytics, and the deleted data is still displayed.
it seems like the analytics table update does not remove data/tables of years,
that are no longer valid.
Since dataValues can be deleted via api in batch, it can happen that dataValues
for a whole year get deleted. Of course these years should not show up in the
analytics (pivot, chart etc…) any more. However, those obsolete tables seem to
remain in the database after running the analytics update. Of course they could
be remove manually, but this is not applicable in a nightly load scenario.
I did empty the cache and checked catalina.out, but I can’t find an error. In
the database the dataValues table remains with only the years, that were not
deleted, so the deletion obviously worked. But the obsolete analytics tables
(e.g. analytics_2012) are still there and populated. You can verify in the
attached excerpt from catalina.out: only years >= 2016 were (correctly) updated,
deleted years <= 2015 remained untouched.
This is quite serious as it results into wrong statistics, if not detected. I
just discovered this because of a major clean-up activity - if there are only
very few years affected, users might not even notice that they are reporting
deleted data.
Currently running on 2.22, build 21765, Standard Ubuntu 14.04 & OracleJava 8.
Short-term workaround is to go to data admin > maintenance and click Clear analytics tables.
regards,
Lars
···
On Thu, Feb 18, 2016 at 3:32 AM, Olav Poppe olav.poppe@me.com wrote:
Hi, I can only support/confirm what Uwe is saying here - I just experieced the same thing this morning: deleted data for one year, ran analytics, and the deleted data is still displayed.
it seems like the analytics table update does not remove data/tables of years,
that are no longer valid.
Since dataValues can be deleted via api in batch, it can happen that dataValues
for a whole year get deleted. Of course these years should not show up in the
analytics (pivot, chart etc…) any more. However, those obsolete tables seem to
remain in the database after running the analytics update. Of course they could
be remove manually, but this is not applicable in a nightly load scenario.
I did empty the cache and checked catalina.out, but I can’t find an error. In
the database the dataValues table remains with only the years, that were not
deleted, so the deletion obviously worked. But the obsolete analytics tables
(e.g. analytics_2012) are still there and populated. You can verify in the
attached excerpt from catalina.out: only years >= 2016 were (correctly) updated,
deleted years <= 2015 remained untouched.
This is quite serious as it results into wrong statistics, if not detected. I
just discovered this because of a major clean-up activity - if there are only
very few years affected, users might not even notice that they are reporting
deleted data.
Currently running on 2.22, build 21765, Standard Ubuntu 14.04 & OracleJava 8.
thanks for taking this over. If there was an api for clearing the analytics
table, which is giving back a result (unlike the one for updating the analytics
tables), that would already be a sufficient solution for nightly runs. Then I
could always delete the analytics tables before generating them afresh. Whatever
is easier for you to implement ...
You have interesting working hours
Regards, Uwe
···
Lars Helge Øverland <lars@dhis2.org> hat am 19. Februar 2016 um 04:38
geschrieben:
Short-term workaround is to go to data admin > maintenance and click Clear
analytics tables.
regards,
Lars
On Thu, Feb 18, 2016 at 3:32 AM, Olav Poppe <olav.poppe@me.com> wrote:
> Hi, I can only support/confirm what Uwe is saying here - I just experieced
> the same thing this morning: deleted data for one year, ran analytics, and
> the deleted data is still displayed.
>
> Olav
>
>
>
> 18. feb. 2016 kl. 07.38 skrev Uwe Wahser <uwe@wahser.de>:
>
> Dear devs,
>
> it seems like the analytics table update does not remove data/tables of
> years,
> that are no longer valid.
>
> Since dataValues can be deleted via api in batch, it can happen that
> dataValues
> for a whole year get deleted. Of course these years should not show up in
> the
> analytics (pivot, chart etc...) any more. However, those obsolete tables
> seem to
> remain in the database after running the analytics update. Of course they
> could
> be remove manually, but this is not applicable in a nightly load scenario.
>
> I did empty the cache and checked catalina.out, but I can't find an error.
> In
> the database the dataValues table remains with only the years, that were
> not
> deleted, so the deletion obviously worked. But the obsolete analytics
> tables
> (e.g. analytics_2012) are still there and populated. You can verify in the
> attached excerpt from catalina.out: only years >= 2016 were (correctly)
> updated,
> deleted years <= 2015 remained untouched.
>
> This is quite serious as it results into wrong statistics, if not
> detected. I
> just discovered this because of a major clean-up activity - if there are
> only
> very few years affected, users might not even notice that they are
> reporting
> deleted data.
>
> Currently running on 2.22, build 21765, Standard Ubuntu 14.04 & OracleJava
> 8.
>
> Thanks for checking on that,
>
> Uwe<catalina.out.2016-02-18>
> _______________________________________________
> Mailing list: DHIS 2 developers in Launchpad
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : DHIS 2 developers in Launchpad
> More help : ListHelp - Launchpad Help
>
>
>
> _______________________________________________
> Mailing list: DHIS 2 developers in Launchpad
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : DHIS 2 developers in Launchpad
> More help : ListHelp - Launchpad Help
>
>
We have an API for clearing analytics cache [1], I’m sure we can make one for clearing analytics tables also. Would you mind creating a blueprint for us? and give me the link, and I will assign the proper people.
@Olav P: would this be an applicable solution for your problem, too?
Regards, Uwe
···
---
Morten Olav Hansen <morten@dhis2.org> hat am 19. Februar 2016 um 14:05
geschrieben:
Hi Uwe
We have an API for clearing analytics cache [1], I'm sure we can make one
for clearing analytics tables also. Would you mind creating a blueprint for
us? and give me the link, and I will assign the proper people.
On Fri, Feb 19, 2016 at 4:25 PM, Uwe Wahser <uwe@wahser.de> wrote:
> Hi Lars,
>
> thanks for taking this over. If there was an api for clearing the analytics
> table, which is giving back a result (unlike the one for updating the
> analytics
> tables), that would already be a sufficient solution for nightly runs.
> Then I
> could always delete the analytics tables before generating them afresh.
> Whatever
> is easier for you to implement ...
>
> You have interesting working hours
>
> Regards, Uwe
>
>
> > Lars Helge Øverland <lars@dhis2.org> hat am 19. Februar 2016 um 04:38
> > geschrieben:
> >
> >
> > Hi Uwe,
> >
> > will work on this.
> >
> > https://bugs.launchpad.net/dhis2/+bug/1408096
> >
> > Short-term workaround is to go to data admin > maintenance and click
> Clear
> > analytics tables.
> >
> > regards,
> >
> > Lars
> >
> >
> > On Thu, Feb 18, 2016 at 3:32 AM, Olav Poppe <olav.poppe@me.com> wrote:
> >
> > > Hi, I can only support/confirm what Uwe is saying here - I just
> experieced
> > > the same thing this morning: deleted data for one year, ran analytics,
> and
> > > the deleted data is still displayed.
> > >
> > > Olav
> > >
> > >
> > >
> > > 18. feb. 2016 kl. 07.38 skrev Uwe Wahser <uwe@wahser.de>:
> > >
> > > Dear devs,
> > >
> > > it seems like the analytics table update does not remove data/tables of
> > > years,
> > > that are no longer valid.
> > >
> > > Since dataValues can be deleted via api in batch, it can happen that
> > > dataValues
> > > for a whole year get deleted. Of course these years should not show up
> in
> > > the
> > > analytics (pivot, chart etc...) any more. However, those obsolete
> tables
> > > seem to
> > > remain in the database after running the analytics update. Of course
> they
> > > could
> > > be remove manually, but this is not applicable in a nightly load
> scenario.
> > >
> > > I did empty the cache and checked catalina.out, but I can't find an
> error.
> > > In
> > > the database the dataValues table remains with only the years, that
> were
> > > not
> > > deleted, so the deletion obviously worked. But the obsolete analytics
> > > tables
> > > (e.g. analytics_2012) are still there and populated. You can verify in
> the
> > > attached excerpt from catalina.out: only years >= 2016 were (correctly)
> > > updated,
> > > deleted years <= 2015 remained untouched.
> > >
> > > This is quite serious as it results into wrong statistics, if not
> > > detected. I
> > > just discovered this because of a major clean-up activity - if there
> are
> > > only
> > > very few years affected, users might not even notice that they are
> > > reporting
> > > deleted data.
> > >
> > > Currently running on 2.22, build 21765, Standard Ubuntu 14.04 &
> OracleJava
> > > 8.
> > >
> > > Thanks for checking on that,
> > >
> > > Uwe<catalina.out.2016-02-18>
> > > _______________________________________________
> > > Mailing list: DHIS 2 developers in Launchpad
> > > Post to : dhis2-devs@lists.launchpad.net
> > > Unsubscribe : DHIS 2 developers in Launchpad
> > > More help : ListHelp - Launchpad Help
> > >
> > >
> > >
> > > _______________________________________________
> > > Mailing list: DHIS 2 developers in Launchpad
> > > Post to : dhis2-devs@lists.launchpad.net
> > > Unsubscribe : DHIS 2 developers in Launchpad
> > > More help : ListHelp - Launchpad Help
> > >
> > >
> >
> >
> > --
> > Lars Helge Øverland
> > Lead developer, DHIS 2
> > University of Oslo
> > Skype: larshelgeoverland
> > http://www.dhis2.org <https://www.dhis2.org/>
>
> _______________________________________________
> Mailing list: DHIS 2 developers in Launchpad
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : DHIS 2 developers in Launchpad
> More help : ListHelp - Launchpad Help
>
thanks for the quick reaction. One more question on this: does the api for creating analytics work delta-sensitive (update only the changed years) or does it always rebuild the complete set of aggregates? If it has a delta mechanism, you might still want to consider doing Lars' blueprint to avoid lenghty aggregation runs every night ...
Regards,
Uwe
···
Am 19.02.2016 um 18:21 schrieb Morten Olav Hansen:
Thanks Uwe, I have put it on one of our new guys. Hopefully it will be part of 2.23.
On Fri, Feb 19, 2016 at 8:40 PM, Uwe Wahser <uwe@wahser.de > <mailto:uwe@wahser.de>> wrote:
Hi Morten,
thanks for joining in. I just created the blu-print:
@Olav P: would this be an applicable solution for your problem, too?
Regards, Uwe
---
> Morten Olav Hansen <morten@dhis2.org <mailto:morten@dhis2.org>>
hat am 19. Februar 2016 um 14:05
> geschrieben:
>
> Hi Uwe
>
> We have an API for clearing analytics cache [1], I'm sure we can
make one
> for clearing analytics tables also. Would you mind creating a
blueprint for
> us? and give me the link, and I will assign the proper people.
>
> [1] http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s34.html
>
> On Fri, Feb 19, 2016 at 4:25 PM, Uwe Wahser <uwe@wahser.de > <mailto:uwe@wahser.de>> wrote:
>
> > Hi Lars,
> >
> > thanks for taking this over. If there was an api for clearing
the analytics
> > table, which is giving back a result (unlike the one for
updating the
> > analytics
> > tables), that would already be a sufficient solution for
nightly runs.
> > Then I
> > could always delete the analytics tables before generating
them afresh.
> > Whatever
> > is easier for you to implement ...
> >
> > You have interesting working hours
> >
> > Regards, Uwe
> >
> > > Lars Helge Øverland <lars@dhis2.org <mailto:lars@dhis2.org>>
hat am 19. Februar 2016 um 04:38
> > > geschrieben:
> > >
> > > Hi Uwe,
> > >
> > > will work on this.
> > >
> > > https://bugs.launchpad.net/dhis2/+bug/1408096
> > >
> > > Short-term workaround is to go to data admin > maintenance
and click
> > Clear
> > > analytics tables.
> > >
> > > regards,
> > >
> > > Lars
> > >
> > > On Thu, Feb 18, 2016 at 3:32 AM, Olav Poppe > <olav.poppe@me.com <mailto:olav.poppe@me.com>> wrote:
> > >
> > > > Hi, I can only support/confirm what Uwe is saying here - I
just
> > experieced
> > > > the same thing this morning: deleted data for one year,
ran analytics,
> > and
> > > > the deleted data is still displayed.
> > > >
> > > > Olav
> > > >
> > > > 18. feb. 2016 kl. 07.38 skrev Uwe Wahser <uwe@wahser.de
<mailto:uwe@wahser.de>>:
> > > >
> > > > Dear devs,
> > > >
> > > > it seems like the analytics table update does not remove
data/tables of
> > > > years,
> > > > that are no longer valid.
> > > >
> > > > Since dataValues can be deleted via api in batch, it can
happen that
> > > > dataValues
> > > > for a whole year get deleted. Of course these years should
not show up
> > in
> > > > the
> > > > analytics (pivot, chart etc...) any more. However, those
obsolete
> > tables
> > > > seem to
> > > > remain in the database after running the analytics update.
Of course
> > they
> > > > could
> > > > be remove manually, but this is not applicable in a
nightly load
> > scenario.
> > > >
> > > > I did empty the cache and checked catalina.out, but I
can't find an
> > error.
> > > > In
> > > > the database the dataValues table remains with only the
years, that
> > were
> > > > not
> > > > deleted, so the deletion obviously worked. But the
obsolete analytics
> > > > tables
> > > > (e.g. analytics_2012) are still there and populated. You
can verify in
> > the
> > > > attached excerpt from catalina.out: only years >= 2016
were (correctly)
> > > > updated,
> > > > deleted years <= 2015 remained untouched.
> > > >
> > > > This is quite serious as it results into wrong statistics,
if not
> > > > detected. I
> > > > just discovered this because of a major clean-up activity
- if there
> > are
> > > > only
> > > > very few years affected, users might not even notice that
they are
> > > > reporting
> > > > deleted data.
> > > >
> > > > Currently running on 2.22, build 21765, Standard Ubuntu
14.04 &
> > OracleJava
> > > > 8.
> > > >
> > > > Thanks for checking on that,
> > > >
> > > > Uwe<catalina.out.2016-02-18>
> > > > _______________________________________________
> > > > Mailing list: DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > > > Post to : dhis2-devs@lists.launchpad.net
<mailto:dhis2-devs@lists.launchpad.net>
> > > > Unsubscribe : DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > > > More help : ListHelp - Launchpad Help
> > > >
> > > > _______________________________________________
> > > > Mailing list: DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > > > Post to : dhis2-devs@lists.launchpad.net
<mailto:dhis2-devs@lists.launchpad.net>
> > > > Unsubscribe : DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > > > More help : ListHelp - Launchpad Help
> > > >
> > >
> > > --
> > > Lars Helge Øverland
> > > Lead developer, DHIS 2
> > > University of Oslo
> > > Skype: larshelgeoverland
> > > http://www.dhis2.org <https://www.dhis2.org/>
> >
> > _______________________________________________
> > Mailing list: DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > Post to : dhis2-devs@lists.launchpad.net
<mailto:dhis2-devs@lists.launchpad.net>
> > Unsubscribe : DHIS 2 developers in Launchpad
<https://launchpad.net/~dhis2-devs>
> > More help : ListHelp - Launchpad Help
> >
>
> --
> Morten
