Data integrity check produces a server error

Hi

I am experimenting with a version 2.15 instance of dhis2.
To populate this instance, I imported data from a production instance running version 2.13 using the web api to grab all the resources in the api/resources list (making sure that I get the right resource where the resource links point to “dimensions”).

I didn’t simply import the metadata resource because I get a 403 error when I try to access it. For the same reason I had to skip the users resource (perhaps access to users resource blocks me from accessing the metaData resource?)

I exported the resources as XML and used the viewClass=export parameter to get the right level of detail.
Import was done with curl, into the metaData resource of my experimental system.

After the initial import, some links were missing (such as the lists of indicators in dataSets). This was solved by re-importing some of the resources until no missing data was seen (verified by comparing api resources visually in browser with production system)

Once all the metadata seemed in order, I ran the data integrity check and got a server error.
I thought perhaps data integrity checks require actual data values so I went ahead and imported all the datavalues using the dataValueSet web api resource.

Data integrity still produced a server error so I created the resource tables.

Trying to run data integrity check again, I got the error below.
Any idea what the cause may be?

Thanks,

Lavi

PS before experimenting with a 2.15 version, I tried the same with a 2.13 version (same as production system) and faced the same issues.
PPS Thanks a million for the great super-fast and accurate replies I’ve been getting so far !

ERROR 2014-07-24 10:55:17,112 Error while executing action (ExceptionInterceptor.java [http-bio-8060-exec-33])
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1234)

    at java.util.regex.Matcher.reset(Matcher.java:308)
    at java.util.regex.Matcher.<init>(Matcher.java:228)
    at java.util.regex.Pattern.matcher(Pattern.java:1088)
    at org.hisp.dhis.dataentryform.DefaultDataEntryFormService.getDataElementsInDataEntryForm(DefaultDataEntryFormService.java:368)

    at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)

    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)

    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

    at com.sun.proxy.$Proxy63.getDataElementsInDataEntryForm(Unknown Source)
    at org.hisp.dhis.dataintegrity.DefaultDataIntegrityService.getDataElementsInDataSetNotInForm(DefaultDataIntegrityService.java:252)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)

    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

    at com.sun.proxy.$Proxy89.getDataElementsInDataSetNotInForm(Unknown Source)
    at org.hisp.dhis.dataadmin.action.dataintegrity.GetDataIntegrityAction.execute(GetDataIntegrityAction.java:228)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)

    at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:450)
    at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:289)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:252)
    at org.hisp.dhis.webportal.interceptor.XWorkPortalUserInterceptor.intercept(XWorkPortalUserInterceptor.java:88)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.webportal.interceptor.XWorkPortalModuleInterceptor.intercept(XWorkPortalModuleInterceptor.java:89)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.webportal.interceptor.XWorkPortalMenuInterceptor.intercept(XWorkPortalMenuInterceptor.java:97)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.webportal.interceptor.XWorkPortalParamsInterceptor.intercept(XWorkPortalParamsInterceptor.java:147)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.interceptor.ContextInterceptor.intercept(ContextInterceptor.java:83)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)

    at org.hisp.dhis.interceptor.UserSettingInterceptor.intercept(UserSettingInterceptor.java:81)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.security.intercept.LoginInterceptor.intercept(LoginInterceptor.java:85)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.security.intercept.XWorkSecurityInterceptor.intercept(XWorkSecurityInterceptor.java:113)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)

    at org.hisp.dhis.interceptor.SystemSettingInterceptor.intercept(SystemSettingInterceptor.java:112)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:136)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:249)
    at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)

    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
    at org.hisp.dhis.interceptor.ExceptionInterceptor.intercept(ExceptionInterceptor.java:110)
    at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)

    at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
    at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:562)
    at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)

    at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.hisp.dhis.security.filter.CustomAuthenticationFilter.doFilter(CustomAuthenticationFilter.java:64)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.hisp.dhis.security.filter.AutomaticAccessFilter.doFilter(AutomaticAccessFilter.java:113)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)

    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:149)

    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)

    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Hi Lavi,

sorry for trouble, see comments below.

···

On Thu, Jul 24, 2014 at 11:34 AM, Lavi Shpigelman shpigi@gmail.com wrote:

Hi

I am experimenting with a version 2.15 instance of dhis2.

To populate this instance, I imported data from a production instance running version 2.13 using the web api to grab all the resources in the api/resources list (making sure that I get the right resource where the resource links point to “dimensions”).

I didn’t simply import the metadata resource because I get a 403 error when I try to access it. For the same reason I had to skip the users resource (perhaps access to users resource blocks me from accessing the metaData resource?)

Okay. Could you check that your user has the ALL or Export meta-data authority (through its user roles)? As you probably know this is a more convenient way of getting all meta-data.

I exported the resources as XML and used the viewClass=export parameter to get the right level of detail.
Import was done with curl, into the metaData resource of my experimental system.

After the initial import, some links were missing (such as the lists of indicators in dataSets). This was solved by re-importing some of the resources until no missing data was seen (verified by comparing api resources visually in browser with production system)

Okay that’s strange.

Once all the metadata seemed in order, I ran the data integrity check and got a server error.
I thought perhaps data integrity checks require actual data values so I went ahead and imported all the datavalues using the dataValueSet web api resource.

Data integrity still produced a server error so I created the resource tables.

Trying to run data integrity check again, I got the error below.
Any idea what the cause may be?

I have seen this error before. It is fixed in latest 2.15 build. The crash below has two causes. First, he export previously contained empty elements even for data sets without custom forms. This lead to the import creating “blank” forms on the receiving side. Second, the code parsing custom forms had a weakness in that it optimistically tried to parse the form content without checking for null. Both these issues have been fixed in latest 2.15 now. You can grab it from dhis2.org/downloads in 40 min.

best regards,

Lars