Task #2896
open
When synchronization LRT fails the synchrohization should not stay in "running" state
0%
Description
In my environment I have IdM database on a different server than the CzechIdM application. The network connection between those two servers was unavailable for a brief moment during a reconciliation of org. structure (tree nodes) at night.
The LRT of the sync. failed with Exception:
org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:563)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:838)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:812)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:552)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:299)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at eu.bcvsolutions.idm.acc.service.impl.DefaultSysSyncLogService$$EnhancerBySpringCGLIB$$70dc3c1e.save(<generated>)
at eu.bcvsolutions.idm.acc.service.impl.TreeSynchronizationExecutor.process(TreeSynchronizationExecutor.java:173)
at eu.bcvsolutions.idm.acc.service.impl.DefaultSynchronizationService.startSynchronization(DefaultSynchronizationService.java:189)
at eu.bcvsolutions.idm.acc.service.impl.DefaultSynchronizationService$$FastClassBySpringCGLIB$$66d7ee75.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
at eu.bcvsolutions.idm.acc.service.impl.DefaultSynchronizationService$$EnhancerBySpringCGLIB$$c1df17d2.startSynchronization(<generated>)
at eu.bcvsolutions.idm.acc.scheduler.task.impl.SynchronizationSchedulableTaskExecutor.process(SynchronizationSchedulableTaskExecutor.java:71)
at eu.bcvsolutions.idm.acc.scheduler.task.impl.SynchronizationSchedulableTaskExecutor.process(SynchronizationSchedulableTaskExecutor.java:31)
at eu.bcvsolutions.idm.core.scheduler.api.service.AbstractLongRunningTaskExecutor.call(AbstractLongRunningTaskExecutor.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at eu.bcvsolutions.idm.core.config.DelegatingTransactionContextRunnable.run(DelegatingTransactionContextRunnable.java:39)
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:122)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:294)
at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:145)
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:559)
... 23 more
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy117.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:370)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:116)
... 26 more
However the sync. is still shown as "running", but it's not processing any items anymore.
The problem of this situation is that the sync. won't start the next day as it should because IdM "thinks" it's still running.
I've managed to simulate the situation in my local environment when I started the sync. and restarted the postgresql service while it was running and processing items.
I think it would be helpful if the sync correctly failed when it's LRT fails. But I don't know whether it's possible since the fail is due to DB connectivity problem.
This happened twice to me, once on 10.6 and once on 10.8. And I did the local simulation with 11.1.
Files
Updated by Vít Švanda over 3 years ago
I'm afraid that 'cancel' synchronization is not possible in this case (without DB). In this scenario, however, a mechanism should work where after IdM restart, all 'running' syncs should be automatically terminate. Does this work?
Updated by Vladimír Kotýnek over 3 years ago
Yes, restart of CzechIdM sets the sync as not running. But it doesn't actually help me in the situation above. You can also cancel the synchronisation manually in GUI. The problem is that I don't know that something went wrong until I check the sync in System configuration detail. Also you don't always want to restart a production application "just in case" as a prevention.
And I've just managed to get the sync to state where both LRT and sync are "running", no exception/error anywhere except catalina.out and no more items are being processed.
I guess this whole topic of recovery from disconnectiong & reconnecting the DB while LRT is running would be much complicated task than i taught.