Feature #2696
closed
HA - move scheduled tasks and events to the current instance
Added by Alena Peterová almost 4 years ago. Updated over 3 years ago.
100%
Description
This feature is proposed for the basic high availability - mode active/passive.
When an active instance (say "idm-primary") shuts down, the second node (say "idm-secondary") is started. After that, the admin needs some way to move all system jobs in IdM to the currently active instance, so it's fully functional.
System jobs: scheduled tasks, Created entity events/states, or anything else which would make sense for the use case.
It could be some button, some LRT, whatever.
In the first version, admin will do this manually. It could be more automatized in the future.
Related issues
Updated by Vít Švanda almost 4 years ago
- Target version changed from 11.0.0 to 11.1.0
Updated by Vít Švanda almost 4 years ago
- Priority changed from High to Normal
- Target version changed from 11.1.0 to 11.0.0
Updated by Vít Švanda over 3 years ago
- Target version changed from 11.0.0 to 11.1.0
Updated by Radek Tomiška over 3 years ago
- Status changed from New to In Progress
Updated by Radek Tomiška over 3 years ago
- Related to Task #1367: Entity event queue - add base permissions added
Updated by Radek Tomiška over 3 years ago
- % Done changed from 0 to 80
Feature is implemented in branch:
https://github.com/bcvsolutions/CzechIdMng/tree/rtomiska/2696-switch-instance-for-asynchronous-processing
Documentation remains - will be added after version 11.0.0 will be released.
Updated by Radek Tomiška over 3 years ago
Updated by Radek Tomiška over 3 years ago
- Status changed from In Progress to Needs feedback
- Assignee changed from Radek Tomiška to Ondrej Husník
- % Done changed from 80 to 90
Documentation added:
https://wiki.czechidm.com/devel/documentation/application_configuration/dev/backend#scheduler
https://wiki.czechidm.com/devel/documentation/application_configuration/dev/backend#bulk_actions
https://wiki.czechidm.com/devel/documentation/application_configuration/dev/backend#entity_events
https://wiki.czechidm.com/devel/documentation/application_configuration/dev/backend#change_server_for_asynchronous_processing_switch_application_instance
Could you provide me a feedback, please?
Updated by Ondrej Husník over 3 years ago
I tested your task and basically works well. I found only some details that could be improved.
1) The value of idm.pub.app.instanceId property set in ....app\src\main\resources\application.properties cannot be seen in IdM Configuration- There is set the Idm instance name in the property file to e.g. idm-XXX
- If we run IdM and perform provisioning (for example ) we can see that corresponding entity event was executed on current Idm instance ie idm-XXX - setting from file works well
- But in the Configuration page there is still displayed idm-primary.
This can be very confusing for common users.
Steps to reproduce:
- Current instace id is eg. idm.pub.app.instanceId=idm-XXX
- Stop async event processing with idm.sec.core.event.asynchronous.stopProcessing=true
- Set new instance for async operation execution idm.sec.core.event.asynchronous.instanceId=idm-YYY
- Perform a kind of async operation eg provisioning. In entity event table there is a new event creted NOT run. It is planned for idm-YYY Idm instance.
- Start async event processing with idm.sec.core.event.asynchronous.stopProcessing=false
- Check entity event table. The newly created event is EXECUTED on current instance idm-XXX although planned for idm-YYY.
- It acts like situation in the point 2)
- The value is empty in Idm Configuration setting. Value form property file is not propagated to Configuration table. Similar to the point 1)
Updated by Ondrej Husník over 3 years ago
To reproduce:
- Create simple script (just return constant).
- Create a LRT executing this script (ExecuteScriptTaskExecutor)
- Delete this script.
- Try to reschedule Idm instance for async.
The instance change fails on the following exception. The problem is that it speaks about missing entity but you will be hardly able to find the LRT it pertains to especially in as of many scheduled tasks. I think there could be a specific exception for the task where rescheduling failed with its name. I guess there may be more situations when this may happen.
2021-06-15 12:11:45.702 ERROR 1237513 --- [base-task-executor-8] e.b.i.c.s.t.i.ExecuteScriptTaskExecutor.getScript : Script with code: [e420c71d-7103-4cf6-b24d-b9a4073e6081], not found.
2021-06-15 12:11:45.712 ERROR 1237523 --- [base-task-executor-8] e.b.i.c.a.b.a.AbstractBulkAction.processEntity : Processing of entity [d7f708b2-96f9-412b-bba7-26336ad2e166] failed.
eu.bcvsolutions.idm.core.api.exception.ResultCodeException: e420c71d-7103-4cf6-b24d-b9a4073e6081 not found.
at eu.bcvsolutions.idm.core.scheduler.task.impl.ExecuteScriptTaskExecutor.getScript(ExecuteScriptTaskExecutor.java:75)
at eu.bcvsolutions.idm.core.scheduler.task.impl.ExecuteScriptTaskExecutor.init(ExecuteScriptTaskExecutor.java:53)
at eu.bcvsolutions.idm.core.scheduler.task.impl.ExecuteScriptTaskExecutor$$FastClassBySpringCGLIB$$5c3aa2b8.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
at eu.bcvsolutions.idm.core.scheduler.task.impl.ExecuteScriptTaskExecutor$$EnhancerBySpringCGLIB$$93bfee6a.init(<generated>)
at eu.bcvsolutions.idm.core.scheduler.service.impl.DefaultSchedulerManager.updateTask(DefaultSchedulerManager.java:286)
at eu.bcvsolutions.idm.core.scheduler.service.impl.DefaultSchedulerManager.switchInstanceId(DefaultSchedulerManager.java:461)
at eu.bcvsolutions.idm.core.scheduler.service.impl.DefaultSchedulerManager$$FastClassBySpringCGLIB$$c2ee60b9.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:752)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at eu.bcvsolutions.idm.core.scheduler.service.impl.DefaultSchedulerManager$$EnhancerBySpringCGLIB$$92bd710d.switchInstanceId(<generated>)
at eu.bcvsolutions.idm.core.bulk.action.impl.configuration.ConfigurationSwitchInstanceBulkAction.processDto(ConfigurationSwitchInstanceBulkAction.java:195)
at eu.bcvsolutions.idm.core.bulk.action.impl.configuration.ConfigurationSwitchInstanceBulkAction.processDto(ConfigurationSwitchInstanceBulkAction.java:1)
at eu.bcvsolutions.idm.core.api.bulk.action.AbstractBulkAction.processEntity(AbstractBulkAction.java:429)
at eu.bcvsolutions.idm.core.api.bulk.action.AbstractBulkAction.processEntities(AbstractBulkAction.java:408)
at eu.bcvsolutions.idm.core.api.bulk.action.AbstractBulkAction.internalProcess(AbstractBulkAction.java:316)
at eu.bcvsolutions.idm.core.api.bulk.action.AbstractBulkAction.process(AbstractBulkAction.java:237)
at eu.bcvsolutions.idm.core.api.bulk.action.AbstractBulkAction.process(AbstractBulkAction.java:1)
at eu.bcvsolutions.idm.core.scheduler.api.service.AbstractLongRunningTaskExecutor.call(AbstractLongRunningTaskExecutor.java:269)
at eu.bcvsolutions.idm.core.scheduler.api.service.AbstractLongRunningTaskExecutor$$FastClassBySpringCGLIB$$f9eae371.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
at eu.bcvsolutions.idm.core.bulk.action.impl.configuration.ConfigurationSwitchInstanceBulkAction$$EnhancerBySpringCGLIB$$184e4351.call(<generated>)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at eu.bcvsolutions.idm.core.config.DelegatingTransactionContextRunnable.run(DelegatingTransactionContextRunnable.java:39)
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Added also small cleanup.
https://github.com/bcvsolutions/CzechIdMng/commit/7f73c40aec7c2488f0948fe4c5af00266c314a8a
Updated by Ondrej Husník over 3 years ago
- Status changed from Needs feedback to In Progress
- Assignee changed from Ondrej Husník to Radek Tomiška
Updated by Radek Tomiška over 3 years ago
- Status changed from In Progress to Needs feedback
- Assignee changed from Radek Tomiška to Ondrej Husník
Thx for feedback!
ad2) Fixed wrong configuration property, used for asynchronous processing.
ad4) Validations are ignored, when scheduled tasks are moved into different instance (it's not related to switch operation => task configuration should be fixed, when task is scheduled / edited).
Commit:
https://github.com/bcvsolutions/CzechIdMng/commit/86f313127fe04071ca305a3cc32939d7dea59b83
Other notes are not reproducible. Could you provide me a feedback again, please?
Updated by Ondrej Husník over 3 years ago
- Status changed from Needs feedback to Resolved
- Assignee changed from Ondrej Husník to Radek Tomiška
- % Done changed from 90 to 100
Thanks for fixes. Both fixed points works correctly now. The irreproducible ones were probably caused by wrong file refresh on my side.
Updated by Radek Tomiška over 3 years ago
- Status changed from Resolved to Closed