Defect #2312
closedImport of several virtual systems at once causes "Too many open files" errors and unavailability of IdM
100%
Description
@10.3.2
I have an export .zip file with 38 virtual systems. After I uploaded the file to Export/Import site in IDM I run "dry run" or "demo mode" - the blue triangle. After the demo mode run started the application was unavailable with error:
<ResultModels>
<_errors>
<_errors>
<id>480f5db2-d836-4793-9796-339e88148038</id>
<creation>2020-06-09T14:06:36.879+02:00</creation>
<statusEnum>INTERNAL_SERVER_ERROR</statusEnum>
<message>java.io.FileNotFoundException: /opt/tomcat/apache-tomcat-8.5.47/webapps/idm/WEB-INF/lib/junit-jupiter-api-5.3.2.jar (Too many open files)</message>
<parameters/>
<module>core</module>
<statusCode>500</statusCode>
<status>INTERNAL_SERVER_ERROR</status>
</_errors>
</_errors>
</ResultModels>
In the catalina.out log I can see a lot of "Too many open files" exceptions. The demo mode run itself failded with following error:
2020-06-09 14:05:47.157 INFO 328262572 --- [base-task-executor-4] e.b.i.c.m.s.i.DefaultImportManager.internalExecuteImport : Internal import [eu.bcvsolutions.idm.core.api.dto.IdmExportImportDto [id=73daa6 b8-67b1-4d01-a617-315b1ac97c13], dry-run: true] ended 2020-06-09 14:05:47.169 ERROR 328262584 --- [base-task-executor-4] e.b.i.c.s.a.s.AbstractLongRunningTaskExecutor.end : [core:IMPORT_ZIP_EXTRACTION_FAILED:cbce0d75-5015-470b-8488-79dc92b88fc5] Extraction of ZIP import failed! ({}) eu.bcvsolutions.idm.core.api.exception.ResultCodeException: Extraction of ZIP import failed! at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.getParentDtoFromBatch(DefaultImportManager.java:853) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.lambda$executeImportForType$5(DefaultImportManager.java:279) at java.util.ArrayList.forEach(ArrayList.java:1257) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.executeImportForType(DefaultImportManager.java:273) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.lambda$internalExecuteImport$1(DefaultImportManager.java:221) at java.util.ArrayList.forEach(ArrayList.java:1257) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.internalExecuteImport(DefaultImportManager.java:220) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager$$FastClassBySpringCGLIB$$6e60f64f.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager$$EnhancerBySpringCGLIB$$87d71c5b.internalExecuteImport(<generated>) at eu.bcvsolutions.idm.core.scheduler.task.impl.ImportTaskExecutor.process(ImportTaskExecutor.java:69) at eu.bcvsolutions.idm.core.scheduler.task.impl.ImportTaskExecutor.process(ImportTaskExecutor.java:32) at eu.bcvsolutions.idm.core.scheduler.api.service.AbstractLongRunningTaskExecutor.call(AbstractLongRunningTaskExecutor.java:212) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.nio.file.FileSystemException: /opt/czechidm/data/temp/3580194995400728524/SysRoleSystemDto: Too many open files at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) at java.nio.file.Files.newDirectoryStream(Files.java:457) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300) at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72) at java.nio.file.Files.walk(Files.java:3574) at java.nio.file.Files.walk(Files.java:3625) at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.getParentDtoFromBatch(DefaultImportManager.java:842) ... 23 common frames omitted
The imported file contains:
15 files in directory . 38 files in directory ./SysSyncIdentityConfigDto 514 files in directory ./SysSystemAttributeMappingDto 4 files in directory ./IdmFormDefinitionDto 1 files in directory ./IdmExportImportDto 3 files in directory ./IdmFormInstanceDto 38 files in directory ./SysSystemDto 76 files in directory ./SysSystemMappingDto 38 files in directory ./SysSchemaObjectClassDto 1 files in directory ./SysProvisioningBreakConfigDto 1 files in directory ./SysProvisioningBreakRecipientDto 15 files in directory ./IdmFormAttributeDto 526 files in directory ./SysRoleSystemDto 482 files in directory ./SysRoleSystemAttributeDto 478 files in directory ./SysSchemaAttributeDto
2213 files in total
Updated by Vít Švanda almost 4 years ago
- Status changed from New to In Progress
- Target version set to 10.4.0
I tried to import your zip into my IdM and this problem did not occur. The limit on the number of open files depends on the server settings. How big is this limit on your server?
Updated by Vladimír Kotýnek almost 4 years ago
for the Tomcat process the limit from /proc/<pid>/limits is:
Limit Soft Limit Hard Limit Units Max open files 4096 4096 files
But for the tomcat user it's:
su - tomcat -c 'ulimit -aHS' -s '/bin/bash' | grep "open files" open files (-n) 1024
This might be the limit that causes this error. I think we shoul prevent the IDM from exhousting it's resources while importing data.
Updated by Vít Švanda almost 4 years ago
- % Done changed from 0 to 30
I am not sure If file search utility can close stream for every file or only after search is completed (I doesn't have this under control now), but I made some modification and optimizations, so we will see.
Commit: https://github.com/bcvsolutions/CzechIdMng/commit/590eb494fd8a94e2ca81596be8feba9462d064f9
Generally workaround here is clear: Increase the number of opens file on the server.
Updated by Vít Švanda almost 4 years ago
- Status changed from In Progress to Needs feedback
- Assignee changed from Vít Švanda to Radek Tomiška
Updated by Vít Švanda almost 4 years ago
Could you please make review and test on some Linux server please? :-)
Updated by Radek Tomiška almost 4 years ago
- Status changed from Needs feedback to Closed
- Assignee changed from Radek Tomiška to Vít Švanda
- % Done changed from 30 to 100
I did test on linux (tested on original zip file, 1024 open files), dry run works now, thx!