Project

General

Profile

Actions

Defect #2312

closed

Import of several virtual systems at once causes "Too many open files" errors and unavailability of IdM

Added by Vladimír Kotýnek almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Vít Švanda
Category:
Export Import
Target version:
Start date:
06/09/2020
Due date:
% Done:

100%

Estimated time:
Affected versions:
Owner:

Description

@10.3.2

I have an export .zip file with 38 virtual systems. After I uploaded the file to Export/Import site in IDM I run "dry run" or "demo mode" - the blue triangle. After the demo mode run started the application was unavailable with error:

<ResultModels>
<_errors>
<_errors>
<id>480f5db2-d836-4793-9796-339e88148038</id>
<creation>2020-06-09T14:06:36.879+02:00</creation>
<statusEnum>INTERNAL_SERVER_ERROR</statusEnum>
<message>java.io.FileNotFoundException: /opt/tomcat/apache-tomcat-8.5.47/webapps/idm/WEB-INF/lib/junit-jupiter-api-5.3.2.jar (Too many open files)</message>
<parameters/>
<module>core</module>
<statusCode>500</statusCode>
<status>INTERNAL_SERVER_ERROR</status>
</_errors>
</_errors>
</ResultModels>

In the catalina.out log I can see a lot of "Too many open files" exceptions. The demo mode run itself failded with following error:

2020-06-09 14:05:47.157  INFO 328262572 --- [base-task-executor-4] e.b.i.c.m.s.i.DefaultImportManager.internalExecuteImport : Internal import [eu.bcvsolutions.idm.core.api.dto.IdmExportImportDto [id=73daa6
b8-67b1-4d01-a617-315b1ac97c13], dry-run: true] ended
2020-06-09 14:05:47.169 ERROR 328262584 --- [base-task-executor-4] e.b.i.c.s.a.s.AbstractLongRunningTaskExecutor.end : [core:IMPORT_ZIP_EXTRACTION_FAILED:cbce0d75-5015-470b-8488-79dc92b88fc5] Extraction of
 ZIP import failed! ({})
eu.bcvsolutions.idm.core.api.exception.ResultCodeException: Extraction of ZIP import failed!
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.getParentDtoFromBatch(DefaultImportManager.java:853)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.lambda$executeImportForType$5(DefaultImportManager.java:279)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.executeImportForType(DefaultImportManager.java:273)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.lambda$internalExecuteImport$1(DefaultImportManager.java:221)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.internalExecuteImport(DefaultImportManager.java:220)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager$$FastClassBySpringCGLIB$$6e60f64f.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager$$EnhancerBySpringCGLIB$$87d71c5b.internalExecuteImport(<generated>)
        at eu.bcvsolutions.idm.core.scheduler.task.impl.ImportTaskExecutor.process(ImportTaskExecutor.java:69)
        at eu.bcvsolutions.idm.core.scheduler.task.impl.ImportTaskExecutor.process(ImportTaskExecutor.java:32)
        at eu.bcvsolutions.idm.core.scheduler.api.service.AbstractLongRunningTaskExecutor.call(AbstractLongRunningTaskExecutor.java:212)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /opt/czechidm/data/temp/3580194995400728524/SysRoleSystemDto: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
        at java.nio.file.Files.newDirectoryStream(Files.java:457)
        at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
        at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
        at java.nio.file.Files.walk(Files.java:3574)
        at java.nio.file.Files.walk(Files.java:3625)
        at eu.bcvsolutions.idm.core.model.service.impl.DefaultImportManager.getParentDtoFromBatch(DefaultImportManager.java:842)
        ... 23 common frames omitted

The imported file contains:

   15 files in directory .
   38 files in directory ./SysSyncIdentityConfigDto
  514 files in directory ./SysSystemAttributeMappingDto
    4 files in directory ./IdmFormDefinitionDto
    1 files in directory ./IdmExportImportDto
    3 files in directory ./IdmFormInstanceDto
   38 files in directory ./SysSystemDto
   76 files in directory ./SysSystemMappingDto
   38 files in directory ./SysSchemaObjectClassDto
    1 files in directory ./SysProvisioningBreakConfigDto
    1 files in directory ./SysProvisioningBreakRecipientDto
   15 files in directory ./IdmFormAttributeDto
  526 files in directory ./SysRoleSystemDto
  482 files in directory ./SysRoleSystemAttributeDto
  478 files in directory ./SysSchemaAttributeDto

2213 files in total

Actions #2

Updated by Vít Švanda almost 4 years ago

  • Status changed from New to In Progress
  • Target version set to 10.4.0

I tried to import your zip into my IdM and this problem did not occur. The limit on the number of open files depends on the server settings. How big is this limit on your server?

Actions #3

Updated by Vladimír Kotýnek almost 4 years ago

for the Tomcat process the limit from /proc/<pid>/limits is:

Limit                     Soft Limit           Hard Limit           Units     
Max open files            4096                 4096                 files

But for the tomcat user it's:

su - tomcat -c 'ulimit -aHS' -s '/bin/bash' | grep "open files" 
open files                      (-n) 1024

This might be the limit that causes this error. I think we shoul prevent the IDM from exhousting it's resources while importing data.

Actions #4

Updated by Vít Švanda almost 4 years ago

  • % Done changed from 0 to 30

I am not sure If file search utility can close stream for every file or only after search is completed (I doesn't have this under control now), but I made some modification and optimizations, so we will see.

Commit: https://github.com/bcvsolutions/CzechIdMng/commit/590eb494fd8a94e2ca81596be8feba9462d064f9

Generally workaround here is clear: Increase the number of opens file on the server.

Actions #5

Updated by Vít Švanda almost 4 years ago

  • Status changed from In Progress to Needs feedback
  • Assignee changed from Vít Švanda to Radek Tomiška
Actions #6

Updated by Vít Švanda almost 4 years ago

Could you please make review and test on some Linux server please? :-)

Actions #7

Updated by Radek Tomiška almost 4 years ago

  • Status changed from Needs feedback to Closed
  • Assignee changed from Radek Tomiška to Vít Švanda
  • % Done changed from 30 to 100

I did test on linux (tested on original zip file, 1024 open files), dry run works now, thx!

Actions

Also available in: Atom PDF