IdStory Identity Manager

Something like this should do the trick:

        <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
            <file>idm.log</file>
            <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                <fileNamePattern>idm.log.%d{yyyy-MM-dd}</fileNamePattern>
                <maxHistory>30</maxHistory>
            </rollingPolicy>
            <encoder>
                <pattern>${FILE_LOG_PATTERN}</pattern>
            </encoder>
        </appender>
        <root level="INFO">
            <appender-ref ref="FILE" />
        </root>

Worked around this but so far for Linux only.

Log rotation happens using logrotate with copytruncate option specified. Default tomcat logger always appends the logfile, so sparse files do not happen.
Documented to wiki: https://wiki.czechidm.com/tutorial/adm/server_preparation#rotating_tomcat_logs .

We are using logback for logging:
https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/base.xml

log rotation is configured by 10MB by default:
https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback/file-appender.xml

This is included in IdM by 'logback-spring.xml' configuration file.

Turning off the default logback configuration with the redundant logs written into CONSOLE (~tomcat) => (copy + update base.xml), can do the trick. Then native tomcat log files will contains only tomcat errors then and will be small enough i hope so (custom linux script for this logs will not be needed).

Official doc:
https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html#howto-configure-logback-for-logging-fileonly

Configuration for rotating logs is not installation tutorial nor in 'logback-spring.xml' configuration file.
Installation tutorial must be updated.

I tried setup local test environment with CentOS and IdM 9.4.0 but tomcat can't start because he can't connect to postgreslq even though I can manually connect to postrgresql and there isn't any restriction on network.

For a testing purpose a used windows machine "BAKALARI".
I updated IdM to v 9.6.0 .

Then I tried set-up logging to file( no rotation) through logback-spring.xml .
Tomcat created logfile but not in tomcat/logs directory.
Next step is force Tomcat to logging to tomcat/logs and set-up file rotation.

Look out, log-back setting schould be configured to prevent usage tomcat logs. Tomcat logs will be on the same place.

I changed configuration in "logback-spring.xml".
I changed "include" because otherwise there will be duplicates in idm.log and tomcat8-stout.log .
I added new appender to write all INFO+ log to file /logs/idm.log

Not final version of "logback-spring.xml".

<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>

        <appender name="POKUS" class="ch.qos.logback.core.FileAppender">
                <encoder>
                        <pattern>
                                %d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{36}.%M - %msg%n
                        </pattern>
                </encoder>
                <file>logs/idm.log</file>
        </appender>

        <root level="INFO">
                <appender-ref ref="POKUS"/>
        </root>

• Change connector logs to log to idm.log.
• Rotate idm.log.

Configure connectors to using logback:

<logger name="org.identityconnectors" level="WARN"/>
<logger name="net.tirasa.connid" level="WARN"/>

More here: #1141

Why do you need configure tomcat logs?

After consultation with Petr Fiser I figured out that changing logback will not help because tomcat, after all redirecting all logs to idm.log through logback, still generates stdout a stderr files. Soo I returned to the idea of changing "logging_properties". I make these changes:

handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler,  5stdout.org.apache.juli.AsyncFileHandler, 6stderr.org.apache.juli.AsyncFileHandler

5stdout.org.apache.juli.AsyncFileHandler.level = INFO
5stdout.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
5stdout.org.apache.juli.AsyncFileHandler.prefix = tomcat8-stdout2.
5stdout.org.apache.juli.AsyncFileHandler.rotatable = true
5stdout.org.apache.juli.AsyncFileHandler.maxDays = 90
5stdout.org.apache.juli.AsyncFileHandler.suffix = .log

6stderr.org.apache.juli.AsyncFileHandler.level = ALL
6stderr.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
6stderr.org.apache.juli.AsyncFileHandler.prefix = tomcat8-stderr2.
6stderr.org.apache.juli.AsyncFileHandler.rotatable = true
6stderr.org.apache.juli.AsyncFileHandler.maxDays = 90
6stderr.org.apache.juli.AsyncFileHandler.suffix = .log

This still doesn't works but if I can force tomcat to redirect stdout/stderr to use this handler, I can make tomcat to rotate logs.

I manage redirection of error logs when I coment out all lines with "java.util.logging.ConsoleHandler" and added this line

 .handlers = 1catalina.org.apache.juli.AsyncFileHandler, 6stderr.org.apache.juli.AsyncFileHandler

I successfully tested rotation of stderr.

I moved closer to resolving tomcat log rotation with this configuration:

• Logs from IdM
  I changed logback spring in IdM configuration. All idm log now will be in idm.log .
  

          <!-- <include resource="org/springframework/boot/logging/logback/base.xml"/> -->
          <include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
          <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  
        <appender name="idm" class="ch.qos.logback.core.rolling.RollingFileAppender">
                  <encoder>
                          <pattern>
                                  %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36}.%M - %msg%n
                          </pattern>
                  </encoder>
                  <file>logs/idm.log</file>
                  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                          <fileNamePattern>logs/idm.%d{yyyy-MM-dd}.log</fileNamePattern>
                          <maxHistory>90</maxHistory>
                  </rollingPolicy>
          </appender>
  
          <root level="INFO">
                  <appender-ref ref="idm"/>
          </root>
  

• Tomcat's stdout
  In tomcat monitor app a turn off logging to tomcat8-stdout.log by changing logging-> redirect stdout : "auto" to "".
  We lost some logs from connectors but in idm.log there should be copy of it.
  

  Thread Id: 17    Time: 2019-06-01 04:17:43.902    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=net.tirasa.connid.bundles.db.scriptedsql bundleVersion=2.2.4 connectorName=net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/net.tirasa.connid.bundles.db.scriptedsql-2.2.4.jar
  Thread Id: 17    Time: 2019-06-01 04:17:43.979    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=net.tirasa.connid.bundles.ldap bundleVersion=1.5.1 connectorName=net.tirasa.connid.bundles.ldap.LdapConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/net.tirasa.connid.bundles.ldap-1.5.1.jar
  Thread Id: 17    Time: 2019-06-01 04:17:44.058    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=net.tirasa.connid.bundles.db.table bundleVersion=2.2.4 connectorName=net.tirasa.connid.bundles.db.table.DatabaseTableConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/net.tirasa.connid.bundles.db.table-2.2.4.jar
  Thread Id: 17    Time: 2019-06-01 04:17:44.152    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=net.tirasa.connid.bundles.csvdir bundleVersion=0.8.5 connectorName=net.tirasa.connid.bundles.csvdir.CSVDirConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/net.tirasa.connid.bundles.csvdir-0.8.5.jar
  Thread Id: 17    Time: 2019-06-01 04:17:44.276    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=net.tirasa.connid.bundles.ad bundleVersion=1.3.4.25 connectorName=net.tirasa.connid.bundles.ad.ADConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/net.tirasa.connid.bundles.ad-1.3.4.25.jar
  Thread Id: 17    Time: 2019-06-01 04:17:44.291    Class: org.identityconnectors.framework.impl.api.local.LocalConnectorInfoManagerImpl    Method: createConnectorInfo(LocalConnectorInfoManagerImpl.java:301)    Level: INFO    Message: Add ConnectorInfo ConnectorKey( bundleName=csv-connector bundleVersion=1.0.0 connectorName=eu.bcvsolutions.idm.connectors.csv.CSVConnConnector ) to Local Connector Info Manager from file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%208.5/webapps/idm/WEB-INF/lib/csv-connector-1.0.0.jar
  

  
  This must be tested

• Tomcat's stderr
  In tomcat monitor app a turn off logging to tomcat8-stderr.log by changing logging-> redirect stderror : "auto" to "".
  Then I edited Tomcat 8.5/conf/logging.properties .
  I commented out java.util.logging.ConsoleHandler to eliminate stderr logging trough console.
  

  handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler
  #handlers =  java.util.logging.ConsoleHandler
  
  .handlers = 1catalina.org.apache.juli.AsyncFileHandler
  #.handlers = java.util.logging.ConsoleHandler,
  
  #java.util.logging.ConsoleHandler.level = FINE
  #java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
  

Then I set-up rotating of catalina.log.
All stderr log will end here.

1catalina.org.apache.juli.AsyncFileHandler.level = INFO
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.rotatable = true
1catalina.org.apache.juli.AsyncFileHandler.maxDays = 90
1catalina.org.apache.juli.AsyncFileHandler.suffix = .log

• TODO
  - Test this configuration on project

After test on project it emerged that logs from connector in tomcat-stdout are important. Soo we can't simply turn them off. I'm trying to find a way to redirect them.

Logs from connector schould be configured to logback too, see above.

I still didn't found the way to redirect conector logs. Soo I ask Radek for help and he found library that can work https://mvnrepository.com/artifact/net.tirasa.connid/slf4j-logging/1.4.2.0 .

Actual state

Idm conector log can't be redirected(in this state) so all of they are saved in tomcat-stdout file. After 2 month of testing process on project these log create 11GB unrotatable file( all other idm logs create 4GB and they rotate).
Is needed to be change conectors's loging in order to stop tomcat from creating these big files

We can add lib slf4j-logging-1.4.2.0.jar into product dependencies, if it works as desired.

• tomcat.log - logs from tomcat container, contain logs only from start and stop of tomcat, will not be rotated
• catalina.log - logs from idm and connectors, will be rotated

Tomcat will still create stdout and stderr, but these files will contain logs from start of tomcat and error if logback will not start properly.

Configuration
Tomcat 8.5/conf/logging.properties
I commented out java.util.logging.ConsoleHandler to eliminate stderr and stdout logging trough console.
Then redirect tomcat container logs from catalina.log to tomcat.log. It's because if there will be only 1 file logback can't rotate it because tomcat holds it open. Tomcat.log is not rotated, but can be rotated if needed.

handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, 3manager.org.apache.juli.AsyncFileHandler, 4host-manager.org.apache.juli.AsyncFileHandler
#handlers =  java.util.logging.ConsoleHandler

.handlers = 1catalina.org.apache.juli.AsyncFileHandler
#.handlers = java.util.logging.ConsoleHandler,

#java.util.logging.ConsoleHandler.level = FINE
#java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
...
...
#1catalina.org.apache.juli.AsyncFileHandler.level = FINE
#1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
1catalina.org.apache.juli.AsyncFileHandler.level = INFO
1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.AsyncFileHandler.prefix = tomcat
1catalina.org.apache.juli.AsyncFileHandler.rotatable = false
1catalina.org.apache.juli.AsyncFileHandler.suffix = .log

<!-- this is comented for tomcat rotatio -->
<!-- <include resource="org/springframework/boot/logging/logback/base.xml"/> -->
<springProperty name="spring.profiles.active" source="spring.profiles.active"/>

        <!--    Begin tomcat rotation -->
<!-- <include resource="org/springframework/boot/logging/logback/base.xml"/> -->
<include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>

<appender name="idm" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder>
                <pattern>
                    %d{yyyy-MM-dd HH:mm:ss.SSS} %5level %relative --- [%thread] %logger{36}.%M : %msg%n
                </pattern>
        </encoder>
        <file>logs/catalina.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                <fileNamePattern>logs/catalina.%d{yyyy-MM-dd}.log</fileNamePattern>
                <maxHistory>90</maxHistory>
        </rollingPolicy>
</appender>

<root level="INFO">
        <appender-ref ref="idm"/>
</root>

Due to great changes in logback-spring.xml between linux and windows usage I will create new lockback configuration for windows. This configuration will contain only things we need for windows production usage. Also windows idm will run on different profile( production -> windows).

New Windows logback-spring.xml with log rotation:

<?xml version="1.0" encoding="UTF-8"?>
<!-- https://springframework.guru/using-logback-spring-boot/ -->
<!-- http://logback.qos.ch/manual/appenders.html -->
<configuration>
        <springProperty name="spring.profiles.active" source="spring.profiles.active"/>
        <include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
        <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
        <springProfile name="production">
                <logger name="eu.bcvsolutions" level="INFO"/>
                <logger name="org.springframework" level="INFO"/>
                <logger name="org.springframework.web" level="INFO"/>
                <logger name="org.hibernate.SQL" level="INFO"/>
                <logger name="org.hibernate.type.descriptor.sql.BasicBinder" level="INFO"/>
                <appender name="idm" class="ch.qos.logback.core.rolling.RollingFileAppender">
                        <encoder>
                                <pattern>
                                         %d{yyyy-MM-dd HH:mm:ss.SSS} %5level %relative --- [%thread] %logger{36}.%M : %msg%n
                                </pattern>
                        </encoder>
                        <file>logs/catalina.log</file>
                        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                                <fileNamePattern>logs/catalina.%d{yyyy-MM-dd}.log</fileNamePattern>
                                <maxHistory>90</maxHistory>
                        </rollingPolicy>
                </appender>     
                <root level="INFO">
                        <appender-ref ref="idm"/>
                </root>
        </springProfile>
</configuration>

I tested hotfix 9.7.11 on Windows and log rotation worked properly.