Delete and roll configuration help

[Shawcs]

Hello community !

After few weeks of iteration I seek your help to configure my log4j2 in kafka properly.

Here is a simplified property file I created to just focus on 2 appender

Configuration:
  #status: debug #to have logs about log4j itself
  Properties:
    Property:
      # Fallback if the system property is not set
      - name: "kafka.logs.dir"
        value: "{{ kafka_logs_dir }}"
      - name: "logPattern"
        value: "[%d] %p %m (%c)%n"

  Appenders:
    Console:
      name: STDOUT
      PatternLayout:
        pattern: "${logPattern}"

    RollingFile:
      - name: KafkaServerAppender
        fileName: "${sys:kafka.logs.dir}/server.log"
        filePattern: "${sys:kafka.logs.dir}/server.%d{yyyy-MM-dd:HH'h':mm'm'}.log"
        PatternLayout:
          pattern: "${logPattern}"
        Policies:
          TimeBasedTriggeringPolicy:
            interval: 1
            modulate: true
          SizeBasedTriggeringPolicy:
            size: 100 MB
        DefaultRolloverStrategy:
          max: 5
        #Taken from: https://logging.apache.org/log4j/2.x/manual/appenders/rolling-file.html#timestamped
        DirectWriteRolloverStrategy:
          Delete:
            basePath: "${sys:kafka.logs.dir}"
            IfFileName:
              regex: "server.log.*"
            IfLastModified:
              age: "PT15M"

      # Request appender
      - name: RequestAppender
        fileName: "${sys:kafka.logs.dir}/kafka-request.log"
        filePattern: "${sys:kafka.logs.dir}/kafka-request.log.%d{yyyy-MM-dd:HH'h'}"
        PatternLayout:
          pattern: "${logPattern}"
        Policies:
          TimeBasedTriggeringPolicy:
            interval: 12
            modulate: false
          SizeBasedTriggeringPolicy:
            size: 100 MB
        DefaultRolloverStrategy:
          max: 5

  Loggers:
    Root:
      level: INFO
      AppenderRef:
        ref: KafkaServerAppender
    Logger:
      # Kafka logger
      - name: kafka
        level: INFO
      # Kafka org.apache logger
      - name: org.apache.kafka
        level: INFO
##############
# Kafka request logger
##############
      - name: kafka.request.logger
        level: INFO
        additivity: false
        AppenderRef:
          ref: RequestAppender
      # Kafka network RequestChannel$ logger
      - name: kafka.network.RequestChannel$
        level: INFO
        additivity: false
        AppenderRef:
          ref: RequestAppender
      # Uncomment the lines below and change log4j.logger.kafka.network.RequestChannel$ to INFO
      # for additional output related to the handling of requests
      - name: kafka.network.Processor
        level: INFO
        additivity: false
        AppenderRef:
          ref: RequestAppender
#      - name: kafka.server.KafkaApis
#        level: INFO
#        additivity: false
#        AppenderRef:
#          ref: RequestAppender

What I expected from the first one was:

• roll file every 1 min OR if the file is more than 100Mb
• if we have 5 rolled file start to suppress older file
• if a file matching server.log.* have been last modified 15min ago suppress it

What I experienced:

• file are rolled sometimes every minutes yes but sometimes no (can't determine why)
• files are not suppressed at all

what I expected for the second one:

• roll the file every 12 hours OR if the file is more than 100Mb
• if we reach 5 rolled files then delete the older one

What I experienced:

• file are rolled but not deleted

If anyone could point me to understanding why it's falling and where to go that would be awesome.

cheers

[ppkarwasz]

Hi @Shawcs,

There are two problems in your KafkaServerAppender:

• There is no %i specifier in the file pattern (${sys:kafka.logs.dir}/server.%d{yyyy-MM-dd:HH'h':mm'm'}.log). This will cause rollovers triggered by the size of the file to simply override the previous one (or fail, while attempting to do it).
• Your regex for the delete action (server.log.*) does not match any of the archived files.