Project Description

The S3Appender is a log4net appender that stores log events in either a MemoryStream or FileStream and sends them to S3 based on time intervals and size thresholds.

This appender is useful for systems living in the cloud -- where servers and application components are dynamically scaling up and down -- where there is no guarantee that a particular box will be up and running. Using this appender ensures that all (or most) of your logging information is saved even if components / servers spin down.

This appender is thread safe.

While not complete, the following describes the basic approach used by the S3Appender:

Append Cycle

  • Logging events are stored to a Stream (either MemoryStream or FileStream)
  • When the Stream reaches its MaxStreamSize or if the timer reaches the m_timeThreshold based on TimeInterval
    • a new Stream is created for storing logging events
    • the old Stream is enqueued to be sent to S3

MemoryStream or FileStream

  • A MemoryStream is the default type of stream used.
  • If the MaxMemoryFootprint is exceeded by all MemoryStreams in use or enqueued, then the appender uses FileStreams until m_currentMemoryFootprint is less than MaxMemoryFootprint

Save To S3 Cycle

  • When the appender is constructed, one or more worker threads are created
  • When there is work to be done, a worker dequeues a Stream from the streamsToSend queue and sends it to S3 using configuration values
    • If the Stream sent is a MemoryStream, then the stream is disposed and m_currentMemoryFootprint is decremented
    • If the Stream sent is a FileStream, then the file is closed and deleted

File Naming

  • S3 Keys are named using this pattern to ensure there are no naming conflicts
  • Using milliseconds and then a semi-random number to name the file ensures that multiple servers running software components that log to the same namespace at nearly the same time will not conflict

string s3Key = S3FileName + 
		        "-" + DateTime.UtcNow.ToString( "yyyy-MM-dd_HH-mm-ss-fff", System.Globalization.CultureInfo.InvariantCulture ) + 
			"-" + String.Format( "{0:0000}", new Random().Next( 9999 ).ToString() + 
			"." + S3FileExtension ); 

Configuration Properties

    <appender name="S3Appender" type="log4net.Appender.S3Appender, S3Appender">

      <MaxStreamSize value="100KB"/>
      <TimeInterval value="60000"/>
      <MaxMemoryFootprint value="1MB"/>

      <S3FileName type="log4net.Util.PatternString" value="S3log"/>
      <S3FileExtension value="log"/>
      <S3BucketName type="log4net.Util.PatternString" value="codeplex/s3appender/testlogs"/>
      <S3AccessKeyID value="#######"/>
      <S3SecretAccessKey value="###########"/>

      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date %level [%thread] %logger %C.%M - %message%newline"/>


Last edited Mar 27, 2010 at 12:58 AM by smitchell360, version 3