Tutorial :How can I convert a log4j timestamp to milliseconds in Perl?


The log4j logs I have contain timestamps in the following format:

2009-05-10 00:48:41,905  

I need to convert it in perl to millseconds since epoch, which in this case would be 124189673005, using the following gawk function. How do I do it in perl?

I have little or no experience in perl, so appreciate if someone can post an entire script that does this

function log4jTimeStampToMillis(log4jts) {      # log4jts is of the form 2009-03-02 20:04:13,474      # extract milliseconds that is after the command      split(log4jts, tsparts, ",");      millis = tsparts[2];        # remove - : from tsstr      tsstr = tsparts[1];      gsub("[-:]", " ", tsstr);      seconds = mktime(tsstr);      print log4jts;      return seconds * 1000 + millis;  }  


Haven't used it, but you might want to check out Time::ParseDate.


Though I almost always tell people to go use one of the many excellent modules from the CPAN for this, most of them do have one major drawback - speed. If you're parsing a large number of log files in real-time, that can sometimes be an issue. In those cases, rolling your own can often be a more suitable solution, but there are many pitfalls and nuances that must be considered and handled properly. Hence the preference for using a known-correct, proven, reliable module written by somebody else. :)

However, before I even considered my advice above, I looked at your code and had converted it to perl in my head... therefore, here is a more-or-less direct conversion of your gawk code into perl. I've tried to write it as simply as possible, so as to highlight some of the more delicate parts of dealing with dates and times in perl by hand.

# import the mktime function from the (standard) POSIX module  use POSIX qw( mktime );    sub log4jTimeStampToMillis {      my ($log4jts, $dst) = @_;        # extract the millisecond field      my ($tsstr, $millis) = split( ',', $log4jts );        # extract values to pass to mktime()      my @mktime_args = reverse split( '[-: ]', $tsstr );        # munge values for posix compatibility (ugh)      $mktime_args[3] -= 1;      $mktime_args[4] -= 1;      $mktime_args[5] -= 1900;      # print Dumper \@mktime_args; ## DEBUG        # convert, make sure to account for daylight savings      my $seconds = mktime( @mktime_args, 0, 0, $dst );        # return that time as milliseconds since the epoch      return $seconds * 1000 + $millis;  }  

One important difference between my code and yours - my log4jTimeStampToMillis subroutine takes two parameters:

  1. the log timestamp string
  2. whether or not that timestamp is using daylight savings time ( 1 for true, 0 for false )

Of course, you could just add code to detect if that time falls in DST or not and adjust automatically, but I was trying to keep it simple. :)

NOTE: If you uncomment the line marked DEBUG, make sure to add "use Data::Dumper;" before that line in your program so it will work.

Here's an example of how you could test that subroutine:

my $milliseconds = log4jTimeStampToMillis( "2009-05-10 00:48:41,905", 1 );      my $seconds = int( $milliseconds / 1000 );  my $local = scalar localtime( $seconds );    print "ms:    $milliseconds\n"; # ms:    1241844521905  print "sec:   $seconds\n";      # sec:   1241844521  print "local: $local\n";        # local: Sat May  9 00:48:41 2009  


You should take advantage of the great DateTime package, specifically use DateTime::Format::Strptime:

use DateTime;  use DateTime::Format::Strptime;    sub log4jTimeStampToMillis {      my $log4jts=shift(@_);        #see package docs for how the pattern parameter works      my $formatter= new DateTime::Format::Strptime(pattern => '%Y-%m-%d %T,%3N');      my $dayObj = $formatter->parse_datetime($log4jts);        return $dayObj->epoch()*1000+$dayObj->millisecond();  }    print log4jTimeStampToMillis('2009-05-10 10:48:41,905')."\n";  #prints my local version of the TS: 1241952521905  

This saves you the pain of figuring out DST yourself (although you'll have to pass your server's TZ to Strptime via the time_zone parameter). It also saves you from dealing with leap everything if it becomes relevant (and I'm sure it will).


SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS");  Date time = dateFormat.parse(log4jts);  long millis = time.getTime();  

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Next Post »