Check logfiles for recent entries only
title: 'Check Logfiles For Recent Entries Only'
title: 'Check Logfiles For Recent Entries Only' date: '2015-03-15T09:57:53-05:00' status: publish permalink: /check-logfiles-for-recent-entries-only author: admin excerpt: '' type: post id: 853 category: - Logging - Python tag:  post_format: 
Frequently I have a cron job to check for specific entries in log files but want to avoid being notified of something already checked. For example I want my 10 minute cron job to only look for most recent 10 minute entries.
Here is what I did in python.
from datetime import datetime, timedelta ## Get time right now. ie cron job execution #now = datetime(2015,3,15,8,55,00) now = datetime.now() ## How long back to check. Making it 11 mins because cron runs every 10 mins checkBack = 11 lines =  print "log entries newer than " + now.strftime('%b %d %H:%M:%S') + " minus " + str(checkBack) + " minutes" with open('/var/log/syslog', 'r') as f: for line in f: ## Linux syslog format like this: ## Mar 15 08:50:01 EP45-DS3L postfix/sendmail: fatal ## Brain dead log has no year. So this hack will not work close to year ticking over myDate = str(now.year) + " " + line[:15] ## What about "Mar 1" having double space vs "Mar 15". That will break strptime %d. ## zero pad string position 4 to make %d work? if myDate == " ": myDate = myDate.replace(myDate,"0") lt = datetime.strptime(myDate,'%Y %b %d %H:%M:%S') diff = now - lt if diff.days <= 0: if lt > now - timedelta(minutes=checkBack): # print myDate + " --- diff: " + str(diff) lines.append(line) if lines: # message = '\n'.join(lines) # do some grepping for my specific errors here.. # send message per mail...
Just for reference here is an older test where no year is used. This is doing a string compare but I have not tested this one good enough. Most likely it will fail when month ticks over Apr will not be bigger than Mar. Also what about midnight 23:59 > 00:00?
from datetime import datetime, timedelta now = datetime.now() lookback = timedelta(minutes=5) ## Linux syslog format "Mar 15 07:30:10 ..." ## Probably need to zero pad string position 4 to make %d work? oldest = (now - lookback).strftime('%b %d %H:%M:%S') with open('/var/log/syslog', 'r') as f: for line in f: if line[:15] > oldest: print "entry: " + line[:15] + " --- " + line[16:50]