Intro to Event Stream Analysis (ESA) & Complex Event Processing (ESPER)

Exploring Event Driven Architectures with Esper

https://www.infoq.com/news/2007/05/esper

  • Event stream processing (ESP)
    • monitors streams of event data, analyzing those events for matching conditions and then notifies listeners
  • Complex event processing (CEP)
    • allows the detection of patterns among events

WHAT IS COMPLEX EVENT PROCESSING (CEP)?

Complex Event Processing (CEP), or Event Stream Stream Processing (ESP) are technologies commonly used in Event-Driven systems. These type of systems consume, and react to a stream of event data in real time. Typically these will be things like financial trading, fraud identification and process monitoring systems – where you need to identify, make sense of, and react quickly to emerging patterns in a stream of data events.

KEY COMPONENTS OF A CEP SYSTEM

A CEP system is like your typical database model turned upside down. Whereas a typical database stores data, and runs queries against the data, a CEP data stores queries, and runs data through the queries.

To do this it basically needs:

  • Data – in the form of ‘Events’
  • Queries – using EPL (‘Event Processing Language’)
  • Listeners – code that ‘does something’ if the queries return results

The Esper query language provides a rich syntax allowing complex temporal logic to be expressed, and includes features such as:

  • Event filtering
  • Sliding window and aggregation (count all assets reported in the last 30 seconds)
  • Grouped windows and output rate limiting (get a count per zone of the last 10 minutes per zone)
  • Joins and outer joins (also joins between event streams)
  • Integration with historic or reference data (accessing relational databases)
  • Creation of virtual streams that all statements can access

References

Examples

  • Esper EQL is an object-oriented event stream query language very similar to SQL in its syntax but that significantly differs to be able to deal with sliding window of streams of data.
  • Esper also includes a pattern language that provides for stateful (state-machine) event pattern matching.
  • EQL and patterns can be used alone or can also be combined to express complex temporal logic.
Example 1: Terminal Monitoring/Alerting

Events are as below
BaseTerminalEvent       (Super Event)

  • Checkin, Completed, Cancelled, Status, OutOfOrder, LowPaper    (Inherited Events)
select * from LowPaper Report when you observe an LowPaper Event
select * from LowPaper
select * from OutOfOrder
Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select a,b from pattern

[ every a=LowPaper or every b=OutOfOrder]

Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select * from BaseTerminalEvent
where type = 'LowPaper' or type = 'OutOfOrder'
Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select 'terminal 1 is offline' from pattern
[ every timer:interval(60 sec) ->
  (timer:interval(65 sec) and not Status(term.id = 'T1'))

]

output first every 5 minutes

Detecting the Absence of Status Events

  • Status Event is produced by terminal every 1 minute
  • Detect of this event is not produced by the terminal

Repeat the action for every 60 seconds
we combine this with a not operator to check for absence of Status events. A 65-second interval during which we look for Status events allows 5 seconds to account for a possible delay in transmission or processing:

  • Create a Pattern
    • Frequency: 60 secs
    • Event Criteria
      • No Status event with term.id=’T1′ in a window of 65 seconds
  • We only want to be alerted first time it happens and do not alert for next 5 minutes when this pattern triggers
select count(*) from Checkin.win:time(10 minutes) Report number of Checkin Events during a window of 10 minutes
select type, count(*)
from BaseTerminalEvent.win:time(10 minutes)
group by type
output all every 1 minutes
For a window of last 10 minutes for BaseTerminalEvent

  • Report Event type, Count
  • Group by Event type

Alert every 1 minute and not at each change

Example 2: Tweets Monitoring (https://www.igvita.com/2011/05/27/streamsql-event-processing-with-esper/)
SELECT sum(retweets) from TweetEvent

(retweets >= 10).win:length(5)

find the sum of retweets of last 5 tweets which saw more than 10 retweets

  • You can use min(), max(), sum(), count(), avg()
SELECT timezone, sum(retweets)
from TweetEvent.win:time_batch(10 sec)
group by timezone
number of retweets, grouped by timezone, buffered in 10 second increments
SELECT sum(retweets)
from TweetEvent.win:time(60 sec)
output snapshot every 30 events
Report the sum of retweets
for TweetEvent s of sliding 60 second window,
and emit count every 30 events
SELECT timezone, sum(retweets)

from TweetEvent.win:time_batch(10 sec)

group by timezone

having sum(retweets) > 10

Report total number of retweets by timezone
for TweetEvent s of batch of 10 second window
where the window of TweetEvents grouped by timezone
and report if each window of events has total retweets > 10

side notes
Checkin.win:time(10 minutes) This tells the engine to consider a time window consisting of

only the last 10 minutes of the Checkin event stream.

TweetEvent.win:time_batch(10 sec) buffered in 10 second increments

  • time()  is applied for sliding window of events
  • time_batch() is applied for batch of events
output first every 5 minutes Alert first time when pattern matched and suppress for next 5 minutes
output all every 1 minutes Alert every 1 minute and not at each change
output snapshot every 30 events Alert every 30 events that matched the pattern
Status(term.id = 'T1') Status Event where term.id = ‘T1’
Advertisements

About wikihead

A Seeker. Information Security Professional, Pursuing Life with Ayurveda.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s