Intro to Event Stream Analysis (ESA) & Complex Event Processing (ESPER)

Exploring Event Driven Architectures with Esper

https://www.infoq.com/news/2007/05/esper

  • Event stream processing (ESP)
    • monitors streams of event data, analyzing those events for matching conditions and then notifies listeners
  • Complex event processing (CEP)
    • allows the detection of patterns among events

WHAT IS COMPLEX EVENT PROCESSING (CEP)?

Complex Event Processing (CEP), or Event Stream Stream Processing (ESP) are technologies commonly used in Event-Driven systems. These type of systems consume, and react to a stream of event data in real time. Typically these will be things like financial trading, fraud identification and process monitoring systems – where you need to identify, make sense of, and react quickly to emerging patterns in a stream of data events.

KEY COMPONENTS OF A CEP SYSTEM

A CEP system is like your typical database model turned upside down. Whereas a typical database stores data, and runs queries against the data, a CEP data stores queries, and runs data through the queries.

To do this it basically needs:

  • Data – in the form of ‘Events’
  • Queries – using EPL (‘Event Processing Language’)
  • Listeners – code that ‘does something’ if the queries return results

The Esper query language provides a rich syntax allowing complex temporal logic to be expressed, and includes features such as:

  • Event filtering
  • Sliding window and aggregation (count all assets reported in the last 30 seconds)
  • Grouped windows and output rate limiting (get a count per zone of the last 10 minutes per zone)
  • Joins and outer joins (also joins between event streams)
  • Integration with historic or reference data (accessing relational databases)
  • Creation of virtual streams that all statements can access

References

Examples

  • Esper EQL is an object-oriented event stream query language very similar to SQL in its syntax but that significantly differs to be able to deal with sliding window of streams of data.
  • Esper also includes a pattern language that provides for stateful (state-machine) event pattern matching.
  • EQL and patterns can be used alone or can also be combined to express complex temporal logic.
Example 1: Terminal Monitoring/Alerting

Events are as below
BaseTerminalEvent       (Super Event)

  • Checkin, Completed, Cancelled, Status, OutOfOrder, LowPaper    (Inherited Events)
select * from LowPaper Report when you observe an LowPaper Event
select * from LowPaper
select * from OutOfOrder
Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select a,b from pattern

[ every a=LowPaper or every b=OutOfOrder]

Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select * from BaseTerminalEvent
where type = 'LowPaper' or type = 'OutOfOrder'
Report when you observe an LowPaper Event
Report when you observe an OutOfOrder Event
select 'terminal 1 is offline' from pattern
[ every timer:interval(60 sec) ->
  (timer:interval(65 sec) and not Status(term.id = 'T1'))

]

output first every 5 minutes

Detecting the Absence of Status Events

  • Status Event is produced by terminal every 1 minute
  • Detect of this event is not produced by the terminal

Repeat the action for every 60 seconds
we combine this with a not operator to check for absence of Status events. A 65-second interval during which we look for Status events allows 5 seconds to account for a possible delay in transmission or processing:

  • Create a Pattern
    • Frequency: 60 secs
    • Event Criteria
      • No Status event with term.id=’T1′ in a window of 65 seconds
  • We only want to be alerted first time it happens and do not alert for next 5 minutes when this pattern triggers
select count(*) from Checkin.win:time(10 minutes) Report number of Checkin Events during a window of 10 minutes
select type, count(*)
from BaseTerminalEvent.win:time(10 minutes)
group by type
output all every 1 minutes
For a window of last 10 minutes for BaseTerminalEvent

  • Report Event type, Count
  • Group by Event type

Alert every 1 minute and not at each change

Example 2: Tweets Monitoring (https://www.igvita.com/2011/05/27/streamsql-event-processing-with-esper/)
SELECT sum(retweets) from TweetEvent

(retweets >= 10).win:length(5)

find the sum of retweets of last 5 tweets which saw more than 10 retweets

  • You can use min(), max(), sum(), count(), avg()
SELECT timezone, sum(retweets)
from TweetEvent.win:time_batch(10 sec)
group by timezone
number of retweets, grouped by timezone, buffered in 10 second increments
SELECT sum(retweets)
from TweetEvent.win:time(60 sec)
output snapshot every 30 events
Report the sum of retweets
for TweetEvent s of sliding 60 second window,
and emit count every 30 events
SELECT timezone, sum(retweets)

from TweetEvent.win:time_batch(10 sec)

group by timezone

having sum(retweets) > 10

Report total number of retweets by timezone
for TweetEvent s of batch of 10 second window
where the window of TweetEvents grouped by timezone
and report if each window of events has total retweets > 10

side notes
Checkin.win:time(10 minutes) This tells the engine to consider a time window consisting of

only the last 10 minutes of the Checkin event stream.

TweetEvent.win:time_batch(10 sec) buffered in 10 second increments

  • time()  is applied for sliding window of events
  • time_batch() is applied for batch of events
output first every 5 minutes Alert first time when pattern matched and suppress for next 5 minutes
output all every 1 minutes Alert every 1 minute and not at each change
output snapshot every 30 events Alert every 30 events that matched the pattern
Status(term.id = 'T1') Status Event where term.id = ‘T1’
Advertisements

About Uma Mahesh

A Creator/Equilizer. Creator/Equalizers are catalysts for positive, well-organized change. They never settle for the status quo. Instead, they see the opportunity for innovation in the processes that others have long taken for granted. They respect what's already operating, but they can't help but want to improve upon it. Their special combination provides innovation tempered with profound logic. They have incredible discernment. Should their efforts fail, they are unhesitating in accepting responsibility. They don't wallow in self-pity but rather see these missed attempts as critical steps on the path to success.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s