simple-evcorr-users Mailing List for Simple Event Correlator

hi Eli,

>
> Thank you so much Risto. It's been very informative.
>
> I ended up doing it a bit differently but you definitely opened my eyes on more opportunities on how to use SEC.
>
...
> Perhaps there should be a built in construct to doing something like that? Not sure whether others would find it useful or not.
>

With the newly added 'trim' action from SEC-2.9.4, you can actually
"release" the events from counting operations, provided that their
number was too small to reach the threshold. Here is an example
ruleset to illustrate that concept (I did use the threshold 5 and
window of 10 seconds, but you can use any other values).

type=EventGroup
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED
desc=multiple interfaces up/down on host $+{host}
init=create REPORT_$+{host}
count=exists %e REPORT_$+{host}; if %e ( add REPORT_$+{host} $0 )
slide=getwpos %pos 0; trim %pos REPORT_$+{host} %events; cevent
RELEASED 0 %events
action=write - %s; delete REPORT_$+{host}
end=exists %e REPORT_$+{host}; if %e ( copy REPORT_$+{host} %events;
cevent RELEASED 0 %events; delete REPORT_$+{host} )
thresh=5
window=10

type=Single
ptype=RegExp
pattern=.
context=RELEASED
desc=print released event
action=write - Released event: $0

To keep track of all events the counting operation has observed, the
context REPORT_$+{host} is set up by the 'init' field and updated by
the 'count' field. This context will be deleted when the counting
operation reaches the threshold, and therefore the 'count' field
checks for the presence of context with 'exists %e REPORT_$+{host}'
before updating it. A previously recorded event is considered suitable
for release only if it is no longer in the event correlation window,
and is thus no longer contributing towards reaching the threshold.
Since the action in the 'slide' field gets executed after event
correlation window moves forward, the following action list releases
suitable events:

getwpos %pos 0; trim %pos REPORT_$+{host} %events; cevent RELEASED 0 %events

The 'getwpos' action finds the new beginning of the operation's event
correlation window after sliding, and the 'trim' action removes events
outside this window from the REPORT_$+{host}, assigning them to the
action list variable %events. Finally, the 'cevent' action releases
these events as no longer relevant for the running operation.

If the REPORT_$+{host} context still exists when the counting
operation terminates, the threshold was never reached (otherwise the
'action' field would have deleted REPORT_$+{host}). If that is the
case, then the 'end' field takes care of releasing the event(s) still
in REPORT_$+{host}, and finally deletes the REPORT_$+{host} context.

Unlike my previously suggested solution which worked with a fixed
event correlation window, the above solution works with sliding window
based counting which is normally used by SEC event counting
operations. I hope the posted example is useful.

kind regards,
risto

hi all,

SEC-2.9.4 has been released and is available from the SEC home page
and through the following download link:
https://github.com/simple-evcorr/sec/releases/download/2.9.4/sec-2.9.4.tar.gz

The new version adds support for the 'trim' action which allows for
removing events from the context event store based on time.

kind regards,
risto

It's just easier to handle it externally as a unix one liner that will count repeated events and only report the first one with the count number. Doing the same in SEC is a bit too involved.

Eli Kagan
DXC Technology
647-231-6047   |   eli...@dx...

-----Original Message-----
From: Risto Vaarandi <ris...@gm...>
Sent: Friday, January 16, 2026 1:36 PM
To: Kagan, Eli <eli...@dx...>
Cc: Sim...@li...
Subject: Re: [Simple-evcorr-users] multiple events

[You don't often get email from ris...@gm.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

hi Eli,

> I know it's not pretty and I could have separated it into multiple rules but I decided to keep this abomination in one place ;-)
> Perhaps there should be a built in construct to doing something like that? Not sure whether others would find it useful or not.
>
> Another "feature request" would be a simple way of doing event deduplication. I know I can do it with built in means but I ended up deduplicating on the reporting side.
>

Did you mean suppression of repeated events by deduplication?
What about using the SingleWithSuppress rule for that purpose, which
allows reacting to the first event and suppressing repeated instances
of the same event in a given time window?

kind regards,
risto

>
> Thanks again!
>
>
>
> Eli Kagan
> DXC Technology
> 647-231-6047   |   eli...@dx...

hi Eli,

> I know it's not pretty and I could have separated it into multiple rules but I decided to keep this abomination in one place ;-)
> Perhaps there should be a built in construct to doing something like that? Not sure whether others would find it useful or not.
>
> Another "feature request" would be a simple way of doing event deduplication. I know I can do it with built in means but I ended up deduplicating on the reporting side.
>

Did you mean suppression of repeated events by deduplication?
What about using the SingleWithSuppress rule for that purpose, which
allows reacting to the first event and suppressing repeated instances
of the same event in a given time window?

kind regards,
risto

>
> Thanks again!
>
>
>
> Eli Kagan
> DXC Technology
> 647-231-6047   |   eli...@dx...

Thank you so much Risto. It's been very informative.

I ended up doing it a bit differently but you definitely opened my eyes on more opportunities on how to use SEC.

type=Single
ptype=RegExp
pattern=^(?<time>\S+) (?<host>\S+) LINK-(?<type>\S+) (?<if>\S+) \[(?<server>[^:\s]+).*\]
context=!MULTI-LINK-RELEASED-EVENTS
desc=$+{time} $+{server} Multiple LINK-UP/DOWN/BOUNCE events
action=exists %exists MULTI-LINK:$+{server}; \
       if %exists ( \
           getsize %n MULTI-LINK:$+{server}; \
           lcall %under_threshold %n -> ( sub { $_[0] <= 2 } ); \
           if %under_threshold ( \
               add MULTI-LINK:$+{server} $0; \
           ); \
           lcall %at_threshold %n -> ( sub { $_[0] == 2 } ); \
           if %at_threshold ( \
               event 0 %s; \
               set MULTI-LINK:$+{server} - none; \
               create MULTI-LINK-REMOVER:$+{server} 15 ( obsolete MULTI-LINK:$+{server} ); \
           ); \
           set MULTI-LINK:$+{server} 5; \
       ) else ( \
           create MULTI-LINK:$+{server} 5 ( copy _THIS %events; cevent MULTI-LINK-RELEASED-EVENTS 0 %events ); \
           add MULTI-LINK:$+{server} $0; \
       );

I know it's not pretty and I could have separated it into multiple rules but I decided to keep this abomination in one place ;-)
Perhaps there should be a built in construct to doing something like that? Not sure whether others would find it useful or not.

Another "feature request" would be a simple way of doing event deduplication. I know I can do it with built in means but I ended up deduplicating on the reporting side.


Thanks again!



Eli Kagan
DXC Technology
647-231-6047   |   eli...@dx...

-----Original Message-----
From: Risto Vaarandi <ris...@gm...>
Sent: Tuesday, January 13, 2026 8:16 AM
To: Kagan, Eli <eli...@dx...>
Cc: Sim...@li...
Subject: Re: [Simple-evcorr-users] multiple events

[You don't often get email from ris...@gm.... Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

hi Eli,

As a followup to your question, the previously posted solution does
not handle one corner case. When only one event appears in the window
of 60 seconds, the EventGroup operation is not able to move the window
forward when it expires, because there are no valid events left to
continue. As a consequence, the 'slide' action would not be executed
in that case, but rather the 'end' action. To handle this corner case,
the 'end' action would need to check if there is just one event in the
EVENTS_$+{host} context and if that is the case, release this event.
That would make the 'end' action more complicated, for example:

end=getsize %n EVENTS_$+{host}; lcall %release %n -> ( sub { $_[0] ==
1 } ); if %release (copy EVENTS_$+{host} %events; cevent
RELEASED_EVENT 0 %events); delete EVENTS_$+{host}

A less convoluted solution would be to handle the scenario with the
event aggregation context itself. For example, you could configure it
to release all aggregated events after its 60 second lifetime expires,
and cancel the release if the context receives three events before
expiration. Here is a relevant example:

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED_EVENT && !EVENTS_$+{host}
continue=TakeNext
desc=create an event aggregation context
action=create EVENTS_$+{host} 60 \
       ( copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events )

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED_EVENT
desc=add events to an aggregation context
action=add EVENTS_$+{host} $0; getsize %n EVENTS_$+{host}; \
       lcall %threshold %n -> ( sub { $_[0] == 3 } ); \
       if %threshold ( event 0 multiple interfaces up/down on host $+{host}; \
                       set EVENTS_$+{host} - none )

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
desc=demo rule for printing released events
action=write - Released event: $0

The first rule sets up an event aggregation context EVENTS_$+{host}
with a lifetime of 60 seconds if it does not exist already. On the
expiration of this context, all collected events are released with:
copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events

The second rule updates the EVENTS_$+{host} context with new events,
checking after each update if three events have been received:
getsize %n EVENTS_$+{host}; lcall %threshold %n -> ( sub { $_[0] == 3 } )
If that is the case, a new event "multiple interfaces up/down on host
$+{host}" is created, and in order to prevent the release of collected
events, the following action is used:
set EVENTS_$+{host} - none
This 'set' action does not modify the expiration time of the
EVENTS_$+{host} context and makes sure that when the context expires,
no action would be executed (action 'none' is no-op).

kind regards,
risto

>
> hi Eli,
>
> you have asked a very interesting question indeed. I hope I have
> understood your goal correctly, and below is a rule which tries to
> address it. The rule relies on the following observation -- if the
> event counting operation does not reach the threshold by the end of
> its window, the window must slide forward. Because we want to release
> the collected events at this particular moment, an action must be
> executed when the window sliding occurs. Unfortunately, the
> SingleWithThreshold rule does not have that functionality, but the
> EventGroup rule (generalization of SingleWithThreshold) does. Here is
> an example which utilizes EventGroup rule and uses the
> 'EVENTS_$+{host}' context for recording all events observed by the
> EventGroup event counting operation for the given host $+{host}:
>
> type=EventGroup
> ptype=RegExp
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
> context=!RELEASED_EVENT
> init=create EVENTS_$+{host}
> count=add EVENTS_$+{host} $0
> end=delete EVENTS_$+{host}
> slide=copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events;
> reset 0; delete EVENTS_$+{host}
> desc=multiple interfaces up/down on host $+{host}
> action=event 0 %s
> window=60
> thresh=3
>
> type=Single
> ptype=RegExp
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
> desc=demo rule for printing released events
> action=write - Released event: $0
>

hi Eli,

As a followup to your question, the previously posted solution does
not handle one corner case. When only one event appears in the window
of 60 seconds, the EventGroup operation is not able to move the window
forward when it expires, because there are no valid events left to
continue. As a consequence, the 'slide' action would not be executed
in that case, but rather the 'end' action. To handle this corner case,
the 'end' action would need to check if there is just one event in the
EVENTS_$+{host} context and if that is the case, release this event.
That would make the 'end' action more complicated, for example:

end=getsize %n EVENTS_$+{host}; lcall %release %n -> ( sub { $_[0] ==
1 } ); if %release (copy EVENTS_$+{host} %events; cevent
RELEASED_EVENT 0 %events); delete EVENTS_$+{host}

A less convoluted solution would be to handle the scenario with the
event aggregation context itself. For example, you could configure it
to release all aggregated events after its 60 second lifetime expires,
and cancel the release if the context receives three events before
expiration. Here is a relevant example:

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED_EVENT && !EVENTS_$+{host}
continue=TakeNext
desc=create an event aggregation context
action=create EVENTS_$+{host} 60 \
       ( copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events )

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED_EVENT
desc=add events to an aggregation context
action=add EVENTS_$+{host} $0; getsize %n EVENTS_$+{host}; \
       lcall %threshold %n -> ( sub { $_[0] == 3 } ); \
       if %threshold ( event 0 multiple interfaces up/down on host $+{host}; \
                       set EVENTS_$+{host} - none )

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
desc=demo rule for printing released events
action=write - Released event: $0

The first rule sets up an event aggregation context EVENTS_$+{host}
with a lifetime of 60 seconds if it does not exist already. On the
expiration of this context, all collected events are released with:
copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events

The second rule updates the EVENTS_$+{host} context with new events,
checking after each update if three events have been received:
getsize %n EVENTS_$+{host}; lcall %threshold %n -> ( sub { $_[0] == 3 } )
If that is the case, a new event "multiple interfaces up/down on host
$+{host}" is created, and in order to prevent the release of collected
events, the following action is used:
set EVENTS_$+{host} - none
This 'set' action does not modify the expiration time of the
EVENTS_$+{host} context and makes sure that when the context expires,
no action would be executed (action 'none' is no-op).

kind regards,
risto

>
> hi Eli,
>
> you have asked a very interesting question indeed. I hope I have
> understood your goal correctly, and below is a rule which tries to
> address it. The rule relies on the following observation -- if the
> event counting operation does not reach the threshold by the end of
> its window, the window must slide forward. Because we want to release
> the collected events at this particular moment, an action must be
> executed when the window sliding occurs. Unfortunately, the
> SingleWithThreshold rule does not have that functionality, but the
> EventGroup rule (generalization of SingleWithThreshold) does. Here is
> an example which utilizes EventGroup rule and uses the
> 'EVENTS_$+{host}' context for recording all events observed by the
> EventGroup event counting operation for the given host $+{host}:
>
> type=EventGroup
> ptype=RegExp
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
> context=!RELEASED_EVENT
> init=create EVENTS_$+{host}
> count=add EVENTS_$+{host} $0
> end=delete EVENTS_$+{host}
> slide=copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events;
> reset 0; delete EVENTS_$+{host}
> desc=multiple interfaces up/down on host $+{host}
> action=event 0 %s
> window=60
> thresh=3
>
> type=Single
> ptype=RegExp
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
> desc=demo rule for printing released events
> action=write - Released event: $0
>

hi Eli,

you have asked a very interesting question indeed. I hope I have
understood your goal correctly, and below is a rule which tries to
address it. The rule relies on the following observation -- if the
event counting operation does not reach the threshold by the end of
its window, the window must slide forward. Because we want to release
the collected events at this particular moment, an action must be
executed when the window sliding occurs. Unfortunately, the
SingleWithThreshold rule does not have that functionality, but the
EventGroup rule (generalization of SingleWithThreshold) does. Here is
an example which utilizes EventGroup rule and uses the
'EVENTS_$+{host}' context for recording all events observed by the
EventGroup event counting operation for the given host $+{host}:

type=EventGroup
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
context=!RELEASED_EVENT
init=create EVENTS_$+{host}
count=add EVENTS_$+{host} $0
end=delete EVENTS_$+{host}
slide=copy EVENTS_$+{host} %events; cevent RELEASED_EVENT 0 %events;
reset 0; delete EVENTS_$+{host}
desc=multiple interfaces up/down on host $+{host}
action=event 0 %s
window=60
thresh=3

type=Single
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
desc=demo rule for printing released events
action=write - Released event: $0

Let me explain the EventGroup rule step by step. The 'init' field of
the rule will set up the event recording context EVENTS_$+{host}, and
the 'count' field of the rule makes sure that every matching event
gets recorded to this context. The 'end' field of the rule will make
sure that the EVENTS_$+{host} context gets dropped from memory when
the event counting operation terminates in a normal way (that is, it
is *not* terminated with the 'reset' action).

The 'slide' field of the rule is a bit more complicated and contains 4
actions. These actions get executed when the event counting operation
has reached the end of its 60 second window and does *not* have enough
events to execute its regular output action (given with the 'action'
field). The entire purpose of the 'slide' field is to "release" events
which the event counting operation has collected -- in other words,
the events which have been recorded to the EVENTS_$+{host} context.

To achieve that goal, the first action of the 'slide' field is 'copy
EVENTS_$+{host} %events', which saves all events from the context to
the action list variable %events. The next action is 'cevent
RELEASED_EVENT 0 %events', which creates synthetic events from the
potentially multiline variable %events, so that each line becomes a
separate synthetic event. In addition, the 'cevent' action makes sure
that RELEASED_EVENT internal context is set up during the processing
of each synthetic event, so that these events could be distinguished
from regular input events. That is important, because otherwise the
EventGroup rule would match these synthetic events, and to prevent
that, the EventGroup rule has the following field:
context=!RELEASED_EVENT

After the first two actions of the 'slide' field have been executed,
the 'reset 0' action will terminate the counting operation, because
releasing the collected events makes this operation pointless. And
because killing the operation with 'reset' action does not execute the
action on the 'end' field, we need the final 'delete EVENTS_$+{host}'
action on the 'slide' field, so that the context of the terminated
operation would be properly dropped from memory.

To test the above ruleset of two rules, you need the --intcontext
command line option, which enables internal contexts for input sources
(otherwise the context RELEASED_EVENT would not be set up). Here is an
example test:

sec --conf=eli.sec --input=- --intcontexts
SEC (Simple Event Correlator) 2.9.2
Reading configuration from eli.sec
2 rules loaded from eli.sec
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Opening input file -
Interactive process, SIGINT can't be used for changing the logging level
Host1 int1 up    <--- input from keyboard
Creating context 'EVENTS_Host1'
Adding event(s) 'Host1 int1 up' to context 'EVENTS_Host1'
Host1 int2 up     <--- input from keyboard
Adding event(s) 'Host1 int2 up' to context 'EVENTS_Host1'
Copying context 'EVENTS_Host1' to variable '%events'     <--- here we
have reached the end of the 60-second window...
Variable '%events' set to 'Host1 int1 up
Host1 int2 up'
Creating event 'Host1 int1 up' with context 'RELEASED_EVENT'
Creating event 'Host1 int2 up' with context 'RELEASED_EVENT'
Terminating event correlation operation 'eli.sec | 0 | multiple
interfaces up/down on host Host1'
Deleting context 'EVENTS_Host1'
Context 'EVENTS_Host1' deleted
Writing event 'Released event: Host1 int1 up' to file '-'     <--- and
here the 2nd demo rule starts to match released events
Released event: Host1 int1 up
Writing event 'Released event: Host1 int2 up' to file '-'
Released event: Host1 int2 up


I hope that the above example ruleset is useful.

kind regards,
risto

Kontakt Kagan, Eli via Simple-evcorr-users
(<sim...@li...>) kirjutas kuupäeval E,
12. jaanuar 2026 kell 19:41:
>
> Hi all,
>
>
>
> I’m struggling with how `SingleWithThreshold` handles events that do not reach the threshold, and I’m hoping someone can point me in the right direction.
>
>
>
> My goal is to correlate multiple interface up/down events into a single aggregated event.
>
> I created a `SingleWithThreshold` rule that looks for three interface up/down events on the same host within one minute and generates a synthetic event such as: “multiple interfaces up/down on host H”
>
> That part works fine.
>
>
>
> The problem is what happens to individual interface events that do not trigger the threshold. These events are still consumed by the `SingleWithThreshold` rule and never reach any other rules.
>
> I tried using a context followed by a `Suppress` rule, but then I end up with two individual up/down events followed by the aggregated event, which is not what I want.
>
>
>
> What I would like is this behavior:
>
> Input:
>
> Host1 int1 up
>
> Host1 int2 up
>
> Host1 int3 up
>
> Host2 int1 up
>
>
>
> Output:
>
> Multiple interfaces up/down on Host1
>
> Host2 int1 up
>
>
>
> In other words:
>
> * If three up/down events occur on the same host within one minute, emit only the aggregated event and suppress the individual ones.
>
> * If fewer than three occur, allow the individual events to be processed by other rules normally.
>
>
>
> Conceptually, what I am trying to achieve is if a `SingleWithThreshold` rule does not reach its threshold, it should “release” the events it has ingested so far so they can be processed by other rules.
>
>
>
> Below is my (failed) attempt:
>
> type=SingleWithThreshold
>
> ptype=RegExp
>
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
>
> desc=multiple interfaces up/down on host $+{host}
>
> action=event 0 %s; create SUPPRESS-IF-ON-HOST:$+{host} 60
>
> window=60
>
> thresh=2
>
> continue=TakeNext
>
>
>
> type=Suppress
>
> ptype=RegExp
>
> pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
>
> desc=$0
>
> context=SUPPRESS-IF-ON-HOST:$+{host}
>
>
>
>
>
> Any guidance on how to implement this correctly would be greatly appreciated.
>
>
>
> Thanks,
>
> Eli
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Hi all,

I'm struggling with how `SingleWithThreshold` handles events that do not reach the threshold, and I'm hoping someone can point me in the right direction.

My goal is to correlate multiple interface up/down events into a single aggregated event.
I created a `SingleWithThreshold` rule that looks for three interface up/down events on the same host within one minute and generates a synthetic event such as: "multiple interfaces up/down on host H"
That part works fine.

The problem is what happens to individual interface events that do not trigger the threshold. These events are still consumed by the `SingleWithThreshold` rule and never reach any other rules.
I tried using a context followed by a `Suppress` rule, but then I end up with two individual up/down events followed by the aggregated event, which is not what I want.

What I would like is this behavior:
Input:
Host1 int1 up
Host1 int2 up
Host1 int3 up
Host2 int1 up

Output:
Multiple interfaces up/down on Host1
Host2 int1 up

In other words:
* If three up/down events occur on the same host within one minute, emit only the aggregated event and suppress the individual ones.
* If fewer than three occur, allow the individual events to be processed by other rules normally.

Conceptually, what I am trying to achieve is if a `SingleWithThreshold` rule does not reach its threshold, it should "release" the events it has ingested so far so they can be processed by other rules.

Below is my (failed) attempt:
type=SingleWithThreshold
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
desc=multiple interfaces up/down on host $+{host}
action=event 0 %s; create SUPPRESS-IF-ON-HOST:$+{host} 60
window=60
thresh=2
continue=TakeNext

type=Suppress
ptype=RegExp
pattern=^(?<host>\S+) (?<if>\S+) (?:up|down)$
desc=$0
context=SUPPRESS-IF-ON-HOST:$+{host}


Any guidance on how to implement this correctly would be greatly appreciated.

Thanks,
Eli

hi Rajeev,

the scheme you have proposed (taking timestamps from an input file and
using them to adjust an artificial clock) does not work with SEC as it
has been explained in the documentation and this tutorial:
https://simple-evcorr.github.io/SEC-tutorial.pdf. Implementing an
artificial clock is not easy and involves a number of advanced issues.
For example, the internal event correlation state is not only changing
on the appearance of new input events, but also on the expiration of
various timers (created and adjusted dynamically at runtime, and thus
difficult to track). To make things worse, timestamps are not always
increasing in log files (e.g., because of multi-threading of logging
applications, but also issues like unsynchronized system clocks,
etc.). As a result, artificial time will not always move forward, but
can also move backwards. Therefore, to implement such an artificial
clock would probably require a separate project, since it is less
connected to real-time event correlation.

Also, can you post further questions to the mailing list as instructed
in the SEC home page? (the list can be found here:
https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/)

kind regards,
risto


Kontakt Rajeev Raut (R) (<Ra...@op...>) kirjutas kuupäeval
R, 24. jaanuar 2025 kell 16:15:
>
> Hi Risto,
>
>
>
> I am typical support guy .. and have written few Domain Specific perl based regex’s to get my job done.. in near real time (automations).
>
>
>
> I find it interesting that your tool sec might come handy in my efforts to tackle further challenges w/o coding too much going forward.
>
>
>
> My problem is:
>
>
>
> I setup data in a file where say I keep some date/time part as data (that can act like actual-time of the events) and the rest of the line.. as dummy events to be handled by sec. I want sec to take this datee/time as if that is the wall clock ? is it possible ? reason to do is – this way we can test far more scenarios w/o really waiting for actual wall-clock. This will speedup doing test-cases.
>
>
>
> Let me know.
>
>
>
> PS: Please keep related-work-groups (CBS...@te...) in cc-list for better response times, as we may get side-tracked due to other tasks.
>
> Stay safe.
>
>
>
> Regards
> Rajeev Raut
> Desk: +27 12 311 4086
> Cell: 081 569 3343 / 061 902 0808
> Meet your future today
>
>
>
> ===== This e-mail and its contents are subject to the Openserve (Pty) Ltd E-mail legal notice https://connect.openserve.co.za/assets/documents/LegalEmail/OpenserveEMailLegalNotice.pdf =====

hi all,

SEC-2.9.3 has been released which is available from SEC home page and
through the following download link:
https://github.com/simple-evcorr/sec/releases/download/2.9.3/sec-2.9.3.tar.gz

The new version adds support for built-in action list variables
%.chr32, ..., %.chr255 that are set to ASCII 32..255 characters. These
variables are useful for various encoding tasks in action lists -- for
example, the sequence %.chr195%.chr132 represents the A-umlaut
character in UTF-8 encoding.

kind regards,
risto

hi all,

during the early days of SEC, its home page was located at
http://kodu.neti.ee/~risto/sec. That site is a part of a web portal owned
by Telia Estonia (www.telia.ee), and Telia has decided to close that portal
on June 30.

Note that http://kodu.neti.ee/~risto/sec has not been actively used since
2009, and when accessing that site, you will be redirected to the current
SEC home page at Github: https://simple-evcorr.github.io. Although the old
site is not likely to be used by end users, there is a slight chance the
old link is still present in some web pages and documentation which were
originally created before 2009. That email serves as a notification to
update the SEC home page address if you have been using the old link.

Finally, we thank Telia Estonia for keeping the kodu.neti.ee portal up and
running as a free service for 25+ years :)

kind regards,
risto

Thanks for the detailed explanation. Appreciate your work Risto.

Tom Damon
LogZilla

From: Risto Vaarandi <ris...@gm...>
Date: Thursday, April 11, 2024 at 5:38 PM
To: Tom Damon <to...@lo...>
Cc: sim...@li... <sim...@li...>
Subject: Re: [Simple-evcorr-users] Problem with action2
hi Tom,

the PairWithWindow rule works as follows (see also the documentation of the PairWithWindow rule in https://simple-evcorr.github.io/man.html#lbAP):

step1) if the incoming event matches the pattern defined with the 'pattern' field, the rule either (a) starts a new event correlation operation if it does not exist yet, or (b) if the operation exists, the rule sends the event to the operation which consumes the event silently.
step2) if the incoming event does not match the pattern defined with the 'pattern' field, the event is processed by all event correlation operations started by the rule, and the operations try to match this event against their 'pattern2' patterns. If any of the patterns matches, corresponding 'action2' of the relevant operation is executed.

Given the scheme described above, if the 'pattern' field matches all events that 'pattern2' matches, all events are handled during step1 and no event will reach step2. You are seeing this behavior, since both patterns are identical in your rule definition. To fix that issue, you need to make the 'pattern' and 'pattern2' fields different enough, so that the first pattern would only match the specific event which should start the event correlation operation, whereas the second pattern would only match the event which should end the operation.

kind regards,
risto

Kontakt Tom Damon via Simple-evcorr-users (<sim...@li...<mailto:sim...@li...>>) kirjutas kuupäeval N, 11. aprill 2024 kell 23:34:
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla
_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Ah, didn’t catch that. Thank you!

Tom Damon
LogZilla

From: Frazier, Jon <Jon...@gm...>
Date: Thursday, April 11, 2024 at 4:41 PM
To: Tom Damon <to...@lo...>, sim...@li... <sim...@li...>
Subject: RE: [Simple-evcorr-users] Problem with action2
Could it be because the two patterns are identical thus you don’t have a “recovered” string for the second to match on.

I haven’t dug into this in years so I may be mistaken.


Regards,
Jon Frazier

From: Tom Damon via Simple-evcorr-users <sim...@li...>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

hi Tom,

the PairWithWindow rule works as follows (see also the documentation of the
PairWithWindow rule in https://simple-evcorr.github.io/man.html#lbAP):

step1) if the incoming event matches the pattern defined with the 'pattern'
field, the rule either (a) starts a new event correlation operation if it
does not exist yet, or (b) if the operation exists, the rule sends the
event to the operation which consumes the event silently.
step2) if the incoming event does not match the pattern defined with the
'pattern' field, the event is processed by all event correlation operations
started by the rule, and the operations try to match this event against
their 'pattern2' patterns. If any of the patterns matches, corresponding
'action2' of the relevant operation is executed.

Given the scheme described above, if the 'pattern' field matches all events
that 'pattern2' matches, all events are handled during step1 and no event
will reach step2. You are seeing this behavior, since both patterns are
identical in your rule definition. To fix that issue, you need to make the
'pattern' and 'pattern2' fields different enough, so that the first pattern
would only match the specific event which should start the event
correlation operation, whereas the second pattern would only match the
event which should end the operation.

kind regards,
risto

Kontakt Tom Damon via Simple-evcorr-users (<
sim...@li...>) kirjutas kuupäeval N, 11.
aprill 2024 kell 23:34:

> Hello list,
>
>   I’m trying to get this rule working.  The action works, but action2 does
> not. What am I missing?
>
>
>
> type=PairWithWindow
>
> ptype=regexp
>
> pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
>
> desc=(WARNING) $1 is $3 from $2
>
> action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
>
> ptype2=regexp
>
> pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
>
> desc2=(NOTICE) You seeing this means, we have seen a recovery event.
>
> action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
>
> window=5
>
>
>
> Thanks,
>
> Tom Damon
>
> LogZilla
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

No problem.


Regards,
Jon Frazier


From: Tom Damon <to...@lo...>
Sent: Thursday, April 11, 2024 4:00 PM
To: Frazier, Jon <Jon...@gm...>; sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Ah, didn’t catch that. Thank you!

Tom Damon
LogZilla

From: Frazier, Jon <Jon...@gm...<mailto:Jon...@gm...>>
Date: Thursday, April 11, 2024 at 4:41 PM
To: Tom Damon <to...@lo...<mailto:to...@lo...>>, sim...@li...<mailto:sim...@li...> <sim...@li...<mailto:sim...@li...>>
Subject: RE: [Simple-evcorr-users] Problem with action2
Could it be because the two patterns are identical thus you don’t have a “recovered” string for the second to match on.

I haven’t dug into this in years so I may be mistaken.


Regards,
Jon Frazier

From: Tom Damon via Simple-evcorr-users <sim...@li...<mailto:sim...@li...>>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...<mailto:sim...@li...>
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

Could it be because the two patterns are identical thus you don't have a "recovered" string for the second to match on.

I haven't dug into this in years so I may be mistaken.


Regards,
Jon Frazier


From: Tom Damon via Simple-evcorr-users <sim...@li...>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I'm trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

hi all,

SEC tutorial has been updated and you can access the new version at
https://simple-evcorr.github.io/SEC-tutorial.pdf.

Many thanks to Jim Van Meggelen for suggestions on how to improve the
tutorial!

Happy Boxing Day to all SEC users :)

risto

Risto, 

That's important advice from a performance perspective, so thank you for that. I would have missed it and suffered as a result. 

If we decide to run SEC like a daemon, as the data is generated (which we might do), then all that stale data would have definitely caused us problems (the callid DOES get re-used, as you surmised). I'm sure I can figure out how to put a handler in to drop everything relating to a call once it's done. I'll do some reading (I always learn new things as I re-read the docs, so that's time well spent). 

I like the option to use Perl to handle the counter; I'll see how the performance does and decide based on that. 

Thanks as always for the help. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Sunday, 24 September, 2023 04:35:50
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,
> many thanks for the clarification! I have a better understanding now about the
> nature of the problem.
> Because you are doing offline processing of an already existing log file,
> removing old contexts or old hash table keys (recommendation from the end of my
> post) might not be so important, provided that call IDs are not reused for
> different calls in the processed log file, and the number of call IDs is not
> very large. That would keep the solution a bit more simple.
> kind regards,
> risto

> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
> jim...@cl... ] >) kirjutas kuupäeval L, 23. september 2023
> kell 22:11:

>> Risto,

>> I cannot change the timestamps, as they are coded in the files I'm working with
>> (old data). Going forward, I also cannot guarantee that I will have control
>> over the source data, and it is likely that it will have a similar granularity.
>> Finally, there's no need for that level of granularity downstream; it is only
>> necessary that two events with the same timestamp are sequenced so that it is
>> clear which happened first.

>> It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, I
>> am almost certain to have to introduce kludge somewhere else, which is worse
>> (this way I only have SEC to maintain, rather than additional technology).

>> This way, I can feed relevant log files to SEC from any source, and trust that
>> I'll be able to handle them, so long as they follow standard Asterisk log
>> format.

>> Thanks again for your generous advice.

>> Jim

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

>>> From: "Risto Vaarandi" < [ mailto:ris...@gm... |
>>> ris...@gm... ] >
>>> To: "Jim Van Meggelen" < [ mailto:jim...@cl... |
>>> jim...@cl... ] >
>>> Cc: "simple-evcorr-users" < [ mailto:sim...@li... |
>>> sim...@li... ] >
>>> Sent: Saturday, 23 September, 2023 03:51:02
>>> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

>>> hi Jim,
>>> the solutions from my previous post were provided without knowing the root cause
>>> of the issue. If the main reason for the problem is the nature of the
>>> timestamps in the logs (that is, they are provided with an accuracy of a
>>> second), I would recommend using a different logging scheme with
>>> high-resolution timestamps. For example, as David has mentioned in his post,
>>> using syslog servers with RFC5424-based syslog protocol ( [
>>> https://datatracker.ietf.org/doc/html/rfc5424 |
>>> https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps
>>> out-of-the-box. I am not sure whether you can change the logging scheme for
>>> these particular application messages, but if you can, these timestamps are
>>> highly useful for many purposes like the investigation of past system faults
>>> and security incidents.
>>> kind regards,
>>> risto

>>>> hi Jim,

>>>> let me provide some suggestions how to accomplish this task. First, you could
>>>> utilize the context based approach that you have described in your post. In
>>>> that case, the numeral that has been retrieved from the context with the 'pop'
>>>> action needs to be incremented, and you can do it with the 'lcall' action that
>>>> invokes the relevant Perl code. Here is one example (I have added several of
>>>> extra actions into the rule for initializing the context):

>>>> type=single
>>>> ptype=regexp
>>>> pattern=call_id (\S+)
>>>> desc=call id $1
>>>> action=exists %e call_uniqueid_seq_$1; \
>>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>>>> pop call_uniqueid_seq_$1 %sequence_num; \
>>>> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>>>> add call_uniqueid_seq_$1 %sequence_num

>>>> The first two actions check if the context call_uniqueid_seq_* for the given
>>>> call exists, and if not, 0 is written into that context to initialize it:

>>>> exists %e call_uniqueid_seq_$1; \
>>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

>>>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
>>>> identical to the actions from your email.

>>>> The 'lcall' action (4th action in the above rule) increments the counter value
>>>> retrieved from the context by executing the following Perl one-line function:
>>>> sub { $_[0] + 1 }
>>>> That function takes the value of its parameter (that is, %sequence_num) and
>>>> returns an incremented value which is stored back into %sequence_num. Note that
>>>> %sequence_num appears twice in the 'lcall' action, since it acts both as an
>>>> input parameter for the Perl function and as the variable where the return
>>>> value from the function is stored.

>>>> There is also an alternative way for addressing the same assignment. This
>>>> approach involves keeping the counter for each call ID not in a SEC context,
>>>> but rather in a Perl hash table. That approach is more efficient, since the
>>>> counter value is not copied back and forth between the Perl function and the
>>>> SEC context each time it needs to be incremented. Instead, the counter value
>>>> stays in a Perl data structure during these increments. Here is a relevant rule
>>>> example:

>>>> type=single
>>>> ptype=regexp
>>>> pattern=call_id (\S+)
>>>> desc=call id $1
>>>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>>>> write - the new value for counter for call $1 is %sequence_num

>>>> The above example involves the use of a Perl hash (associative array) 'calls',
>>>> where the call ID serves as a key into the hash. For each key (call ID), the
>>>> hash stores the counter for that call ID. Each time the function
>>>> sub { ++$calls{$_[0]} }
>>>> gets called, the counter for the relevant call ID (the value of $1) is
>>>> incremented in the hash 'calls', and the result is stored to %sequence_num.

>>>> If you just want to retrieve the current value of the counter for some call ID
>>>> (the value of $1) without incrementing it, you can simply invoke the following
>>>> action:

>>>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

>>>> With both the context-based and Perl-hash-based approach, one needs to think
>>>> about dropping stale contexts (or stale hash keys) for calls that are no longer
>>>> relevant. With contexts, you can remove them with the 'delete' action. In the
>>>> case you use the Perl hash for storing the counters, you can use the following
>>>> 'lcall' action for deleting a key from the hash 'calls':

>>>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

>>>> I hope the above examples are useful.

>>>> kind regards,
>>>> risto

>>>> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
>>>> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
>>>> kell 19:50:

>>>>> I am using SEC for an atypical use case, where rather than using log events to
>>>>> correlate events for things like security events, I am using it to parse
>>>>> through Asterisk logs in order to capture a sequence of phone call events (call
>>>>> comes in, auto attendant answers, user selects a digit, call is transferred,
>>>>> etc). I have SEC write each to a CSV file, which can then be processed
>>>>> downstream for reporting (for example, I produced a Sankey chart using this
>>>>> data).

>>>>> SEC has proven fantastic for this, however one minor issue has been that the
>>>>> timestamps in the logs are not granular to smaller than a second, so it's
>>>>> possible for two or more events to occur within the same second. Generally this
>>>>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>>>>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>>>>> So, what I've been trying to figure out is how to store a counter that is tied
>>>>> to the uniqueid of the call, and increment that with each event. I figured I'd
>>>>> be able to do this by storing an integer in the event store of a context (tied
>>>>> to the uniqueid for that call). I can then increment it as the various log
>>>>> lines of the call are processed in turn.

hi Jim,
many thanks for the clarification! I have a better understanding now about
the nature of the problem.
Because you are doing offline processing of an already existing log file,
removing old contexts or old hash table keys (recommendation from the end
of my post) might not be so important, provided that call IDs are not
reused for different calls in the processed log file, and the number of
call IDs is not very large. That would keep the solution a bit more simple.
kind regards,
risto

Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
kuupäeval L, 23. september 2023 kell 22:11:

> Risto,
>
> I cannot change the timestamps, as they are coded in the files I'm working
> with (old data). Going forward, I also cannot guarantee that I will have
> control over the source data, and it is likely that it will have a similar
> granularity. Finally, there's no need for that level of granularity
> downstream; it is only necessary that two events with the same timestamp
> are sequenced so that it is clear which happened first.
>
> It does seem a bit kludgy, but if I try to reduce the kludge in my SEC
> file, I am almost certain to have to introduce kludge somewhere else, which
> is worse (this way I only have SEC to maintain, rather than additional
> technology).
>
> This way, I can feed relevant log files to SEC from any source, and trust
> that I'll be able to handle them, so long as they follow standard Asterisk
> log format.
>
> Thanks again for your generous advice.
>
> Jim
>
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"Jim Van Meggelen" <jim...@cl...>
> *Cc: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Saturday, 23 September, 2023 03:51:02
> *Subject: *Re: [Simple-evcorr-users] Storing a sequence counter in a
> context
>
> hi Jim,
> the solutions from my previous post were provided without knowing the root
> cause of the issue. If the main reason for the problem is the nature of the
> timestamps in the logs (that is, they are provided with an accuracy of a
> second), I would recommend using a different logging scheme with
> high-resolution timestamps. For example, as David has mentioned in his
> post, using syslog servers with RFC5424-based syslog protocol (
> https://datatracker.ietf.org/doc/html/rfc5424) provides such timestamps
> out-of-the-box. I am not sure whether you can change the logging scheme for
> these particular application messages, but if you can, these timestamps are
> highly useful for many purposes like the investigation of past system
> faults and security incidents.
> kind regards,
> risto
>
>
> hi Jim,
>>
>> let me provide some suggestions how to accomplish this task. First, you
>> could utilize the context based approach that you have described in your
>> post. In that case, the numeral that has been retrieved from the context
>> with the 'pop' action needs to be incremented, and you can do it with the
>> 'lcall' action that invokes the relevant Perl code. Here is one example (I
>> have added several of extra actions into the rule for initializing the
>> context):
>>
>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=exists %e call_uniqueid_seq_$1; \
>>        if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>>        pop call_uniqueid_seq_$1 %sequence_num; \
>>        lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>>        add call_uniqueid_seq_$1 %sequence_num
>>
>> The first two actions check if the context call_uniqueid_seq_* for the
>> given call exists, and if not, 0 is written into that context to initialize
>> it:
>>
>> exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);
>>
>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule)
>> are identical to the actions from your email.
>>
>> The 'lcall' action (4th action in the above rule) increments the counter
>> value retrieved from the context by executing the following Perl one-line
>> function:
>> sub { $_[0] + 1 }
>> That function takes the value of its parameter (that is, %sequence_num)
>> and returns an incremented value which is stored back into %sequence_num.
>> Note that %sequence_num appears twice in the 'lcall' action, since it acts
>> both as an input parameter for the Perl function and as the variable where
>> the return value from the function is stored.
>>
>> There is also an alternative way for addressing the same assignment. This
>> approach involves keeping the counter for each call ID not in a SEC
>> context, but rather in a Perl hash table. That approach is more efficient,
>> since the counter value is not copied back and forth between the Perl
>> function and the SEC context each time it needs to be incremented. Instead,
>> the counter value stays in a Perl data structure during these increments.
>> Here is a relevant rule example:
>>
>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>>        write - the new value for counter for call $1 is %sequence_num
>>
>> The above example involves the use of a Perl hash (associative array)
>> 'calls', where the call ID serves as a key into the hash. For each key
>> (call ID), the hash stores the counter for that call ID. Each time the
>> function
>> sub { ++$calls{$_[0]} }
>> gets called, the counter for the relevant call ID (the value of $1) is
>> incremented in the hash 'calls', and the result is stored to %sequence_num.
>>
>> If you just want to retrieve the current value of the counter for some
>> call ID (the value of $1) without incrementing it, you can simply invoke
>> the following action:
>>
>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )
>>
>> With both the context-based and Perl-hash-based approach, one needs to
>> think about dropping stale contexts (or stale hash keys) for calls that are
>> no longer relevant. With contexts, you can remove them with the 'delete'
>> action. In the case you use the Perl hash for storing the counters, you can
>> use the following 'lcall' action for deleting a key from the hash 'calls':
>>
>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )
>>
>> I hope the above examples are useful.
>>
>> kind regards,
>> risto
>>
>>
>>
>>
>>
>>
>>
>>
>> Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
>> kuupäeval R, 22. september 2023 kell 19:50:
>>
>>> I am using SEC for an atypical use case, where rather than using log
>>> events to correlate events for things like security events, I am using it
>>> to parse through Asterisk logs in order to capture a sequence of phone call
>>> events (call comes in, auto attendant answers, user selects a digit, call
>>> is transferred, etc). I have SEC write each to a CSV file, which can then
>>> be processed downstream for reporting (for example, I produced a Sankey
>>> chart using this data).
>>>
>>> SEC has proven fantastic for this, however one minor issue has been that
>>> the timestamps in the logs are not granular to smaller than a second, so
>>> it's possible for two or more events to occur within the same second.
>>> Generally this doesn't cause a problem, but when sorting the CSV file
>>> elsewhere, this can result in out-of-sequence lines, if they both contain
>>> the exact same timestamp.
>>>
>>> So, what I've been trying to figure out is how to store a counter that
>>> is tied to the uniqueid of the call, and increment that with each event. I
>>> figured I'd be able to do this by storing an integer in the event store of
>>> a context (tied to the uniqueid for that call). I can then increment it as
>>> the various log lines of the call are processed in turn.
>>>
>>>
>>>
>

Risto, 

I cannot change the timestamps, as they are coded in the files I'm working with (old data). Going forward, I also cannot guarantee that I will have control over the source data, and it is likely that it will have a similar granularity. Finally, there's no need for that level of granularity downstream; it is only necessary that two events with the same timestamp are sequenced so that it is clear which happened first. 

It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, I am almost certain to have to introduce kludge somewhere else, which is worse (this way I only have SEC to maintain, rather than additional technology). 

This way, I can feed relevant log files to SEC from any source, and trust that I'll be able to handle them, so long as they follow standard Asterisk log format. 

Thanks again for your generous advice. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Saturday, 23 September, 2023 03:51:02
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,
> the solutions from my previous post were provided without knowing the root cause
> of the issue. If the main reason for the problem is the nature of the
> timestamps in the logs (that is, they are provided with an accuracy of a
> second), I would recommend using a different logging scheme with
> high-resolution timestamps. For example, as David has mentioned in his post,
> using syslog servers with RFC5424-based syslog protocol ( [
> https://datatracker.ietf.org/doc/html/rfc5424 |
> https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps
> out-of-the-box. I am not sure whether you can change the logging scheme for
> these particular application messages, but if you can, these timestamps are
> highly useful for many purposes like the investigation of past system faults
> and security incidents.
> kind regards,
> risto

>> hi Jim,

>> let me provide some suggestions how to accomplish this task. First, you could
>> utilize the context based approach that you have described in your post. In
>> that case, the numeral that has been retrieved from the context with the 'pop'
>> action needs to be incremented, and you can do it with the 'lcall' action that
>> invokes the relevant Perl code. Here is one example (I have added several of
>> extra actions into the rule for initializing the context):

>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>> pop call_uniqueid_seq_$1 %sequence_num; \
>> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>> add call_uniqueid_seq_$1 %sequence_num

>> The first two actions check if the context call_uniqueid_seq_* for the given
>> call exists, and if not, 0 is written into that context to initialize it:

>> exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
>> identical to the actions from your email.

>> The 'lcall' action (4th action in the above rule) increments the counter value
>> retrieved from the context by executing the following Perl one-line function:
>> sub { $_[0] + 1 }
>> That function takes the value of its parameter (that is, %sequence_num) and
>> returns an incremented value which is stored back into %sequence_num. Note that
>> %sequence_num appears twice in the 'lcall' action, since it acts both as an
>> input parameter for the Perl function and as the variable where the return
>> value from the function is stored.

>> There is also an alternative way for addressing the same assignment. This
>> approach involves keeping the counter for each call ID not in a SEC context,
>> but rather in a Perl hash table. That approach is more efficient, since the
>> counter value is not copied back and forth between the Perl function and the
>> SEC context each time it needs to be incremented. Instead, the counter value
>> stays in a Perl data structure during these increments. Here is a relevant rule
>> example:

>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>> write - the new value for counter for call $1 is %sequence_num

>> The above example involves the use of a Perl hash (associative array) 'calls',
>> where the call ID serves as a key into the hash. For each key (call ID), the
>> hash stores the counter for that call ID. Each time the function
>> sub { ++$calls{$_[0]} }
>> gets called, the counter for the relevant call ID (the value of $1) is
>> incremented in the hash 'calls', and the result is stored to %sequence_num.

>> If you just want to retrieve the current value of the counter for some call ID
>> (the value of $1) without incrementing it, you can simply invoke the following
>> action:

>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

>> With both the context-based and Perl-hash-based approach, one needs to think
>> about dropping stale contexts (or stale hash keys) for calls that are no longer
>> relevant. With contexts, you can remove them with the 'delete' action. In the
>> case you use the Perl hash for storing the counters, you can use the following
>> 'lcall' action for deleting a key from the hash 'calls':

>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

>> I hope the above examples are useful.

>> kind regards,
>> risto

>> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
>> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
>> kell 19:50:

>>> I am using SEC for an atypical use case, where rather than using log events to
>>> correlate events for things like security events, I am using it to parse
>>> through Asterisk logs in order to capture a sequence of phone call events (call
>>> comes in, auto attendant answers, user selects a digit, call is transferred,
>>> etc). I have SEC write each to a CSV file, which can then be processed
>>> downstream for reporting (for example, I produced a Sankey chart using this
>>> data).

>>> SEC has proven fantastic for this, however one minor issue has been that the
>>> timestamps in the logs are not granular to smaller than a second, so it's
>>> possible for two or more events to occur within the same second. Generally this
>>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>>> So, what I've been trying to figure out is how to store a counter that is tied
>>> to the uniqueid of the call, and increment that with each event. I figured I'd
>>> be able to do this by storing an integer in the event store of a context (tied
>>> to the uniqueid for that call). I can then increment it as the various log
>>> lines of the call are processed in turn.

Risto, 

Thank you for this. It makes sense, and you've taken the trouble to explain all the nuances that I would have otherwise missed. Wow. 

I'm going to work through this on Monday. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Friday, 22 September, 2023 15:42:28
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,

> let me provide some suggestions how to accomplish this task. First, you could
> utilize the context based approach that you have described in your post. In
> that case, the numeral that has been retrieved from the context with the 'pop'
> action needs to be incremented, and you can do it with the 'lcall' action that
> invokes the relevant Perl code. Here is one example (I have added several of
> extra actions into the rule for initializing the context):

> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
> pop call_uniqueid_seq_$1 %sequence_num; \
> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
> add call_uniqueid_seq_$1 %sequence_num

> The first two actions check if the context call_uniqueid_seq_* for the given
> call exists, and if not, 0 is written into that context to initialize it:

> exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
> identical to the actions from your email.

> The 'lcall' action (4th action in the above rule) increments the counter value
> retrieved from the context by executing the following Perl one-line function:
> sub { $_[0] + 1 }
> That function takes the value of its parameter (that is, %sequence_num) and
> returns an incremented value which is stored back into %sequence_num. Note that
> %sequence_num appears twice in the 'lcall' action, since it acts both as an
> input parameter for the Perl function and as the variable where the return
> value from the function is stored.

> There is also an alternative way for addressing the same assignment. This
> approach involves keeping the counter for each call ID not in a SEC context,
> but rather in a Perl hash table. That approach is more efficient, since the
> counter value is not copied back and forth between the Perl function and the
> SEC context each time it needs to be incremented. Instead, the counter value
> stays in a Perl data structure during these increments. Here is a relevant rule
> example:

> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
> write - the new value for counter for call $1 is %sequence_num

> The above example involves the use of a Perl hash (associative array) 'calls',
> where the call ID serves as a key into the hash. For each key (call ID), the
> hash stores the counter for that call ID. Each time the function
> sub { ++$calls{$_[0]} }
> gets called, the counter for the relevant call ID (the value of $1) is
> incremented in the hash 'calls', and the result is stored to %sequence_num.

> If you just want to retrieve the current value of the counter for some call ID
> (the value of $1) without incrementing it, you can simply invoke the following
> action:

> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

> With both the context-based and Perl-hash-based approach, one needs to think
> about dropping stale contexts (or stale hash keys) for calls that are no longer
> relevant. With contexts, you can remove them with the 'delete' action. In the
> case you use the Perl hash for storing the counters, you can use the following
> 'lcall' action for deleting a key from the hash 'calls':

> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

> I hope the above examples are useful.

> kind regards,
> risto

> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
> kell 19:50:

>> I am using SEC for an atypical use case, where rather than using log events to
>> correlate events for things like security events, I am using it to parse
>> through Asterisk logs in order to capture a sequence of phone call events (call
>> comes in, auto attendant answers, user selects a digit, call is transferred,
>> etc). I have SEC write each to a CSV file, which can then be processed
>> downstream for reporting (for example, I produced a Sankey chart using this
>> data).

>> SEC has proven fantastic for this, however one minor issue has been that the
>> timestamps in the logs are not granular to smaller than a second, so it's
>> possible for two or more events to occur within the same second. Generally this
>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>> So, what I've been trying to figure out is how to store a counter that is tied
>> to the uniqueid of the call, and increment that with each event. I figured I'd
>> be able to do this by storing an integer in the event store of a context (tied
>> to the uniqueid for that call). I can then increment it as the various log
>> lines of the call are processed in turn.

>> The part I think I'm not getting is due to my lack of understanding of Perl (and
>> specifically perl syntax).

>> The first rule can create the context:

>> add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call)

>> But then in subsequent rules I want to do something like this:

>> call_uniqueid_seq_$4 ++
>> However I don't expect that to be valid syntax (especially since that store is
>> for strings), so I have to figure out something else.

>> I was working my head around just 'pop'ing the value off the event store, and
>> then adding 1 to it, but I realized I have no clue how to do this.

>> So, I am hoping that I am just missing something obvious, but, failing that, I
>> guess I need to understand how I would do something like:

>> pop call_uniqueid_seq_$4 %sequence_num
>> add call_uniqueid_seq_$4 %sequence_num+1
>> I feel like I'm on the wrong track here, but I'm not finding the answers in the
>> man page, or tutorial, or anywhere else, and my knowledge of Perl is
>> insufficient to help make sense of this.

>> Any thoughts or advice or clues would be greatly appreciated.

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> [ mailto:Sim...@li... |
>> Sim...@li... ]
>> [ https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ]

hi Jim,
the solutions from my previous post were provided without knowing the root
cause of the issue. If the main reason for the problem is the nature of the
timestamps in the logs (that is, they are provided with an accuracy of a
second), I would recommend using a different logging scheme with
high-resolution timestamps. For example, as David has mentioned in his
post, using syslog servers with RFC5424-based syslog protocol (
https://datatracker.ietf.org/doc/html/rfc5424) provides such timestamps
out-of-the-box. I am not sure whether you can change the logging scheme for
these particular application messages, but if you can, these timestamps are
highly useful for many purposes like the investigation of past system
faults and security incidents.
kind regards,
risto


hi Jim,
>
> let me provide some suggestions how to accomplish this task. First, you
> could utilize the context based approach that you have described in your
> post. In that case, the numeral that has been retrieved from the context
> with the 'pop' action needs to be incremented, and you can do it with the
> 'lcall' action that invokes the relevant Perl code. Here is one example (I
> have added several of extra actions into the rule for initializing the
> context):
>
> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=exists %e call_uniqueid_seq_$1; \
>        if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>        pop call_uniqueid_seq_$1 %sequence_num; \
>        lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>        add call_uniqueid_seq_$1 %sequence_num
>
> The first two actions check if the context call_uniqueid_seq_* for the
> given call exists, and if not, 0 is written into that context to initialize
> it:
>
> exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0);
>
> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
> identical to the actions from your email.
>
> The 'lcall' action (4th action in the above rule) increments the counter
> value retrieved from the context by executing the following Perl one-line
> function:
> sub { $_[0] + 1 }
> That function takes the value of its parameter (that is, %sequence_num)
> and returns an incremented value which is stored back into %sequence_num.
> Note that %sequence_num appears twice in the 'lcall' action, since it acts
> both as an input parameter for the Perl function and as the variable where
> the return value from the function is stored.
>
> There is also an alternative way for addressing the same assignment. This
> approach involves keeping the counter for each call ID not in a SEC
> context, but rather in a Perl hash table. That approach is more efficient,
> since the counter value is not copied back and forth between the Perl
> function and the SEC context each time it needs to be incremented. Instead,
> the counter value stays in a Perl data structure during these increments.
> Here is a relevant rule example:
>
> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>        write - the new value for counter for call $1 is %sequence_num
>
> The above example involves the use of a Perl hash (associative array)
> 'calls', where the call ID serves as a key into the hash. For each key
> (call ID), the hash stores the counter for that call ID. Each time the
> function
> sub { ++$calls{$_[0]} }
> gets called, the counter for the relevant call ID (the value of $1) is
> incremented in the hash 'calls', and the result is stored to %sequence_num.
>
> If you just want to retrieve the current value of the counter for some
> call ID (the value of $1) without incrementing it, you can simply invoke
> the following action:
>
> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )
>
> With both the context-based and Perl-hash-based approach, one needs to
> think about dropping stale contexts (or stale hash keys) for calls that are
> no longer relevant. With contexts, you can remove them with the 'delete'
> action. In the case you use the Perl hash for storing the counters, you can
> use the following 'lcall' action for deleting a key from the hash 'calls':
>
> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )
>
> I hope the above examples are useful.
>
> kind regards,
> risto
>
>
>
>
>
>
>
>
> Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
> kuupäeval R, 22. september 2023 kell 19:50:
>
>> I am using SEC for an atypical use case, where rather than using log
>> events to correlate events for things like security events, I am using it
>> to parse through Asterisk logs in order to capture a sequence of phone call
>> events (call comes in, auto attendant answers, user selects a digit, call
>> is transferred, etc). I have SEC write each to a CSV file, which can then
>> be processed downstream for reporting (for example, I produced a Sankey
>> chart using this data).
>>
>> SEC has proven fantastic for this, however one minor issue has been that
>> the timestamps in the logs are not granular to smaller than a second, so
>> it's possible for two or more events to occur within the same second.
>> Generally this doesn't cause a problem, but when sorting the CSV file
>> elsewhere, this can result in out-of-sequence lines, if they both contain
>> the exact same timestamp.
>>
>> So, what I've been trying to figure out is how to store a counter that is
>> tied to the uniqueid of the call, and increment that with each event. I
>> figured I'd be able to do this by storing an integer in the event store of
>> a context (tied to the uniqueid for that call). I can then increment it as
>> the various log lines of the call are processed in turn.
>>
>>
>>

most syslog servers have the ability to log with subsecond accuracy, and 
RFC-5424 requires it in the spec.

David Lang

Hi Jim,
Interesting use case!
If it helps, LogZilla (I am the founder) tracks events with millisecond accuracy and has SEC built in. Using them together, you could just ask LZ for the timestamp.
Might be something worth checking out.



From: Jim Van Meggelen <jim...@cl...>
Date: Friday, September 22, 2023 at 12:50 PM
To: simple-evcorr-users <sim...@li...>
Subject: [Simple-evcorr-users] Storing a sequence counter in a context
I am using SEC for an atypical use case, where rather than using log events to correlate events for things like security events, I am using it to parse through Asterisk logs in order to capture a sequence of phone call events (call comes in, auto attendant answers, user selects a digit, call is transferred, etc). I have SEC write each to a CSV file, which can then be processed downstream for reporting (for example, I produced a Sankey chart using this data).

SEC has proven fantastic for this, however one minor issue has been that the timestamps in the logs are not granular to smaller than a second, so it's possible for two or more events to occur within the same second. Generally this doesn't cause a problem, but when sorting the CSV file elsewhere, this can result in out-of-sequence lines, if they both contain the exact same timestamp.

So, what I've been trying to figure out is how to store a counter that is tied to the uniqueid of the call, and increment that with each event. I figured I'd be able to do this by storing an integer in the event store of a context (tied to the uniqueid for that call). I can then increment it as the various log lines of the call are processed in turn.

The part I think I'm not getting is due to my lack of understanding of Perl (and specifically perl syntax).

The first rule can create the context:

add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call)

But then in subsequent rules I want to do something like this:

call_uniqueid_seq_$4 ++

However I don't expect that to be valid syntax (especially since that store is for strings), so I have to figure out something else.


I was working my head around just 'pop'ing the value off the event store, and then adding 1 to it, but I realized I have no clue how to do this.

So, I am hoping that I am just missing something obvious, but, failing that, I guess I need to understand how I would do something like:

pop call_uniqueid_seq_$4 %sequence_num
add call_uniqueid_seq_$4 %sequence_num+1

I feel like I'm on the wrong track here, but I'm not finding the answers in the man page, or tutorial, or anywhere else, and my knowledge of Perl is insufficient to help make sense of this.

Any thoughts or advice or clues would be greatly appreciated.


--
Jim Van Meggelen
ClearlyCore Inc.

[cid:e655b21c1463e37f7bba0bc97c53e4cb81753f44@zimbra]

+1-416-639-6001 (DID)
+1-877-253-2716 (Canada)
+1-866-644-7729 (USA)
+1-416-425-6111 x6001
jim...@cl...
http://www.clearlycore.com

Asterisk: The Definitive Guide
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf