[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [tracecompass-dev] TotalADS: Total Anomaly Detection System
|
On 11/27/2014 11:34 AM, Shariyar wrote:
Hi Alexandre,
My replies are inline.
On Wed, Nov 26, 2014 at 10:47 PM, Alexandre Montplaisir <
alexmonthy@xxxxxxxxxxxx> wrote:
Hi Shariyar,
First, congrats on getting this out! I've seen demos of TotalADS many
times so far, and I'm very happy to see this coming out as open-source.
Props to you and all the people at Concordia who worked on this!
Thank you :).
Some questions:
Is this meant to be a project of its own? I'm wondering if it wouldn't be
better to have this in a separate git repository. It could for example be
under the Trace Compass project umbrella at Eclipse, but have its own
separate git repo. Especially since it has its own dependencies that aren't
shared with the rest. It could also be advertised on its own, or as a
plugin to Trace Compass.
Just throwing the idea out there, we'll have to see more in details how
everything integrates together.
It actually depends on Trace Compass for parsing CTF traces, XML files and
text files via regular expression. So I am wondering whether a separate git
repo will work or not. Will a new developer has to pull both repos. for
development? It would be better to have its own git repository but I think
it has to be under Trace Compass right now.
It would be perfectly fine to have a separate repo and depend on the
main Trace Compass plugins. Just as you have other dependencies already,
TC would just be another one of those.
If a developer wants to pull the TotalADS code to work on it, they could
get the TC dependencies automatically from the target definition, for
example.
To me, a separate git repo under the "Eclipse Trace Compass" project
would make a lot of sense for this. Anyone else has an opinion?
Also how do I create a new git
repo in Gerrit. From the website I learnt that I have to file a new bug
report for a particular plugin for Eclipse to create a new repo. Is that
right or is there any other way?
It's the project maintainers (us) who have to take care of this. It's
quite straightforward to do. I'm just waiting to see the conclusion of
this discussion first ;)
Related to the previous question, who will be maintaining this? What is
the plan going forward? I assume it's not "done done", and that there will
be people around for fixing bugs? Will more people from Concordia be
working on this? Ideally it shouldn't be a case of "dump & run" ;)
Unfortunately, I shall not be doing bug fixing and development for it in
future. It will be maintained by someone else at Concordia, essentially it
will be the responsibility of Wahab. Having said that, I will be making
sure that any new features and algorithms added to TotalADS are
theoretically and syntactically correct before pushing them further to
Trace Compass. So, yes there will be someone maintaining this.
Excellent! Initially, all code will have to go through existing
committers, but if people from Concordia become regularly involved, then
eventually some could become Trace Compass committers themselves, and
continue working on the TotalADS repo directly.
Some remarks:
- Everything Geneviève said!
I am not sure how to divide it further into small patches. I can push one
plugin or two plugins at a time(e.g., core and test) . However, if we are
going to keep a separate repository then I can push all the code there and
others can review it gradually as they like.
Even if we make a new repo, the initial push should be relatively in
good shape (no embedded libraries for instance, we wouldn't be allowed
to even push this). You could start by squashing all the commits on your
Github repo into one, to eliminate for example multiple edits to the
same file.
Afterwards, you could split that one big change into smaller commits,
such as:
- Provide empty plugins (no code, just the settings and Manifests, etc.
Just that is usually quite big!)
- Add the run-time code
- Add the test cases
- It seems to be based on a very recent version of Trace Compass, good job
getting it up to date!
Yes, I went to push it to Linuxtools. tmf and then I found out that it has
moved out. So I decided to change it in the beginning before being too late.
- Binary test traces should not be committed in the git tree. We host our
test traces on archive.eclipse.org, and we have our build system download
them on-demand. We can help you set that up once we get to that point.
Yes, please I am not sure how to do that. Are there any instructions any
where?
You can take a look at the "org.eclipse.tracecompass.ctf.core.tests"
plugin. We use an Ant script to download the test files. The only tricky
part is to then use assumeTrue(traceExists) in the test cases, to make
sure the test is skipped if the traces are not present (as in these
cases it's possible for the traces to "not be there").
Once the test cases are ready to be pushed, we can upload those traces
to our repo on a.e.o
Cheers,
Alexandre
- Embedded libraries are a big no-no. You could look into using a target
definition to allow Eclipse to download the dependencies (see the
"org.eclipse.tracecompass.target" plugin in our git tree). Eclipse
projects that depend on external libraries have to get them packaged in
Orbit [1] first. All the ones you are depending on are already in Orbit,
with the exception of Mahout. But that one seems to be under the Apache
License, so there should be no problem getting it into Orbit, once we get
there.
Okay, I shall try to create the target definition file for libraries.
Cheers,
Alexandre
[1] Latest Orbit build: http://download.eclipse.org/
tools/orbit/downloads/drops/R20140525021250/
Regards,
Shariyar
On 2014-11-25 11:31 AM, Shariyar wrote:
Dear all,
I would like to introduce a new plugin for Tracecompass, called TotalADS
(Total Anomaly Detection System). Here is a brief introduction:
TotalADS is a novel framework for automated host-based anomaly detection.
TotalADS is an open source tool developed as a plug-in for Eclipse. It
integrates different anomaly detection algorithms (or techniques),
different trace readers and a rich set of trace views in one common
platform.
Currently, TotalADS encompasses three different algorithms, such as
Sequence Matching (SQM), Kernel State Modeling (KSM), and Hidden Markov
Model (HMM). It supports execution traces and logs in CTF, XML and text
format. It also supports live anomaly detection using trace streaming
along
with real time training and testing.
TotalADS also extends another Eclipse plugin called Tracecompass by using
the rich set of views present in it for the visualization of traces, such
as control flow of processes,resource usages and etc.
TotalADS has a number of applications, such as automatic detection of zero
day attacks, diagnosis of anomalous paths in failure traces, and diagnosis
of performance faults in the system.
TotalADS is extendible through simple Java interfaces: new algorithms and
trace readers can be easily added.algorithms and trace readers can be
easily added.
Here is the wiki of TotalADS:
https://github.com/sshahriyar/org.eclipse.tracecompass/wiki
The fork repository of Tracecompass and the new plugin TotalADS:
https://github.com/sshahriyar/org.eclipse.tracecompass
TotalADS adds four new plugins to Tracecompass, namely:
totalads.core
totalads.core.tests
totalads.ui
totalads.ui.swtbot.tests
TotalADS does not make changes to the source code of existing plugins of
Tracecompass except adding references to the plugins where necessary.
Screenshots are available here:
http://users.encs.concordia.ca/~abdelw/sba/totalads/features.html
A Use Case:
https://github.com/sshahriyar/org.eclipse.tracecompass/wiki/Use-Case
Let me know how can I push the code for review.
Regards,
Shariyar
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/tracecompass-dev
