Hi,
I have removed the dependency on embedded libraries by using the
target file. However, Mahout is missing in Orbit. I cannot add them
there. So can any one add Mahout there? Specifically, I require the
following jars:
mahout-core-0.9-job.jar
mahout-core-0.9.jar
mahout-integration-0.9.jar
mahout-math-0.9.jar
Thanks,
Shariyar
On Thu, Nov 27, 2014 at 2:04 PM, Alexandre Montplaisir
<alexmonthy@xxxxxxxxxxxx <mailto:alexmonthy@xxxxxxxxxxxx>> wrote:
On 11/27/2014 11:34 AM, Shariyar wrote:
Hi Alexandre,
My replies are inline.
On Wed, Nov 26, 2014 at 10:47 PM, Alexandre Montplaisir <
alexmonthy@xxxxxxxxxxxx <mailto:alexmonthy@xxxxxxxxxxxx>> wrote:
Hi Shariyar,
First, congrats on getting this out! I've seen demos of
TotalADS many
times so far, and I'm very happy to see this coming out as
open-source.
Props to you and all the people at Concordia who worked on
this!
Thank you :).
Some questions:
Is this meant to be a project of its own? I'm wondering if
it wouldn't be
better to have this in a separate git repository. It could
for example be
under the Trace Compass project umbrella at Eclipse, but
have its own
separate git repo. Especially since it has its own
dependencies that aren't
shared with the rest. It could also be advertised on its
own, or as a
plugin to Trace Compass.
Just throwing the idea out there, we'll have to see more
in details how
everything integrates together.
It actually depends on Trace Compass for parsing CTF traces,
XML files and
text files via regular expression. So I am wondering whether a
separate git
repo will work or not. Will a new developer has to pull both
repos. for
development? It would be better to have its own git repository
but I think
it has to be under Trace Compass right now.
It would be perfectly fine to have a separate repo and depend on
the main Trace Compass plugins. Just as you have other
dependencies already, TC would just be another one of those.
If a developer wants to pull the TotalADS code to work on it, they
could get the TC dependencies automatically from the target
definition, for example.
To me, a separate git repo under the "Eclipse Trace Compass"
project would make a lot of sense for this. Anyone else has an
opinion?
Also how do I create a new git
repo in Gerrit. From the website I learnt that I have to file
a new bug
report for a particular plugin for Eclipse to create a new
repo. Is that
right or is there any other way?
It's the project maintainers (us) who have to take care of this.
It's quite straightforward to do. I'm just waiting to see the
conclusion of this discussion first ;)
Related to the previous question, who will be maintaining
this? What is
the plan going forward? I assume it's not "done done", and
that there will
be people around for fixing bugs? Will more people from
Concordia be
working on this? Ideally it shouldn't be a case of "dump &
run" ;)
Unfortunately, I shall not be doing bug fixing and
development for it in
future. It will be maintained by someone else at Concordia,
essentially it
will be the responsibility of Wahab. Having said that, I will
be making
sure that any new features and algorithms added to TotalADS are
theoretically and syntactically correct before pushing them
further to
Trace Compass. So, yes there will be someone maintaining this.
Excellent! Initially, all code will have to go through existing
committers, but if people from Concordia become regularly
involved, then eventually some could become Trace Compass
committers themselves, and continue working on the TotalADS repo
directly.
Some remarks:
- Everything Geneviève said!
I am not sure how to divide it further into small patches. I
can push one
plugin or two plugins at a time(e.g., core and test) .
However, if we are
going to keep a separate repository then I can push all the
code there and
others can review it gradually as they like.
Even if we make a new repo, the initial push should be relatively
in good shape (no embedded libraries for instance, we wouldn't be
allowed to even push this). You could start by squashing all the
commits on your Github repo into one, to eliminate for example
multiple edits to the same file.
Afterwards, you could split that one big change into smaller
commits, such as:
- Provide empty plugins (no code, just the settings and Manifests,
etc. Just that is usually quite big!)
- Add the run-time code
- Add the test cases
- It seems to be based on a very recent version of Trace
Compass, good job
getting it up to date!
Yes, I went to push it to Linuxtools. tmf and then I found out
that it has
moved out. So I decided to change it in the beginning before
being too late.
- Binary test traces should not be committed in the git
tree. We host our
test traces on archive.eclipse.org
<http://archive.eclipse.org>, and we have our build system
download
them on-demand. We can help you set that up once we get to
that point.
Yes, please I am not sure how to do that. Are there any
instructions any
where?
You can take a look at the
"org.eclipse.tracecompass.ctf.core.tests" plugin. We use an Ant
script to download the test files. The only tricky part is to then
use assumeTrue(traceExists) in the test cases, to make sure the
test is skipped if the traces are not present (as in these cases
it's possible for the traces to "not be there").
Once the test cases are ready to be pushed, we can upload those
traces to our repo on a.e.o
Cheers,
Alexandre
- Embedded libraries are a big no-no. You could look into
using a target
definition to allow Eclipse to download the dependencies
(see the
"org.eclipse.tracecompass.target" plugin in our git tree).
Eclipse
projects that depend on external libraries have to get
them packaged in
Orbit [1] first. All the ones you are depending on are
already in Orbit,
with the exception of Mahout. But that one seems to be
under the Apache
License, so there should be no problem getting it into
Orbit, once we get
there.
Okay, I shall try to create the target definition file for
libraries.
Cheers,
Alexandre
[1] Latest Orbit build: http://download.eclipse.org/
tools/orbit/downloads/drops/R20140525021250/
Regards,
Shariyar
On 2014-11-25 11:31 AM, Shariyar wrote:
Dear all,
I would like to introduce a new plugin for
Tracecompass, called TotalADS
(Total Anomaly Detection System). Here is a brief
introduction:
TotalADS is a novel framework for automated host-based
anomaly detection.
TotalADS is an open source tool developed as a plug-in
for Eclipse. It
integrates different anomaly detection algorithms (or
techniques),
different trace readers and a rich set of trace views
in one common
platform.
Currently, TotalADS encompasses three different
algorithms, such as
Sequence Matching (SQM), Kernel State Modeling (KSM),
and Hidden Markov
Model (HMM). It supports execution traces and logs in
CTF, XML and text
format. It also supports live anomaly detection using
trace streaming
along
with real time training and testing.
TotalADS also extends another Eclipse plugin called
Tracecompass by using
the rich set of views present in it for the
visualization of traces, such
as control flow of processes,resource usages and etc.
TotalADS has a number of applications, such as
automatic detection of zero
day attacks, diagnosis of anomalous paths in failure
traces, and diagnosis
of performance faults in the system.
TotalADS is extendible through simple Java interfaces:
new algorithms and
trace readers can be easily added.algorithms and trace
readers can be
easily added.
Here is the wiki of TotalADS:
https://github.com/sshahriyar/org.eclipse.tracecompass/wiki
The fork repository of Tracecompass and the new plugin
TotalADS:
https://github.com/sshahriyar/org.eclipse.tracecompass
TotalADS adds four new plugins to Tracecompass, namely:
totalads.core
totalads.core.tests
totalads.ui
totalads.ui.swtbot.tests
TotalADS does not make changes to the source code of
existing plugins of
Tracecompass except adding references to the plugins
where necessary.
Screenshots are available here:
http://users.encs.concordia.ca/~abdelw/sba/totalads/features.html
<http://users.encs.concordia.ca/%7Eabdelw/sba/totalads/features.html>
A Use Case:
https://github.com/sshahriyar/org.eclipse.tracecompass/wiki/Use-Case
Let me know how can I push the code for review.
Regards,
Shariyar
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
<mailto:tracecompass-dev@xxxxxxxxxxx>
To change your delivery options, retrieve your
password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/tracecompass-dev
