[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[tsf-dev] TSF process feedback, part 1: statement graphs and references
|
Hello,
I recently became team lead on CTRL, a Linux OS maintained by
Codethink. CTRL was the first project to use TSF and is still its main
proving ground. Internally we have been reflecting on our use of TSF,
what works and what doesn't and I want to share our findings here with
the wider TSF community.
In CTRL we use the following things that you know from TSF:
1. A set of Statements which present an assurance case, which is
managed with the trudag tool.
2. References as evidence for the statements.
3. Evaluation of the statements by subject matter experts (SMEs).
4. Evaluation of the statements using validator scripts.
5. Aggregate scores
6. Monitoring when evidence changes
7. Modifying the statements via Gitlab
8. The RAFIA process and STPA.
I'm going to send a series of mails this week with various learnings
that will hopefully be helpful for fellow TSF users, and for the
project owners to triage improvements.
If you haven't, take a moment to revisit the TSF "Rationale" page:
<https://pages.eclipse.dev/eclipse/tsf/tsf/rationale.html>. The
rationale doesn't mention development processes, beyond saying that
today's are not fit for purpose, but it implies that correct usage
of TSF involves storing the assurance case for your project in the
same Git repo where you define your project. Which is what we do.
Here are the learnings for the first two items in my list.
Statement graphs
----------------
The TSF "Model" page gives guidance on writing clear statements:
<https://pages.eclipse.dev/eclipse/tsf/tsf/model/model.html>.
Internally, I found where people write new TSF statements they
are clear and well written. This part fundamentally works well.
One gotcha: people have at times imported existing texts into TSF as
Statements, without taking time to rewrite them. One example was a step
by step process where each step became a statement. It might have been
a good process, but once imported into TSF we would see particular steps
out-of-context and they'd make no sense.
The lesson here is to use TSF statements for what they're designed for:
setting expectations, and providing evidence:
<https://pages.eclipse.dev/eclipse/tsf/tsf/model/methodology.html>.
Another gotcha: our graph of statements has become simply too big,
and has too many different owners.
This is something we have seen coming for a long time, and is driving
the development of the TSF remote graphs feature:
<https://pages.eclipse.dev/eclipse/tsf/tsf/tools/remote-graph.html>.
On reflection, it seems like 100 statements is a sensible target
for a TSF graph, and we should perhaps enforce an upper limit of 500
statements.
The lesson here is to treat your TSF argument like a document that you
can print out and read from start to finish. If its too long for one
person to read through, how can you ever be sure that it's well written
and free from contradictions?
One bonus lesson learned. We originally numbered the statements, like
FOO-PROCESS-001 and so on. This quickly becomes not a good idea. We
nowadays aim to use meaningful names, e.g.
FOO-PROCESS-EVERYTHING-IN-GIT. The names can get long. But its less
cognitive load than having to keep hundreds of numbered codes in your
head.
References as evidence
----------------------
Many statements in our graph have one or more references. The intention
is to provide supporting evidence for the statement, on the principle
that trust should be based on evidence. The principle is sound and in
many cases this works nicely, but you have to always be asking:
* Is this the *right* evidence?
* Is this *enough* evidence?
* What context do you need to *understand* this evidence?
* How often will this piece of evidence change and *require re-review*?
Some anti-patterns that we've seen:
* Evidence that relates to the topic of the statement, but doesn't
demonstrate that it's true.
* Evidence that is a small section of a file, but someone referenced
the entire file.
* Evidence that is serialized data in some format, but without any
link to documentation of the data model or the program that reads
it.
An illustrative example that demonstrates all these anti-patterns: one
statement said "Configuration of Foo is managed solely via this config
file", and as evidence it linked to a huge config file for Foo.
This of course does *not* demonstrate that Foo is solely configured via
that file, only that we have a config file. Better evidence would be
the following:
* Documentation or code that demonstrates where Foo reads its config
from.
* A link to where we provide the config to Foo
The *contents* of the config might be interesting context, but they
don't prove anything and should not be listed as references.
Since TSF allows references via plugin, we have developed several
plugins that allow referencing specific parts of file, e.g. you might
point to a specific function in a C program, or a specific key in a YAML
file, or one heading in a Markdown file. Some of these are due to be
published in the open, see:
<https://gitlab.eclipse.org/eclipse/tsf/tsf/-/issues/497>.
References might be in the same repo as the graph, but in many cases
they are stored outside the Git repo. Each external reference adds a
cost, as Trudag will fetch it on startup to see if it changed. If
this comes in via network, this means that generating your assurance
case now depends on some network infrastructure, and if you don't host
the file yourself, you should think about mirroring it.
References have another cost, due to the "Suspect item" and "Suspect
link" detection that flags when they change. I'll discuss this below in
the "Modifying the statements" section.
Lesson: keep references to the minimum necessary, and review them
carefully.
---
Thanks for reading. Has anyone else learned these lessons the hard way,
or have hints for how we can use TSF more effectively here?
Best regards
San
--
Sam Thursfield (he/him), Software Engineer
Codethink Ltd. http://www.codethink.co.uk/
