Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[tsf-dev] TSF process feedback, part 3: aggregate scores, changing evidence 
Hello again,

Here's part 3 of my reflections on our use of TSF in the CTRL project.

In CTRL we use the following things that you know from TSF:

  1. A set of Statements which present an assurance case, which is
     managed with the trudag tool.
  2. References as evidence for the statements.
  3. Evaluation of the statements by subject matter experts (SMEs).
  4. Evaluation of the statements using validator scripts.
  5. Aggregate scores
  6. Monitoring when evidence changes
  7. Modifying the statements via Gitlab
  8. The RAFIA process and STPA.

This mail covers items 5 and 6. If you missed the first two parts,
you can find them here in the archives:

https://www.eclipse.org/lists/tsf-dev/msg00037.html
https://www.eclipse.org/lists/tsf-dev/msg00040.html

Aggregate scores
----------------

If a statement doesn't have an evaluation by SME or plugin, but it
does have links to other statements, then Trudag generates a score
based on some maths that I don't fully understand:
<https://pages.eclipse.dev/eclipse/tsf/tsf/model/scoring/implementation.html#calculating-scores-for-all-items>.

In our experience, the further away you get from edge nodes in the
graph, the less useful these aggregate scores become. The top level
"TRUSTABLE-SOFTWARE" number can vary wildly. Imagine you have 2
statements linked to TA-INPUTS, and 100 more statements linked to
TA-BEHAVIOURS. If a reference for one of the TA-INPUTS statements
changes from 1.0 to 0.0, the TRUSTABLE-SOFTWARE score might go down
by 0.25. Meanwhile, if the same happens under TA-BEHAVIOURS it might
decreate by 0.005. You might infer from a dramatic decrease in the
toplevel score that TA-INPUTS is more important than TA-BEHAVIOURS,
but that's actually not the case at all!

Lesson: be very wary of taking any meaning from the aggregate scores.

Monitoring when evidence changes
--------------------------------

This relates to the "Modification" section in the Trustable Methodology:
<https://pages.eclipse.dev/eclipse/tsf/tsf/model/methodology.html#modification>.

We store our TSF graph in Git, as recommended. In some cases we
reference evidence that's in our Git repo: internal references. And in
other cases, we reference evidence that's outside our Git repo: external
references.

Here I'm going to talk about external references. I'm not sure if this
was intentional, but our external references function as something of
an alarm system.

For example, the inputs to our project are mirrored. We share the
mirroring config with other teams. We have a statement `INPUTS-MIRRORED`
saying "All inputs are mirrored", which includes the mirroring config as
evidence. (It's not the only evidence ;-)

When the mirroring config changes, Trudag flags the `INPUTS-MIRRORED`
statement as "Suspect", meaning someone on our team goes to review it.
This can catch real problems, such as people unintentially deleting
mirrors that we still need.

However, the alarm only sounds when we run `trudag` for some unrelated
task. Perhaps we need a nightly "check external references" job that
emails the team when something has changed.

---

That's all for today. Tomorrow I'll give my final set of notes, on
how we manage change in the argument via Gitlab. See you then.

Best regards,
Sam

--
Sam Thursfield (he/him), Software Engineer
  Codethink Ltd.                            http://www.codethink.co.uk/

Back to the top