Hello,
Current TRS Server implementation in Lyo is rather naïve when it comes to long-term server
operation. One thing that can be improved is how paging is done. Another is how to use Lyo Store and/or Redis for persisting the pages.
Currently, a TRS Server keeps its Change Log in memory and uses simple URI patterns:
-
/services/trs/ points to
-
/services/trs/base/1
-
/services/trs/changeLog/2
-
…
-
/services/trs/changeLog/m
When the rebase happens, it rebuilds the Change Log completely. Finally, Change Log pages are formed on-the-fly.
Issues 1 & 2 can cause the following problems:
-
Keeping things in memory means
rebase would happen on every restart.
-
Stateful TRS Server also prevents an OSLC microservice from being placed behind a load balancer.
-
When a new event happens, the contents of all Change Log pages would change and a TRS Client would see previously observed Change Events in the subsequent Change Log pages.
-
trs:order property may be assigned to different resources upon rebase and/or restart and the Cutoff Event would lose sense. TRS Client would detect this and would perform a full rebase.
Because of the URI patterns & issue 3, the only way a TRS Client can detect a rebase is to follow the TRS Base link and check on its first page if
the Cutoff Event URI has changed or wait to fail to find a Cutoff Event (or the most recent Change Event observed by the TRS Client).
Jad suggested an idea to use Lyo Store to persist Change Events under a triplestore. But without nice and clean paging that allows pages to be persisted
once and be deleted completely once they are “evicted”, doing this would be challenging. After discussions in OSLC Core committee (special thanks to Nick for extensive analysis and detailed examples in the slides), I came up with the following:
-
The TRS Resource should display a
variable number of the most recent changes, the pages should have fixed size “n”.
-
When the number of Change Events to be returned with the TRS Resource exceeds the page size, a new page is created and the number of Change Events returned in the TRS Resource should go from n+1 to 1.
-
The Change Log pages should be numbered in reverse (see an example below).
-
A truncated hash of a Cutoff Event URI is used to provide an ability to return 410 gone or 404 Not Found when the rebase happens in the middle of the client’s traversal of the change log pages.
Here is an example:
-
/services/trs includes a TrackedResourceSet resource with
-
a ChangeLog resource that
-
via trs:previous points to /services/trs/log/ABCD/9 which
-
via trs:previous points to /services/trs/log/ABCD/8…
When m events get added, and the “root” ChangeLog resource grows beyond the page size, we add a page log/ABCD/10:
-
/trs includes a TrackedResourceSet resource with
-
a ChangeLog resource that
-
via trs:previous points to /services/trs/log/ABCD/10 which
-
via trs:previous points to /services/trs/log/ABCD/9…
-
the contents of /services/trs/log/ABCD/9 are identical to its contents before page 10 was added
-
the ChangeLog resource under /trs now contains only one Change Event
When rebase happens and we decide to keep 3 pages worth of events:
-
/trs includes a TrackedResourceSet resource with
-
a ChangeLog resource that
-
via trs:previous points to /services/trs/log/9FE2/2 which
-
via trs:previous points to /services/trs/log/9FE2/1…
-
a request to /services/trs/log/ABCD/5 would return 404 Not Found if we only keep the hash of the current base or 410 Gone if we also keep a set of older bases.
At the cost of keeping the list of the most recent changes separate from the paged log, we get a perfectly cacheable solution with predictable behavior
that can be persisted in a triplestore:
-
each page must be persisted once
-
all pages from a given key get removed upon rebase
-
or we keep all pages for the current and the last keys
-
Varnish can be employed to cache whole Change Log pages for extended periods of time (>60s) and the TRS Resource page for shorter periods (<10s).
Feedback is welcome!
/Andrew
