| Re: [rdf4j-dev] timelimit, cpu, and memory checks |
Hi Matthew, We don’t have anything for putting memory limits. I know it’s fairly tricky to do with Java-based databases, but I know that some estimate the size of objects and others allocate using Unsafe or ByteBuffer to keep track of memory usage. It’s mostly distinct, order by and group by that directly cause memory issues, as well as hash joins for joins with scope issues. We have a way to set a limit for spilling to disk (iterationCacheSyncThreshold), which kinda limits the amount of memory a query can use while still allowing it to run until it times out. Want to test that out to see if it works with the queries you’re having issues with? A smarter approach might be to measure how much free memory there is and either spill to disk early or terminate a query that is likely to use more memory than we have. We already do something like this for transaction isolation when adding large amounts of data in READ_COMMITTED or above. Probably also a good idea to have a global memory monitor that throttles queries when memory is critically low. For CPU usage limit most queries are single threaded. I think you have to use federation in order to trigger any multithreading. But i don’t think we have anything to manage the balance between letting current queries finish vs accepting new ones. Or any query prioritisation. Any ideas would be very welcome. Cheers, Håvard On 24 Sep 2025, at 14:19, Matthew Nguyen via rdf4j-dev <rdf4j-dev@xxxxxxxxxxx> wrote:
|