| [CDO] Delete EObjects and BLOBs [message #1857642] |
Fri, 17 February 2023 18:36  |
 Alexander Stenzer
Messages: 9
Registered: April 2022 |
Junior Member |
|
|
Hello,
we are facing some strange exceptions and leftovers during delete.
1. Delete BinaryResource:
We want to delete a BinaryResource:
First we create one:
var transaction = session.openTransaction(rSet);
try{
final var pathClean = CDOURIUtil.sanitizePath(path);
final var binaryResource = transaction.getOrCreateBinaryResource(pathClean);
try (var is = Files.newInputStream(file)) {
final CDOBlob blob = new CDOBlob(is, transaction.getSession().options().getLobCache());
binaryResource.setContents(blob);
transaction.commit();
}
finally {
transaction.close();
}
Then we delete one:
var transaction = session.openTransaction(rSet);
try {
final var binaryResource = transaction.getOrCreateBinaryResource(pathClean);
binaryResource.delete(null);
transaction.commit();
} finally {
transaction.close();
}
After that in the table CDO_LOBS the binary entries are still there.
In the table CDOBINARYRESOURCE every thing is clean.
Do we have to delete CDOBLOB manually?
2. Delete EObject with references:
If we delete a large amount of EObject in one transaction we face during delete execption.
In the worst case the exception can lead to a corrupt cdo reposiotry which can not be used anymore.
The problem is, it is not deterministic. Sometimes 1000 elements are no probleme, sometime after 100 the execption occures.
So now how we do it.
The main EObject
public interface MyFile extends CDOObject {
EList<Tags> getTags();
}
The delete code looks like that:
var transaction = session.openTransaction(rSet);
try {
var objectToDelete = transaction.getObject(myfile);
// lock element before delete
objectToDelete.cdoWriteLock().lock();
// clear all references
objectToDelete.getTags().clear();
EcoreUtil.remove(objectToDelete)
transaction.commit();
} finally {
transaction.close();
}
And now the exception:
[ERROR] java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'I2342342' defined on 'MYFILE_TAGS_LIST'.
org.eclipse.net4j.db.DBException: java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'I2342342' defined on 'MYFILE_TAGS_LIS'.
at org.eclipse.net4j.db.DBUtil.executeBatch(DBUtil.java:908)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeShiftsDown(NonAuditListTableMapping.java:436)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeShifts(AbstractBasicListTableMapping.java:777)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeShifts(NonAuditListTableMapping.java:424)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeResultToDatabase(AbstractBasicListTableMapping.java:642)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeResultToDatabase(NonAuditListTableMapping.java:362)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeListDeltas(AbstractBasicListTableMapping.java:229)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping.processDelta(NonAuditListTableMapping.java:227)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping$FeatureDeltaWriter.visit(HorizontalNonAuditClassMapping.java:602)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDOListFeatureDeltaImpl.accept(CDOListFeatureDeltaImpl.java:613)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl.accept(CDORevisionDeltaImpl.java:389)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl.accept(CDORevisionDeltaImpl.java:371)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping$FeatureDeltaWriter.doProcess(HorizontalNonAuditClassMapping.java:561)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractHorizontalClassMapping$AbstractFeatureDeltaWriter.process(AbstractHorizontalClassMapping.java:1235)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping.writeRevisionDelta(HorizontalNonAuditClassMapping.java:511)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisionDelta(DBStoreAccessor.java:541)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisionDeltas(DBStoreAccessor.java:527)
at org.eclipse.emf.cdo.spi.server.StoreAccessor.doWrite(StoreAccessor.java:126)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.doWrite(DBStoreAccessor.java:882)
at org.eclipse.emf.cdo.spi.server.StoreAccessorBase.write(StoreAccessorBase.java:144)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext.writeAccessor(TransactionCommitContext.java:1894)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext.write(TransactionCommitContext.java:839)
at org.eclipse.emf.cdo.spi.server.InternalCommitContext$1.runLoop(InternalCommitContext.java:49)
at org.eclipse.emf.cdo.spi.server.InternalCommitContext$1.runLoop(InternalCommitContext.java:1)
at org.eclipse.net4j.util.om.monitor.ProgressDistributor.run(ProgressDistributor.java:95)
at org.eclipse.emf.cdo.internal.server.Repository.commitUnsynced(Repository.java:1522)
at org.eclipse.emf.cdo.internal.server.Repository.commit(Repository.java:1510)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicatingCommit(CommitTransactionIndication.java:283)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:96)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CDOServerIndicationWithMonitoring.indicating(CDOServerIndicationWithMonitoring.java:118)
at org.eclipse.net4j.signal.IndicationWithMonitoring.indicating(IndicationWithMonitoring.java:62)
After that the database can be unusable.
What are we missing? Do we somesthing wrong?
We tried the parameter "forceZeroBasedIndex" https://bugs.eclipse.org/bugs/show_bug.cgi?id=445443 .
This allows to delete more elements but the error does occure.
And it has the drawback that you can not change the parameter on en existing repository.
Thanks for your time.
Cheers
/Alexander
[Updated on: Mon, 27 February 2023 13:09] Report message to a moderator |
|
|
|
| Re: [CDO] Delete EObjects and BLOBs [message #1857787 is a reply to message #1857642] |
Sat, 25 February 2023 19:15  |
|
Regarding your Issue 2):
a) Is your MyFile::tags a containment reference? If not, if it's a cross-reference, is it bidirectional? I'm trying to understand why you're explicietly calling objectToDelete.getTags().clear()...
b) This index 'I2342342' defined on 'MYFILE_TAGS_LIST', what columns does it comprise?
c) When you say "the database can be unusable", how exactly does this problem manifest itself?
d) Even without having a clue about the cause of the problem, I don't think that the 'forceZeroBasedIndex' property is related.
e) In another post you mentioned a Derby DB. Is this also about/with Derby?
f) What CDO version are you using?
Answers to these questions would help me to understand and ideally repoduce your problem.
BTW., when you open your CDO transaction in a try-with-resources block, I don't think that you need to close the transaction at the end of that block ;-)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
|
|
| Re: [CDO] Delete EObjects and BLOBs [message #1857814 is a reply to message #1857789] |
Mon, 27 February 2023 13:37 |
Alexander Stenzer
Messages: 9
Registered: April 2022 |
Junior Member |
|
|
Hi Eike,
> 1) What you observe is the expected behavior. The actual content of a CDOBinaryResource or CDOTextResource, i.e., the CDOBlob or CDOClob, is (can be) shared across different objects of the repository. That's similar to Git blobs. Currently there's no cleanup of orphaned Lobs implemented in CDO, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=565687 .
Thanks for the links. The solution in this post could solve our leftovers.
> 2) I've never heard that an exception before the commit of the JDBC connection can lead to a corrupt DB, but I'd have to be able to reproduce the problem. So please submit a bugzilla and attach complete stack traces (the one you showed here seems to be truncated before the causes are shown). Thanks ;-)
The database is not corrupt in the view of the database mangement system, but CDO has problems to read the containment list, which results in timeout exceptions during first load of the data. Therefore my assumption is, there are wrong index entries in the database tables. But I did not track it down. I will help to solve the probleme as much I can. Bugzilla
>a) Is your MyFile::tags a containment reference? If not, if it's a cross-reference, is it bidirectional? I'm trying to understand why you're explicietly calling objectToDelete.getTags().clear()...
The error can occure on containment references and corss-reference. In the example in the post which is simplified a lot it is a bidirectional corss-reference.
I have to clear all tags to remove dangling references.
>b) This index 'I2342342' defined on 'MYFILE_TAGS_LIST', what columns does it comprise?
The index is on the column CDO_SOURCE combinded with CDO_IDX. So I think it has soming to to with the order of elements in EList which are stored in the CDO_IDX column.
>c) When you say "the database can be unusable", how exactly does this problem manifest itself?
In our application it results in a timeout exception during start up. And after that we are not able to read or write certain ELists.
>d) Even without having a clue about the cause of the problem, I don't think that the 'forceZeroBasedIndex' property is related.
Ok, what we observed is if forceZeroBasedIndex is set to true on a new database it takes longer until the error occures. The problem is that we could not find a way to force the error to occures. It looks like a probleme which occures after a certain amount of operations on the cdo repository.
> e) In another post you mentioned a Derby DB. Is this also about/with Derby?
We use Derby as database yes. But we tested it also with Postgres. And there was the some error. Therefore I think it has noting to do with the database managment system its self.
>f) What CDO version are you using?
We are using 4.17. I know it is quite old. We are migrating to the new version 4.21 or 4.22.
>g) When you say "1000 elements", element means Tag, right? Not MyFile?
The error can occure on both elements. It is always if we remove something from a EList. Sometimes it is a containment reference sometimes it is corss-refernce.
>h) How did you manage to open your CDOTransaction in a try-with-resources block, given that CDOTransaction does not extend AutoCloseable?
> When you open your CDO transaction in a try-with-resources block, I don't think that you need to close the transaction at the end of that block ;-)
Sorry that was my fault. We wrap the CDO transaction therefore we can use try-with-resources with auto close. I updated the post to remove that.
We try on our side to create a test case to repoduce the error. We are also writing a export of our data if the repository is corrupt and reimport it into a new one.
Cheers
/Alexander
|
|
|
| Re: [CDO] Delete EObjects and BLOBs [message #1858391 is a reply to message #1857814] |
Fri, 31 March 2023 14:56 |
Alexander Stenzer
Messages: 9
Registered: April 2022 |
Junior Member |
|
|
Hello Eike,
i did a lot of test to ensuse that the error is not part of our code.
I now did a showcase. There error always occures on the same element. But we need a certain amount of elements.
I attached our TestApp. It creates 750 Files and Tags.
On element 54 started by 1 it allways fails deleteing elements in a EList.
See Exception:
org.eclipse.emf.cdo.util.CommitException: Rollback in DBStore: org.eclipse.net4j.db.DBException: java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'I1680272084714_23' defined on 'FILE_TAGS_LIST'.
at org.eclipse.net4j.db.DBUtil.executeBatch(DBUtil.java:908)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeShiftsDown(NonAuditListTableMapping.java:436)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeShifts(AbstractBasicListTableMapping.java:777)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeShifts(NonAuditListTableMapping.java:424)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeResultToDatabase(AbstractBasicListTableMapping.java:642)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping$ListDeltaWriter.writeResultToDatabase(NonAuditListTableMapping.java:362)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractBasicListTableMapping$AbstractListDeltaWriter.writeListDeltas(AbstractBasicListTableMapping.java:229)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.NonAuditListTableMapping.processDelta(NonAuditListTableMapping.java:227)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping$FeatureDeltaWriter.visit(HorizontalNonAuditClassMapping.java:602)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDOListFeatureDeltaImpl.accept(CDOListFeatureDeltaImpl.java:613)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl.accept(CDORevisionDeltaImpl.java:389)
at org.eclipse.emf.cdo.internal.common.revision.delta.CDORevisionDeltaImpl.accept(CDORevisionDeltaImpl.java:371)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping$FeatureDeltaWriter.doProcess(HorizontalNonAuditClassMapping.java:561)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractHorizontalClassMapping$AbstractFeatureDeltaWriter.process(AbstractHorizontalClassMapping.java:1235)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalNonAuditClassMapping.writeRevisionDelta(HorizontalNonAuditClassMapping.java:511)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisionDelta(DBStoreAccessor.java:541)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisionDeltas(DBStoreAccessor.java:527)
at org.eclipse.emf.cdo.spi.server.StoreAccessor.doWrite(StoreAccessor.java:126)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.doWrite(DBStoreAccessor.java:882)
at org.eclipse.emf.cdo.spi.server.StoreAccessorBase.write(StoreAccessorBase.java:144)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext.writeAccessor(TransactionCommitContext.java:1894)
at org.eclipse.emf.cdo.internal.server.TransactionCommitContext.write(TransactionCommitContext.java:839)
at org.eclipse.emf.cdo.spi.server.InternalCommitContext$1.runLoop(InternalCommitContext.java:49)
at org.eclipse.emf.cdo.spi.server.InternalCommitContext$1.runLoop(InternalCommitContext.java:1)
at org.eclipse.net4j.util.om.monitor.ProgressDistributor.run(ProgressDistributor.java:95)
at org.eclipse.emf.cdo.internal.server.Repository.commitUnsynced(Repository.java:1522)
at org.eclipse.emf.cdo.internal.server.Repository.commit(Repository.java:1510)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicatingCommit(CommitTransactionIndication.java:283)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:96)
at org.eclipse.emf.cdo.server.internal.net4j.protocol.CDOServerIndicationWithMonitoring.indicating(CDOServerIndicationWithMonitoring.java:118)
at org.eclipse.net4j.signal.IndicationWithMonitoring.indicating(IndicationWithMonitoring.java:62)
at org.eclipse.net4j.signal.IndicationWithResponse.doExtendedInput(IndicationWithResponse.java:99)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:423)
at org.eclipse.net4j.signal.IndicationWithResponse.execute(IndicationWithResponse.java:72)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:329)
at org.eclipse.net4j.signal.SignalReactor.runSync(SignalReactor.java:74)
at org.eclipse.net4j.signal.Signal.run(Signal.java:201)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.sql.BatchUpdateException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'I1680272084714_23' defined on 'FILE_TAGS_LIST'.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.derby.impl.jdbc.Util.newBatchUpdateException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)
at org.eclipse.net4j.db.jdbc.DelegatingPreparedStatement.executeBatch(DelegatingPreparedStatement.java:354)
at org.eclipse.net4j.db.DBUtil.executeBatch(DBUtil.java:883)
... 39 more
The test app is a OSGI Application with an embedded Derby Database. It will start the test automatically.
I hope we can fix this error, because if the error occures the cdo database is done.
Cheers
/Alexander
[Updated on: Tue, 04 April 2023 15:37] Report message to a moderator |
|
|
Powered by
FUDForum. Page generated in 0.20976 seconds
] 
Search
Help
Register
Login
Home
