Re: [emf-dev] [Galileo] Failed for build 2009-05-19 due to invalid site.

[Eike Stepper <stepper@xxxxxxxxxx>]

Nick Boldt schrieb:
> I suspect the reason for these failures is that the site*.xml files
> used to generate the metadata contain two </site> tags, making the
> files invalid XML.
>
> The only reason I can think of for how this happens is concurrent
> attempts to write to the same file at the same time. (The location of
> the first </site> varies within the file, while the second one is
> always at the end.)
>
> [1]http://emft.eclipse.org/promo_logs/promo_log_cdo_2.0.0.S200905191440_2009-05-19-16.06.04.txt
>
> [2]http://emft.eclipse.org/promo_logs/promo_log_net4j_2.0.0.S200905191048_2009-05-19-16.05.39.txt
>
>
> Search for "components' category*.xml into single site.xml" and you'll
> see that on the build server, these happened a few seconds apart (no
> problem) but on download.eclipse, both changes to the site*.xml files
> started at 16:10:24. Despite starting the promotes a minute apart,
> they did this crucial step at the exact same time. And then two
> threads began  generating the same two p2 metadata files within a
> second of each other, probably colliding along the way.
>
> So, to hopefully avoid this (since people continue to think that an
> XML file is a multi-threaded database and can just fire multiple
> promotes concurrently), 
I can not speak for others, but I for my part did not even think about
single (XML) files being shared by many clients ;-) Maybe I'm even too
involved in concurrent programming so that I tend to interpret the word
"client" too literally. Whenever I'm considered a client of some service
(here: the build system) I usually expect the service to care for
concurrency control. Are you suggesting we use the wiki for this purpose? :P

An XML file is certainly not a multi-threaded database, but even in a
multi-threaded database no two clients can write to the same record at
the same time. In general the trick is not to enable concurrent access
to exclusive resources but rather to control it in a way that it
internally happens sequentially, either by letting the second client
wait or fail.

A file system as a whole is more like a multi-threaded database and I'm
pretty sure that most operating systems prevent concurrent writes to
files. That leads to the assumption that, if shared files get corrupted,
the application (here: the build system) built on top of that file
system is not properly designed. If an application modifies a shared
file by opening and closing it multiple times it should use locks o
demarcate these logical transactions.

Anyway, I apologize that I was ignorant and caused others (and me) a
headache! ;-)

> I've added a line to the generator (buildUpdateSiteXML.sh) that scrubs
> out any existing </site> tags BEFORE appending the closing tag at the
> end of the file.
>
> I can't stop people from abusing the site*.xml files 
You can: use locks!

> (and therefore corrupting the p2 metadata), but I can at least try to
> make the resulting file *valid* XML.
>
> If you have multiple updates to publish, please do them in series, not
> in parallel. Use the IRC channel #eclipse-modeling to coordinate your
> attacks.
Oh, not the wiki :P

Cheers
/Eike

----
http://thegordian.blogspot.com


>
> Nick
>
> David Williams wrote:
>> The following error occured when building Galileo:
>>
>> org.eclipse.equinox.internal.provisional.p2.core.ProvisionException:
>> No repository found at
>> http://download.eclipse.org/modeling/emf/updates/milestones/.
>>
>> Check the log file for more information:
>> https://build.eclipse.org/hudson/view/BuckyBuild/job/galileo.runBuckyBuild/268/console
>>
>>
>