Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [smila-user] SMILA in a Cluster

Hi Lorenzo,

 

there’s nothing special to define in a job to have its work shared, if you setup your cluster as described in the documentation.

 

However, the unit to share are tasks, and a task corresponds to a bulk (of records).

So, if you have only a small amount of data, you have to limit the size of the bulks running through the workflow.

 

Have a look at the bulkbuilder, and its configuration:

http://wiki.eclipse.org/SMILA/Documentation/Bulkbuilder#Configuration

 

You could define very low limits if your amount of data is small, e.g. bulkLimitSize=1

 

Regards,

Andreas

 

Von: smila-user-bounces@xxxxxxxxxxx [mailto:smila-user-bounces@xxxxxxxxxxx] Im Auftrag von Lorenzo Eccher
Gesendet: Freitag, 10. Januar 2014 17:58
An: smila-user@xxxxxxxxxxx
Betreff: [smila-user] SMILA in a Cluster

 

Hallo SMILAers
As you read in my old emails, I am testing the new features of SMILA, working in a cluster.
After having tested the provided processes such as file system crawling and web crawling I tried do run a stupid workflow that reads records from a text file and index them into solr.

As expected the job doesn't involve directly each machine (just the job is created but the records are not shared). I suppose that the process must be designed in a specific way to be shared in the cluster. Isn't it?

Is there possible to have a description about the right design of the process?

How should I use the objectstore fot implementing precisely the feature?

Thank you.

--


Lorenzo Eccher
lorenzo.eccher@xxxxxx
   (+39) 0461 312 306
Engineering Ingegneria informatica s.p.a
www.eng.it

ENGINEERING Society and Territory Trento Research Office
EIT-ITC Labs, Trento node

ESTRO Lab at FBK building
via Sommarive, 18
Povo - 38123 Trento

  Le informazioni trasmesse sono destinate esclusivamente alla persona o alla società in indirizzo e sono da intendersi confidenziali e riservate. Ogni trasmissione, inoltro, diffusione o altro uso di queste informazioni a persone o società differenti dal destinatario è proibita. Se ricevete questa comunicazione per errore, contattate il mittente e cancellate le informazioni da ogni computer.
  The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
  Die Informationen in dieser E-Mail-Mitteilung sind vertraulich und deren Verbreitung in jeglicher Art oder Form ist untersagt. Sollten Sie diese Nachricht irrtümlich erhalten haben, ersuchen wir Sie, sofort den Absender darüber zu informieren und die Mail zu löschen.


Back to the top