Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [stem-dev] STEM Feature for single KML file animation -- Dev Questions

Hey James,

No problem -- thanks for your help; developing on STEM has actually been quite enjoyable because STEM is such a mature platform. I originally was working with Bioclipse and R for Gene Ontology, but STEM is alot more stable and easy to develop upon.

I'd like to finish the Google Earth refactoring as a Beta release by the end of the month. I'd be happy to put all this info on the Wiki.

Just drop me a link to a page where I should write it up, and I will put all the information on the Wiki site.

Also: The way the new Google Earth integration works is a 'two-step' process right now. STEM writes a KML animation file as output, I PK-zip the file and change the extension to .kmz, then I double-click the .kmz file and Google Earth launches.

I think a 'two-step' process is better for the moment... I think it will be easiest to have the Google Earth KML animation writer work along the same lines as the STEM CSV logger, where the output is de-coupled from secondary analysis.

I will send you the video so you can see the results.

I have a pretty decent laptop (Core i7 / 8GB RAM) and it runs GE KML animation simulations fine as long as the raw kml file is under about 400 megs. I think you're right that a long-term solution might be to host the global polygons (or shape files) online, so the processing gets offloaded from the local machine. Good idea on that one.


  I'm getting ready to move
On 2014-10-08 10:02, James Kaufman wrote:
Thank you Alex!!
We really appreciate your efforts here!
Would you be willing to create a stem wiki page on kml/google earth
integration documenting what you have been able to do for others?

I'm not sure about your application but another short term fix might
be to run STEM and generate the kml files as step 1 and play back on
google earth as step 2.

or,... maybe get a massive server with lots of cores so you can up the
number of threads.

Anyway please let us know. Thank you for looking at the KML display

Best Regards,

 IBM Almaden Research Center, 650 Harry Rd.
 San Jose, CA 95120-6099
 email: jhkauf@xxxxxxxxxx
 phone: (408) 927-2477 (tie 457-2477)

 From: alex@xxxxxxxxxxxxxx
 To: stem-dev@xxxxxxxxxxx,
 Cc: matt.davis@xxxxxxxxxxx, James Kaufman/Almaden/IBM@IBMUS
 Date: 10/08/2014 08:18 AM
 Subject: re: STEM Feature for single KML file animation -- Dev


Hello James,

 No worries... I'm sure you're very busy. I appreciate the time you
 taken to explain the parameters.

 A. Thanks. I will see if I can extract the population data using the
 Java code you've included.

 B. Also great info here. I'll hunt down the Sequencer object to get
 the start and end dates.

 C. I wrote: "Why is my simulation's 'cycle' counter jumping

 You were absolutely correct... it turned out to be a threading
 I resolved it by doing the following:

 -Increased the STEM simulation step time from 250ms to 750ms
 -Set the STEM Process Priority to High in the OS
 -Allocated STEM 2GB/2GB of memory on launch
 -Slowed the new KML writer so it writes only once per 5 cycles
 than each cycle).
 -Reduced the complexity of the simulation... the max output of the
 file can only be about 400MB.

 I learned that if the single KML file is greater than about 400 MB,
 Google Earth will crash when loading it.

 So unfortunately, this limits the 'granularity' of KML file
 There is probably a way around this... at present, I write the
 to represent a county repeatedly (at each simulation step). This
 county-level data cause the KML animation file to be very large
 (hundreds of megs) .. There is lots of redundant data since the same
 polygon coordinates gets written over and over, repeated on each

 I see three workarounds to the large KML output file sizes... The key

 will be to eliminate the KML polygon data redundancy over

 C.1. Utilize image overlays (jpegs) for countries, regions and
 etc... rather than having Google Earth draw STEM's region polygons
 and over, use colored JPG overlays, shape files, something like that.

 C.2. OR: Have GE import STEM county-level coordinates as a resource
 'as-needed', and use Google Earth's 'Feature' loading capability to
 cache only the data in view. This way it doesn't all get loaded into
 memory at once, and might eliminate redundancy.

 C.3. OR: Quick and Dirty solution: Reduce the number of points
 (resolution) that is used to represent county data when writing to
 KML file by Ramer–Douglas–Peucker algorithm. So the county
 will not be so complex in the KML, and will take up less space. ie.
 Reduce a county polygon from ~50 coordinate pairs to 10 coordinate

 D. Thanks. I think I will try to use your standard methods (the
 object, scaling, etc) for coloring the KML output polygons. That way
 everything is consistent across the STEM platform. I want the KML
 animation to support log scaling , just like the GUI does.

 E. Okay. I'm pretty sure the STEM GE Servlet stuff is broken on my
 Windows 7 x64 environment. For some reason, the web server has not
 run locally (nothing is bound on port 8080 or whatever it is). I
 this functionality might be broken or deprecated in the current
 of Java and Google Earth.

 The existing GE functionality in the latest STEM release was able to
 two major tasks (1) write a directory with one kml file per iteration
 numbered, or (2) launch Google Earth to load a control.kml file once
 iteration. I don't think I was able to get the GE Servlet stuff
 working at all, but I will test again.

 F. > What is the difference between KmlDisplay, KmlDisplayCustom, and

 KmlDisplaySelection? Is any of this functionality deprecated? "Some
 these are really old and may go back to older versions of google

 Okay, I may try to re-factor some of this code. I think
 KmlDisplaySelection was meant to display a sub-set of a geographic
 to KML, but I don't know if it's working. I think some of this code
 be pulled out, fixed, or refactored, especially if the Servlet stuff
 actually broken for everyone.

 Anyway, I will keep digging and clean up the code as I go. I'd like
 pull out or fix any deprecated features that are broken.

 Also, I'm tempted to convert the KML write functionality to a Logger
 object, unless there is a reason not to do that...

 Alex (aka Vincent R)

 On 2014-10-07 16:38, James Kaufman wrote:
 > Vincent,
 > Sorry for the slow response.
 > Re your questions
 > > A. Where is the 'county' population data stored? For example, how
 > do I go from a key like 'US-NY-36039' (county identifier) to
 > the original population in the county and the current population in
 > the county at a time-step? (to determine number of fatalities , or
 > total number of Infections as an absolute value rather than
 > if you have the node you can use something like the following
 > EList<NodeLabel> labelList = node.getLabels();
 > FOR(NodeLabel nl:labelList) {
 > IF(nl INSTANCEOF PopulationLabel) {
 > PopulationLabel pl = (PopulationLabel) nl;
 > DOUBLE population = pl.getCurrentPopulationValue().getCount();
 > }
 > }
 > You should be able to get the node from the graph. (ie
 > graph.getNode(nodeURI))
 > > B. Where can I find a simulation's actual 'start' and 'end' dates
 > in String or DateTime format? What object and where is it?
 > Get the Sequencer from the scenerio (from the simulation). The
 > sequencer has the start and end dates.
 > the sequencer has method getStartTime and getEndTime
 > they return "STEMtime" objects but STEMtime.getDate() gives you a
 > Date.
 > > C. Why is my simulation's 'cycle' counter jumping around at the
 > beginning? It starts at cycle=0, then goes to cycle=2, then to
 > cycle=7,8,9,10 etc?
 > The gui runs in it's own thread so you are running either a big or
 > small simulation. On the menu bar you can go to
 > then >STEM>Simulation Managment and either increase the simulation
 > delay so the gui keeps up. Or decrease it if you need.
 > > D. Is there a 'proper' way to go about scaling an 'I' value to a
 > polygon color? Should I just use the Aspect object? Any tips on
 > I think it just depends what you are trying to show. We usually
 > normalize I by the population but you might be interested in small
 > values at the start of an epidemic so you might need a scale factor
 > log scaling.
 > > E. Are there any known bugs with the GE Servlet stuff? I'm
 > some Servlet exceptions (can't connect to local web server). Is
 > feature broken (the network pipe)? It's not
 > important -- just curious.
 > We have not run this in a very long time. Matt, any ideas on this?
 > > F. Why are there multiple classes which seem to be involved in
 > writing KML files... What is the difference between KmlDisplay,
 > KmlDisplayCustom, and KmlDisplaySelection? Is any of this
 > functionality deprecated? Some of these are really old and may go
 > to older versions of google earth.
 > I'm sorry not to be more helpful....
 > Best Regards,
 > Jamie
 > IBM Almaden Research Center, 650 Harry Rd.
 > San Jose, CA 95120-6099
 > email: jhkauf@xxxxxxxxxx
 > phone: (408) 927-2477 (tie 457-2477)

Back to the top