Eclipse Community Forums: BIRT » Reading delimited data from web

Home » Archived » BIRT » Reading delimited data from web(How to read '|' delimited data from web?)

Reading delimited data from web [message #650647]

Tue, 25 January 2011 14:52

Eclipse User

Is there a way how to read delimited data from http protocol using BIRT? The sample dataset I would like to use is here ( http://www.cnb.cz/cs/financni_trhy/devizovy_trh/kurzy_devizo veho_trhu/rok.txt?rok=2011 - FX history).

I think the best way would be to use scripted data source, but from examples I have found I was not able to create anything usable.

There are other ways possible, like:
- use some kind of csv/plain-text jdbc driver
- use ETL solution to extract data from web and than store data in the database (this works - using Clover does the job well, but I think there must be some simple solution using BIRT only)

I have the following questions:
- do you know any example, how scripted data source is used to read delimited plain text data from web?
- is there any plain text jdbc driver you can use in BIRT to extract data from web?
- any other simple solution?

Thank you for any ideas.

Tomas

Re: Reading delimited data from web [message #650827 is a reply to message #650647]

Wed, 26 January 2011 12:40

Eclipse User

Tomas

This is pretty simple to do with a scripted dataset. Add one to your
report and put this in the open method:

//open
importPackage(Packages.java.net);
importPackage(Packages.java.io);
dataURL = new
URL(" http://www.cnb.cz/cs/financni_trhy/devizovy_trh/kurzy_devizo veho_trhu/rok.txt?rok=2011");
dataConnection = dataURL.openConnection();
inp = new BufferedReader( new
InputStreamReader(dataConnection.getInputStream()));
inputLine = inp.readLine(); //throw away header

Put this in for the fetch
//fetch
importPackage(Packages.java.lang);
var inputLine;
if ((inputLine = inp.readLine()) == null)return false;
var myrowarray =inputLine.split("\\|");
row["col1"] = myrowarray[2];
return true

Put this in for the close
//close
inp.close();

In this example I only did one column. The full report is attached. If
this is a public url that will stay up I would like to put the example
on BIRT exchange devshare. Is that ok?

Jason

<?xml version="1.0" encoding="UTF-8"?>
<report xmlns="http://www.eclipse.org/birt/2005/design" version="3.2.22"
id="1">
<property name="createdBy">Eclipse BIRT Designer Version
2.6.1.v20100902 Build <2.6.1.v20100915-1750></property>
<property name="units">in</property>
<property name="iconFile">/templates/blank_report.gif</property>
<property name="bidiLayoutOrientation">ltr</property>
<property name="imageDPI">96</property>
<data-sources>
<script-data-source name="Data Source" id="7"/>
</data-sources>
<data-sets>
<script-data-set name="Data Set" id="8">
<list-property name="resultSetHints">
<structure>
<property name="position">0</property>
<property name="name">col1</property>
<property name="dataType">string</property>
</structure>
</list-property>
<list-property name="columnHints">
<structure>
<property name="columnName">col1</property>
</structure>
</list-property>
<structure name="cachedMetaData">
<list-property name="resultSet">
<structure>
<property name="position">1</property>
<property name="name">col1</property>
<property name="dataType">string</property>
</structure>
</list-property>
</structure>
<property name="dataSource">Data Source</property>
<method name="open"><![CDATA[importPackage(Packages.java.net);
importPackage(Packages.java.io);

dataURL = new
URL(" http://www.cnb.cz/cs/financni_trhy/devizovy_trh/kurzy_devizo veho_trhu/rok.txt?rok=2011");
dataConnection = dataURL.openConnection();
inp = new BufferedReader( new
InputStreamReader(dataConnection.getInputStream()));
inputLine = inp.readLine(); //throw away header
]]></method>
<method
name="fetch"><![CDATA[importPackage(Packages.java.lang);
var inputLine;
if ((inputLine = inp.readLine()) == null)return false;
var myrowarray =inputLine.split("\\|");
row["col1"] = myrowarray[2];
return true

]]></method>
<method name="close"><![CDATA[inp.close();]]></method>
</script-data-set>
</data-sets>
<styles>
<style name="report" id="4">
<property name="fontFamily">sans-serif</property>
<property name="fontSize">10pt</property>
</style>
<style name="crosstab-cell" id="5">
<property name="borderBottomColor">#CCCCCC</property>
<property name="borderBottomStyle">solid</property>
<property name="borderBottomWidth">1pt</property>
<property name="borderLeftColor">#CCCCCC</property>
<property name="borderLeftStyle">solid</property>
<property name="borderLeftWidth">1pt</property>
<property name="borderRightColor">#CCCCCC</property>
<property name="borderRightStyle">solid</property>
<property name="borderRightWidth">1pt</property>
<property name="borderTopColor">#CCCCCC</property>
<property name="borderTopStyle">solid</property>
<property name="borderTopWidth">1pt</property>
</style>
<style name="crosstab" id="6">
<property name="borderBottomColor">#CCCCCC</property>
<property name="borderBottomStyle">solid</property>
<property name="borderBottomWidth">1pt</property>
<property name="borderLeftColor">#CCCCCC</property>
<property name="borderLeftStyle">solid</property>
<property name="borderLeftWidth">1pt</property>
<property name="borderRightColor">#CCCCCC</property>
<property name="borderRightStyle">solid</property>
<property name="borderRightWidth">1pt</property>
<property name="borderTopColor">#CCCCCC</property>
<property name="borderTopStyle">solid</property>
<property name="borderTopWidth">1pt</property>
</style>
</styles>
<page-setup>
<simple-master-page name="Simple MasterPage" id="2">
<page-footer>
<text id="3">
<property name="contentType">html</property>
<text-property
name="content"><![CDATA[<value-of>new Date()</value-of>]]></text-property>
</text>
</page-footer>
</simple-master-page>
</page-setup>
<body>
<table id="9">
<property name="dataSet">Data Set</property>
<list-property name="boundDataColumns">
<structure>
<property name="name">col1</property>
<text-property name="displayName">col1</text-property>
<expression name="expression"
type="javascript">dataSetRow["col1"]</expression>
<property name="dataType">string</property>
</structure>
</list-property>
<column id="18"/>
<header>
<row id="10">
<cell id="11">
<label id="12">
<text-property name="text">col1</text-property>
</label>
</cell>
</row>
</header>
<detail>
<row id="13">
<cell id="14">
<data id="15">
<property name="whiteSpace">nowrap</property>
<property
name="resultSetColumn">col1</property>
</data>
</cell>
</row>
</detail>
<footer>
<row id="16">
<cell id="17"/>
</row>
</footer>
</table>
</body>
</report>

Is that a public url?
On 1/25/2011 2:52 PM, TomasGreif wrote:
> Is there a way how to read delimited data from http protocol using BIRT?
> The sample dataset I would like to use is here
> ( http://www.cnb.cz/cs/financni_trhy/devizovy_trh/kurzy_devizo veho_trhu/rok.txt?rok=2011
> - FX history).
> I think the best way would be to use scripted data source, but from
> examples I have found I was not able to create anything usable.
>
> There are other ways possible, like:
> - use some kind of csv/plain-text jdbc driver
> - use ETL solution to extract data from web and than store data in the
> database (this works - using Clover does the job well, but I think there
> must be some simple solution using BIRT only)
>
> I have the following questions:
> - do you know any example, how scripted data source is used to read
> delimited plain text data from web?
> - is there any plain text jdbc driver you can use in BIRT to extract
> data from web?
> - any other simple solution?
>
> Thank you for any ideas.
>
> Tomas

Re: Reading delimited data from web [message #651097 is a reply to message #650827]

Thu, 27 January 2011 13:16

Eclipse User

Thank you Jason, this works!

This is a public URL (Czech National Bank - daily FX rates) and I believe it should be stable. (You can change the only one parameter at the URL end and you will get data for different year)

Tomas

Re: Reading delimited data from web [message #651121 is a reply to message #651097]

Thu, 27 January 2011 16:14

Eclipse User

If its ok I will add a bit more to the example and then post it to birt
exchange.

On 1/27/2011 1:16 PM, TomasGreif wrote:
> Thank you Jason, this works!
>
> This is a public URL (Czech National Bank - daily FX rates) and I
> believe it should be stable. (You can change the only one parameter at
> the URL end and you will get data for different year)
>
> Tomas

Re: Reading delimited data from web [message #651176 is a reply to message #651121]

Fri, 28 January 2011 03:01

Eclipse User

Jason, it is completely ok if you use it on birt-exchange (anyway, it is your solution).

Maybe you can consider also the following typical issues (at least for me) when reading such kind of data:
- solving different locale (like different decimal separator - "," instead of "." in the example above, or different date format)
- skipping more rows at the beginning (another stable dataset - interbank interest rates, http://www.cnb.cz/cs/financni_trhy/penezni_trh/pribor/rok.tx t?year=2011)

Re: Reading delimited data from web [message #651275 is a reply to message #651176]

Fri, 28 January 2011 11:36

Eclipse User

Thanks. I posted it this morning on Birt Exchange.

Jason

On 1/28/2011 3:02 AM, TomasGreif wrote:
> Jason, it is completely ok if you use it on birt-exchange (anyway, it is
> your solution).
>
> Maybe you can consider also the following typical issues (at least for
> me) when reading such kind of data:
> - solving different locale (like different decimal separator - ","
> instead of "." in the example above, or different date format)
> - skipping more rows at the beginning (another stable dataset -
> interbank interest rates,
> http://www.cnb.cz/cs/financni_trhy/penezni_trh/pribor/rok.tx t?year=2011)

Previous Topic:	Locate a java class
Next Topic:	BIRT developer wanted

Goto Forum:

-=] Back to Top [=-

Current Time: Mon Nov 03 22:20:35 EST 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter