Hi folks,
Sorry for not updating this blog for a while; been super busy getting the latest mapViewer released. Now that MapViewer version 11.1.1.7.0 (PS6) is officially out, be sure to check out the latest and greatest version here:
http://www.oracle.com/technetwork/middleware/mapviewer/downloads/index.html
Some key new features include our brand new, written from scratch HTML5 JavaScript mapping API, a brand new spatial data editor, a new sample data set, a standalone samples app, a Glassfish-based Quick Start kit, as well as many other cool new features.
We are also glad to announce that we now have an official Oracle MapViewer blog hosted by none other than Oracle's own blog server. Please be sure to check us out at the new address:
https://blogs.oracle.com/oracle_maps_blog
Key team members and managers are in the process of getting their account approved and will be joining the new blog soon. You will sure see a much more active blog for sure.
Thanks and till our next post at the new site,
LJ
Tuesday, April 9, 2013
Wednesday, September 21, 2011
Finding sales offices within a certain distance of a customer
In this post lets do a simple demo that shows all the customers (from the MVDEMO's customers table) on the map. Now, when you click on a customer marker, I want to display all the cities that are within 50 miles radius of the customer. I mentioned sales office in the title to make it sound more interesting, but our small MVDEMO sample data set does not have such a table, so we will use the readily available cities table instead.
Basically, we want to run a query like following when you click on any given customer marker:
select city, state_abrv, location from cities
where sdo_within_distance(location,
sdo_geometry(2001, 8307, sdo_point_type(:1, :2, null), null, null),
:3)='TRUE';
And display the query result as a new theme on the map. Note the three binding parameters in the above query. The first two should be the longitude and latitude of the customer location. The 3rd parameter is a Oracle Spatial WITHIN_DISTANCE operator parameter, typically in the form of 'distance=50 unit=mile'. So a properly bound query looks like this (you can execute it in the mvdemo schema).
select city, state_abrv, location from cities
where sdo_within_distance(location,
sdo_geometry(2001, 8307, sdo_point_type(-122, 37.5, null), null, null),
'distance=50 unit=mile')='TRUE';
We could use a dynamic-query based theme to display the above query result. But, ever mindful of safety and efficiency, we will instead create a pre-defined theme in the mvdemo schema. The trick is to add a query condition that contains these binding variables and disable the default (window-based) spatial filter for the new theme. Below is a screenshot of such a theme (cities_near_customer) and its query condition:
Now that we have the theme defined, we can work on the actual demo itself. The demo page source is found below. Just copy and paste it into an html file, save it in your web server where MapViewer is deployed, and you are all set (assuming the usual mvdemo datasource exists in the MapViewer instance).
The main things I want to point out here are:
1. It uses a mouse-click listener on the customers theme, so that we can do the magic when user clicks on a customer marker.
2. in the mouse click function (foiClick), we create a new theme-based FOI layer for the cities_near_customers theme. Then we set the 3 binding variables this theme's query condition is expecting, using setQueryParameters().
3. In the demo I want to find out all the cities that are within 50 mile distance of a customer; you can easily use a different distance by changing the value of the distanceParam variable above.
That's all for this post.
-LJ
Basically, we want to run a query like following when you click on any given customer marker:
select city, state_abrv, location from cities
where sdo_within_distance(location,
sdo_geometry(2001, 8307, sdo_point_type(:1, :2, null), null, null),
:3)='TRUE';
And display the query result as a new theme on the map. Note the three binding parameters in the above query. The first two should be the longitude and latitude of the customer location. The 3rd parameter is a Oracle Spatial WITHIN_DISTANCE operator parameter, typically in the form of 'distance=50 unit=mile'. So a properly bound query looks like this (you can execute it in the mvdemo schema).
select city, state_abrv, location from cities
where sdo_within_distance(location,
sdo_geometry(2001, 8307, sdo_point_type(-122, 37.5, null), null, null),
'distance=50 unit=mile')='TRUE';
We could use a dynamic-query based theme to display the above query result. But, ever mindful of safety and efficiency, we will instead create a pre-defined theme in the mvdemo schema. The trick is to add a query condition that contains these binding variables and disable the default (window-based) spatial filter for the new theme. Below is a screenshot of such a theme (cities_near_customer) and its query condition:
Now that we have the theme defined, we can work on the actual demo itself. The demo page source is found below. Just copy and paste it into an html file, save it in your web server where MapViewer is deployed, and you are all set (assuming the usual mvdemo datasource exists in the MapViewer instance).
<html>
<head>
<META http-equiv="Content-Type" content="text/html" charset=UTF-8">
<TITLE>Map Cache Server/Map Client</TITLE>
<link rel="stylesheet" type="text/css" href="../t.css" />
<script language="Javascript" src="/mapviewer/fsmc/jslib/oraclemaps.js"></script>
<script language=javascript>
var mapview;
var nearestCitiesTheme;
function showMap()
{
var baseURL = "http://"+document.location.host+"/mapviewer";
var mapCenterLon = -122.45;
var mapCenterLat = 37.7706;
var mapZoom = 3;
var mpoint = MVSdoGeometry.createPoint(mapCenterLon,mapCenterLat,8307);
mapview = new MVMapView(document.getElementById("map"), baseURL);
mapview.addMapTileLayer(new MVMapTileLayer("mvdemo.demo_map"));
mapview.setCenter(mpoint);
mapview.setZoomLevel(mapZoom);
var customersTheme = new MVThemeBasedFOI('themebasedfoi1','mvdemo.customers') ;
customersTheme.setBringToTopOnMouseOver(true);
customersTheme.attachEventListener(MVEvent.MOUSE_CLICK, foiClick);
mapview.addThemeBasedFOI(customersTheme);
mapview.display();
}
function foiClick(pointOfClick,foi)
{
alert("Locating all cities within 50 mile radius.");
if(nearestCitiesTheme)
{
mapview.removeThemeBasedFOI(nearestCitiesTheme);
}
nearestCitiesTheme = new MVThemeBasedFOI('cities-theme', 'mvdemo.cities_near_customer');
//set the binding variable for this theme.
var x = foi.x; // :1
var y = foi.y; // :2
var distanceParam = "distance=50 unit=mile"; // :3
nearestCitiesTheme.setQueryParameters(x,y, distanceParam);
nearestCitiesTheme.setBringToTopOnMouseOver(true);
mapview.addThemeBasedFOI(nearestCitiesTheme);
}
</script>
</head>
<body onload="javascript:showMap()">
<h3>Oracle Maps example - finding cities closest to a customer </h3>
<div id="map" style="left:0px; top:10px;width:100%; height:60%"></div>
</body>
</html>
The main things I want to point out here are:
1. It uses a mouse-click listener on the customers theme, so that we can do the magic when user clicks on a customer marker.
2. in the mouse click function (foiClick), we create a new theme-based FOI layer for the cities_near_customers theme. Then we set the 3 binding variables this theme's query condition is expecting, using setQueryParameters().
3. In the demo I want to find out all the cities that are within 50 mile distance of a customer; you can easily use a different distance by changing the value of the distanceParam variable above.
That's all for this post.
-LJ
Monday, July 25, 2011
iOS support in Oracle Maps API
With the latest patch release (version11.1.1.5.1 found here ), MapViewer's Oracle Maps JavaScript API now fully supports iPhone/iPad map interaction using touch events such as pinch to zoom out, double tap to zoom in and other general touch events. Your existing application will automatically get these support once you upgrade to the 11.1.1.5.1 version of MapViewer's oraclemaps.js lib.
In addition to the iOS support, we are also looking into Android support in the future.
In addition to the iOS support, we are also looking into Android support in the future.
Monday, July 18, 2011
Deploying MapViewer on IBM WebSphere 7
I just finished deploying MapViewer (the 11.1.1.5.1 patch release found here ) on IBM WebSphere 7, using the main steps outlined below. Overall I would say it is just a typical process of deploying any J2EE enterprise applications on WebSphere.
1. Log into WebSphere console, and pick the server instance to deploy MapViewer.
2. Start deploying MapViewer by creating a new enterprise application.
3. Download and save the mapviewer.ear file to your local file system if not already done so. Then select this EAR file when prompted. You do not need to unpack this .ear file.
4. Go through the necessary steps by accepting all the defaults as instructed, but note the following special instructions.
At step 7 "Map context roots for web modules", make sure the context root is set to "/mapviewer".
Then at Step 8 "Map security roles to users or groups", make sure the admin user (virtuser in my case) is mapped to the map_admin_role defined by MapViewer. You can ignore the secure_maps_role which is for demo purpose only. Alternatively, you can also map other WebSphere users/groups to this map_admin_role. Later you will use the mapped user or group to log into MapViewer's Admin page.
5. You should get a summery page like below at the end of the deployment steps.
Now save the configuration and finish the deployment process. You should see MapViewer in the applications list, as illustrated below.
6. Next, you need to find where WebSphere stores the mapviewer.ear's unpacked contents. The easiest way to do so, is to start MapViewer and log in to its admin page (by clicking on the Admin button from the home page at /mapviewer). Click the Management tab, then Configuration. Near the top of the page, you will see a line indicating where the mapViewerConfig.xml file was loaded from. In my case, it is:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/AppSrv01_Cell01/OracleAS MapViewer.ear/web.war/WEB-INF/conf/mapViewerConfig.xml
This tells us that WebSphere has expanded the mapviewer.ear file into the directory:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/AppSrv01_Cell01/OracleAS MapViewer.ear/.
7. Now, we need to modify MapViewer's libraries in order for it to work properly in WebSphere 7.
8. Go to MapViewer's web.war/WEB-INF/lib folder, and physically remove the two jars: jsf-api.jar and jsf-impl.jar. This ensures all MapViewer web pages (which are written in JSF 1.0) work properly in WebSphere, since WebSphere already comes with a set of JSF 1.2 jars, and we don't want our (older) JSF jar files to conflict with these.
9. From your Oracle 11g database install, copy the two JDBC driver related jars, ojdbc5.jar and orai18n.jar, and put them in MapViewer's web.war/WEB-INF/lib folder. Alternatively, you can download these two jars from Oracle's JDBC OTN web page. I used version 11.2.0.2 jars, but earlier versions of these jars should work too.
10. Now, go to WebSphere console and restart the MapViewer application.
From this point on, you will just need to perform typical MapViewer configuration tasks such as adding a new data source et al.
Final note, if before performing step 8 above (where you removed the two jsf jars), you already tried visiting some of the MapViwer pages, then it is likely you will have seen errors related to JSP exceptions. What you need to do (after performing step 8), is go to MapViewer's web.war folder, and touch every .jspx file so that their last modified timestamp gets changed. Do the same for the .jspx files under web.war/admin folder too. This will force WebSphere to re-compile all the .jspx files (using the JSF 1.2 jars that came with WebSphere).
1. Log into WebSphere console, and pick the server instance to deploy MapViewer.
2. Start deploying MapViewer by creating a new enterprise application.
3. Download and save the mapviewer.ear file to your local file system if not already done so. Then select this EAR file when prompted. You do not need to unpack this .ear file.
4. Go through the necessary steps by accepting all the defaults as instructed, but note the following special instructions.
At step 7 "Map context roots for web modules", make sure the context root is set to "/mapviewer".
Then at Step 8 "Map security roles to users or groups", make sure the admin user (virtuser in my case) is mapped to the map_admin_role defined by MapViewer. You can ignore the secure_maps_role which is for demo purpose only. Alternatively, you can also map other WebSphere users/groups to this map_admin_role. Later you will use the mapped user or group to log into MapViewer's Admin page.
5. You should get a summery page like below at the end of the deployment steps.
Now save the configuration and finish the deployment process. You should see MapViewer in the applications list, as illustrated below.
6. Next, you need to find where WebSphere stores the mapviewer.ear's unpacked contents. The easiest way to do so, is to start MapViewer and log in to its admin page (by clicking on the Admin button from the home page at /mapviewer). Click the Management tab, then Configuration. Near the top of the page, you will see a line indicating where the mapViewerConfig.xml file was loaded from. In my case, it is:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/AppSrv01_Cell01/OracleAS MapViewer.ear/web.war/WEB-INF/conf/mapViewerConfig.xml
This tells us that WebSphere has expanded the mapviewer.ear file into the directory:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/AppSrv01_Cell01/OracleAS MapViewer.ear/.
7. Now, we need to modify MapViewer's libraries in order for it to work properly in WebSphere 7.
8. Go to MapViewer's web.war/WEB-INF/lib folder, and physically remove the two jars: jsf-api.jar and jsf-impl.jar. This ensures all MapViewer web pages (which are written in JSF 1.0) work properly in WebSphere, since WebSphere already comes with a set of JSF 1.2 jars, and we don't want our (older) JSF jar files to conflict with these.
9. From your Oracle 11g database install, copy the two JDBC driver related jars, ojdbc5.jar and orai18n.jar, and put them in MapViewer's web.war/WEB-INF/lib folder. Alternatively, you can download these two jars from Oracle's JDBC OTN web page. I used version 11.2.0.2 jars, but earlier versions of these jars should work too.
10. Now, go to WebSphere console and restart the MapViewer application.
From this point on, you will just need to perform typical MapViewer configuration tasks such as adding a new data source et al.
Final note, if before performing step 8 above (where you removed the two jsf jars), you already tried visiting some of the MapViwer pages, then it is likely you will have seen errors related to JSP exceptions. What you need to do (after performing step 8), is go to MapViewer's web.war folder, and touch every .jspx file so that their last modified timestamp gets changed. Do the same for the .jspx files under web.war/admin folder too. This will force WebSphere to re-compile all the .jspx files (using the JSF 1.2 jars that came with WebSphere).
Friday, October 8, 2010
Sending command-line admin requests to MapViewer
As you already know, MapViewer has several web pages where you can (after logging in) perform various admin tasks using XML requests. These tasks range from managing mapviewer's metadata cache, data sources, to tile layer pre-fetching.
Sometimes one may have the need to perform these tasks from a command line or script. This is especially helpful if you need to automate certain aspects of managing a MapViewer instance (such as periodically clearing its tile layer cache or metadata cache).
A while back we uploaded a simple tool that enables just that. If you are interested, please go to MapViewer's download page on OTN:
http://www.oracle.com/technetwork/middleware/mapviewer/downloads/index.html
and look for the Oracle Fusion Middleware MapViewer Admin Tool. It comes with a detailed readme describing how to use this tool to send admin requests to MapViewer server from a command line or from a script.
Basically this tool allows you to specify the user name and password for the MapViewer admin URLs, and a XML document containing the admin request itself. It will then authenticate to the MapViewer and post the request doc to the server for processing.
Sometimes one may have the need to perform these tasks from a command line or script. This is especially helpful if you need to automate certain aspects of managing a MapViewer instance (such as periodically clearing its tile layer cache or metadata cache).
A while back we uploaded a simple tool that enables just that. If you are interested, please go to MapViewer's download page on OTN:
http://www.oracle.com/technetwork/middleware/mapviewer/downloads/index.html
and look for the Oracle Fusion Middleware MapViewer Admin Tool. It comes with a detailed readme describing how to use this tool to send admin requests to MapViewer server from a command line or from a script.
Basically this tool allows you to specify the user name and password for the MapViewer admin URLs, and a XML document containing the admin request itself. It will then authenticate to the MapViewer and post the request doc to the server for processing.
Tuesday, September 14, 2010
MapViewer performance tuning tips - part 4
In this post we will discuss best practices that deal with overall MapViewer deployment and configuration. We will also briefly touch base on how to monitor a running MapViewer instance to find troubling themes.
Deployment and Configuration
MapViewer is a J2EE application that runs in all modern J2EE containers. Deploying MapViewer is no different from deploying other J2EE applications.
Ideally MapViewer should be deployed to a dedicated JVM. In Oracle iAS world, this will be a dedicated OC4J instance, and in the WebLogic Server (WLS) world, this will be a dedicated server in a domain.
The reason is simple, MapViewer, while having a small foot print itself, often requires large amount of memory and CPU resources at run time depending on the work load. So it is best to deploy MapViewer to a JVM instance that runs only MapViewer (and minimum amount of other J2EE supporting services), so that unnecessary contentions or even interruptions with other applications can be minimized. It will also be easier to monitor MapViewer's performance when it is the only application running in the JVM.
Note however there is nothing inherently wrong to deploy and run MapViewer with other J2EE applications in a shared JVM.
JVM heap size
One of the key factors that will affect mapViewer performance is the amount of Heap memory allocated to the JVM. A minimum of 512 MB is recommended. On 32 bit operating systems, the maxium heap memory should not exceed 1.5GB (or 1500 MB). A typical MapViewer install will run happily with 1GB of heap memory.
You set heap memory via JVM's -Xmx option.
Another important memory allocation parameter is the young generation size within the JVM. Young generation is a pool of memory reserved for short-lived objects. A typical map request will result in MapViewer creating many such objects (for instance, to represent geographic features in memory), so a sufficiently large young generation within the heap is crucial in preventing the JVM spending too much time on garbage collection.
The young generation size can be set via the JVM parameter:
-XX:newSize=80M
and
-XX:maxNewSize=200M
The first parameter specifies a starting young generation size of 80MB, while the second parameter indicates a maximum young generation size of 200M.
Typically the young generation size should not be more than 10 to 20 percent of the total heap size.
Spatial data cache
MapViewer lets you optionally configure an in-memory geometry data cache via the "spatial_data_cache" tag in mapViewerConfig.xml. This cache is used to buffer geometry features generated for themes whose caching mode is either set to ALL or NORMAL. (Note that you set a theme's caching mode in Map Builder's theme editor, in the Advanced tab).
When a theme's caching mode is set to ALL, MapViewer will load all of its features upon first request, and store the data in the cache. It also creates an in-memory R-Tree index on the features. All subsequent requests for this theme will be satisfied out of this cache, with no trips to the database.
When a theme's caching mode is set to NORMAL, MapViewer will store the theme's features as they are loaded. Note that MapViewer still queries the database every time such a theme is requested, but, in the event a feature is already in the cache, the un-marshaling or parsing of the database formatted geometry object is skipped, and the corresponding Java geometry object is reused out of the cache.
Our recommendation regarding theme caching, is to use caching mode ALL for small amount of themes that are based on small to medium sized tables (with couple hundred to tens of thousands of features) with static contents. And turn off NORMAL caching for all other themes. In other words, a pre-defined geometry theme should have its cache mode set to either NONE or (in rare cases) ALL.
The reason we do not recommend using the NORMAL caching mode is that it does not save much resource or time, in fact, it makes memory usage go up without much benefit at all. We could even argue that if too many theme features are cached the JVM may start doing more and more garbage collections which can adversely affect performance.
It is especially important to turn themes' caching off (by setting caching mode to NONE) before you start a tile pre-fetching task. During a tile prefetching, large amount of map requests will be sent to the MapViewer, and any cached geometries will soon become out dated, or rather, out of the region of interest.
Alternatively, prior to a large tile prefetching task, you can simply set the spatial data cache's max_cache_size to 0, which then automatically disables the caching of any theme's features.
We recommend the spatial_data_cache to have a max size of no more than 200MB (on a 32 bit OS), or 15 percent of total heap memory. In fact, we often found a performance boost by simply turning this cache off (setting its max size to zero)!
Note that when you have themes with caching mode ALL, mapViewer may not be able to fit all of its data in the cache, in which case it automatically switches the theme's cache mode to NORMAL (which as we mentioned earlier is not very ideal). So you need to carefully monitor any exception or error messages in the MapViewer log files to see if it failed to load all the data for themes with caching mode ALL. In the event this happens, you should simply set the theme's cache mode to NONE.
Note further that this internal memory cache will likely undergo major changes in MapViewer 11g R2 to truly boost performance.
Monitoring theme processing time
When you notice a slow map response time, the first thing you do is open the URL /mapviewer/admin.html, and check for slow themes.
This URL is only available in MapViewer version 11g R1 and later. It will require admin log in. Once logged in, you will see the Top Theme Queries text field, which you can submit to check which theme(s) are taking the most time to process. This can be very useful in quickly catching themes that are taking too long to return data from the database (indicating either a database performance issue, or a badly defined theme), or are returning too many rows/features, or both.
You can use the Reset Top Theme Queries form to wipe out existing top theme statistics, and MapViewer will automatically start collecting new stats for future map requests. This is useful if you have made changes to some themes and want to see if they are still the top (bad) themes.
You can also use the same URL to check the status of your MapViewer, such as how many active DB connections/sessions are being used currently.
Deployment and Configuration
MapViewer is a J2EE application that runs in all modern J2EE containers. Deploying MapViewer is no different from deploying other J2EE applications.
Ideally MapViewer should be deployed to a dedicated JVM. In Oracle iAS world, this will be a dedicated OC4J instance, and in the WebLogic Server (WLS) world, this will be a dedicated server in a domain.
The reason is simple, MapViewer, while having a small foot print itself, often requires large amount of memory and CPU resources at run time depending on the work load. So it is best to deploy MapViewer to a JVM instance that runs only MapViewer (and minimum amount of other J2EE supporting services), so that unnecessary contentions or even interruptions with other applications can be minimized. It will also be easier to monitor MapViewer's performance when it is the only application running in the JVM.
Note however there is nothing inherently wrong to deploy and run MapViewer with other J2EE applications in a shared JVM.
JVM heap size
One of the key factors that will affect mapViewer performance is the amount of Heap memory allocated to the JVM. A minimum of 512 MB is recommended. On 32 bit operating systems, the maxium heap memory should not exceed 1.5GB (or 1500 MB). A typical MapViewer install will run happily with 1GB of heap memory.
You set heap memory via JVM's -Xmx option.
Another important memory allocation parameter is the young generation size within the JVM. Young generation is a pool of memory reserved for short-lived objects. A typical map request will result in MapViewer creating many such objects (for instance, to represent geographic features in memory), so a sufficiently large young generation within the heap is crucial in preventing the JVM spending too much time on garbage collection.
The young generation size can be set via the JVM parameter:
-XX:newSize=80M
and
-XX:maxNewSize=200M
The first parameter specifies a starting young generation size of 80MB, while the second parameter indicates a maximum young generation size of 200M.
Typically the young generation size should not be more than 10 to 20 percent of the total heap size.
Spatial data cache
MapViewer lets you optionally configure an in-memory geometry data cache via the "spatial_data_cache" tag in mapViewerConfig.xml. This cache is used to buffer geometry features generated for themes whose caching mode is either set to ALL or NORMAL. (Note that you set a theme's caching mode in Map Builder's theme editor, in the Advanced tab).
When a theme's caching mode is set to ALL, MapViewer will load all of its features upon first request, and store the data in the cache. It also creates an in-memory R-Tree index on the features. All subsequent requests for this theme will be satisfied out of this cache, with no trips to the database.
When a theme's caching mode is set to NORMAL, MapViewer will store the theme's features as they are loaded. Note that MapViewer still queries the database every time such a theme is requested, but, in the event a feature is already in the cache, the un-marshaling or parsing of the database formatted geometry object is skipped, and the corresponding Java geometry object is reused out of the cache.
Our recommendation regarding theme caching, is to use caching mode ALL for small amount of themes that are based on small to medium sized tables (with couple hundred to tens of thousands of features) with static contents. And turn off NORMAL caching for all other themes. In other words, a pre-defined geometry theme should have its cache mode set to either NONE or (in rare cases) ALL.
The reason we do not recommend using the NORMAL caching mode is that it does not save much resource or time, in fact, it makes memory usage go up without much benefit at all. We could even argue that if too many theme features are cached the JVM may start doing more and more garbage collections which can adversely affect performance.
It is especially important to turn themes' caching off (by setting caching mode to NONE) before you start a tile pre-fetching task. During a tile prefetching, large amount of map requests will be sent to the MapViewer, and any cached geometries will soon become out dated, or rather, out of the region of interest.
Alternatively, prior to a large tile prefetching task, you can simply set the spatial data cache's max_cache_size to 0, which then automatically disables the caching of any theme's features.
We recommend the spatial_data_cache to have a max size of no more than 200MB (on a 32 bit OS), or 15 percent of total heap memory. In fact, we often found a performance boost by simply turning this cache off (setting its max size to zero)!
Note that when you have themes with caching mode ALL, mapViewer may not be able to fit all of its data in the cache, in which case it automatically switches the theme's cache mode to NORMAL (which as we mentioned earlier is not very ideal). So you need to carefully monitor any exception or error messages in the MapViewer log files to see if it failed to load all the data for themes with caching mode ALL. In the event this happens, you should simply set the theme's cache mode to NONE.
Note further that this internal memory cache will likely undergo major changes in MapViewer 11g R2 to truly boost performance.
Monitoring theme processing time
When you notice a slow map response time, the first thing you do is open the URL /mapviewer/admin.html, and check for slow themes.
This URL is only available in MapViewer version 11g R1 and later. It will require admin log in. Once logged in, you will see the Top Theme Queries text field, which you can submit to check which theme(s) are taking the most time to process. This can be very useful in quickly catching themes that are taking too long to return data from the database (indicating either a database performance issue, or a badly defined theme), or are returning too many rows/features, or both.
You can use the Reset Top Theme Queries form to wipe out existing top theme statistics, and MapViewer will automatically start collecting new stats for future map requests. This is useful if you have made changes to some themes and want to see if they are still the top (bad) themes.
You can also use the same URL to check the status of your MapViewer, such as how many active DB connections/sessions are being used currently.
Wednesday, May 19, 2010
new public APIs for GeoCoding & Routing
Some of you probably already know or are using various XML APIs to interact with our hosted LBS service at elocation.oracle.com. These XML APIs can be used to geocode your street addresses and get driving directions between two addresses, and also to get maps.
Recently we have published a new set of simple JavaScript APIs for geocoding and routing. These APIs, or rather JavaScript libraries, are hosted on the same public elocation.oracle.com server, and are very easy to use from your HTML/AJAX applications.
You can find more details about the new GeoCoding and Routing APIs in a presentation given during the Oracle Spatial User Conference held in Phoneix, Arizona last month. Please download the presentation here:
http://download.oracle.com/otndocs/products/spatial/pdf/osuc2010_presentations/osuc2010_geocoder_sravada.pdf
The presentation itself actually covers a lot of ground on Spatial's geocoding and routing capabilities. The new JS APIs and sample codes start from page 78.
For more presentations on both Spatial and MapViewer from this user conference (which was a smashing success), visit here:
http://www.oracle.com/technology/products/spatial/htdocs/spatial_conf_1004_idx.html
Recently we have published a new set of simple JavaScript APIs for geocoding and routing. These APIs, or rather JavaScript libraries, are hosted on the same public elocation.oracle.com server, and are very easy to use from your HTML/AJAX applications.
You can find more details about the new GeoCoding and Routing APIs in a presentation given during the Oracle Spatial User Conference held in Phoneix, Arizona last month. Please download the presentation here:
http://download.oracle.com/otndocs/products/spatial/pdf/osuc2010_presentations/osuc2010_geocoder_sravada.pdf
The presentation itself actually covers a lot of ground on Spatial's geocoding and routing capabilities. The new JS APIs and sample codes start from page 78.
For more presentations on both Spatial and MapViewer from this user conference (which was a smashing success), visit here:
http://www.oracle.com/technology/products/spatial/htdocs/spatial_conf_1004_idx.html
Thursday, March 25, 2010
Out of memory when pre-fetching tiles on Linux?
If you suspect that your MapViewer instance is having memory related issues when pre-fetching large number of map tiles for a tile layer, and your OS is Linux, then maybe this tip will help you.
Basically on Linux, the underlying file system itself will often aggressively use memory to cache file handles et al. When you start a large tile pre-fetching task, it is not uncommon for MapViewer to create hundreds of thousands of files (map tile images) in a short period. If the OS keeps caching these file system related "things" in memory, sooner or later your system will experience memory shortages, and this may cause MapViewer instance's JVM to crash or throw OutOfMemory exceptions.
Fortunately there is an easy workaround (thanks to Ji who found and tested it). All you (or your system admin) need to do is execute the following shell command as root:
#> sync; echo 3 > /proc/sys/vm/drop_caches
This command will force the Linux kernel to release file system related caches. It is best to run this script periodically (by using a cron job) like every 3 hours.
Basically on Linux, the underlying file system itself will often aggressively use memory to cache file handles et al. When you start a large tile pre-fetching task, it is not uncommon for MapViewer to create hundreds of thousands of files (map tile images) in a short period. If the OS keeps caching these file system related "things" in memory, sooner or later your system will experience memory shortages, and this may cause MapViewer instance's JVM to crash or throw OutOfMemory exceptions.
Fortunately there is an easy workaround (thanks to Ji who found and tested it). All you (or your system admin) need to do is execute the following shell command as root:
#> sync; echo 3 > /proc/sys/vm/drop_caches
This command will force the Linux kernel to release file system related caches. It is best to run this script periodically (by using a cron job) like every 3 hours.
Wednesday, February 3, 2010
MapViewer performance tuning tips - part 3
Middletier (MapViewer) side of things
Lets now look at various factors within MapViewer itself that may affect overall performance. There are also settings in the J2EE container level that can affect both performance and scalability, which we will also touch upon later.
Types of map requests
In order to properly tune MapViewer performance and size your hardware for deployment, you need to first understand where and how MapViewer spends its resources. Because MapViewer resources are primarily spent on processing map requests, it is critical to understand the types of requests that MapViewer may receive.
XML map request
The most basic type of request is simply a single XML document containing a complete map request posted to a MapViewer server’s URL directly. In this request doc, you can specify a base map, one or more additional themes (pre-defined or dynamic), a map query window, and a response format, among many other things. When MapViewer receives such a request, it breaks down the basemap (if specified) into a list of composing themes, and then adds those additional themes to form a final theme list. It then queries and fetches data for all the themes on the list, rendering their data into one or more images. The result images are either directly streamed back to the client, or saved to the disk (in which case it is the URLs to the saved image files that are sent back to the client).
WMS requests
MapViewer can also act as a WMS (Web Mapping Service) server. When it receives an incoming WMS request, the request is first converted into an XML map request, followed by the same workflow as if the client had just sent the XML request directly.
Oracle Maps requests
With the Oracle Maps JavaScript API, a typical “map” displayed by your application is often composed of one or more map tile layers, plus one or more Feature Of Interest (FOI) layers. Whenever such a map is displayed or refreshed, the Mapviewer server will receive one FOI request for each visible FOI layer, plus multiple map tile image requests.
For each FOI request, you can think of it as a simplified XML map request containing just a single theme. Once the features of this theme are loaded, MapViewer can either render them into individual FOI images (one small image for each feature), or combine all the features into a single image equal to the size of the application map window (when the whole-image option is enabled on the FOI layer).
For each map tile request, if the tile (an image file stored on the server) does not already exist, MapViewer will need to generate it on the fly, by issuing itself an XML map request that contains just the name of the base map of the tile layer. The query window of this XML map request will be the data coverage area for this particular tile. The normal workflow then follows, and the result image is always stored to a specific folder on the server. It then serves this tile image file to the client.
Note that once a map tile is generated and saved, all subsequent requests to that tile can be fulfilled extremely quickly, because MapViewer no longer needs to process/render it. Serving 100s of tile images per second is a typical performance. Whether your browser can load and display that many images quickly enough however is another matter and depends on such factors as your network connection speed.
Because on the fly rendering of map tiles is expensive, you should consider tile pre-fetching by issuing an admin request to MapViewer when it is not under load (or on a non-production server). When MapViewer receives a pre-fetching request, it simply issues many concurrent map tile requests to itself, which are basically XML map requests as mentioned earlier.
JavaBean API requests
If your application uses the JavaBean API of MapViewer, what happens is that as soon as you call oracle.lbs.mapclient.MapViewer’s run, pan or zoom method, it will construct a XML request document based on the information you have provided (name of basemap, themes you added, et al), and send it to the MapViewer server for processing.
In the end it's all XML
So as you can see, no matter which API your application uses, all that the MapViewer rendering engine sees are XML map request docs. From now on we will shorten the verbose “XML map request doc” to just “map request”. Note that client side map actions (clicking, zooming, panning et al) may or may not result in a map request to the server. And sometimes one action on the client side may even lead to multiple map requests (such as when zooming an Oracle Maps map with multiple visible FOI layers).
While the structure of MapViewer’s XML map request document can be very complex (the DTD for which can be found in the MapViewer User Guide), for simplicity we can think of it simply as a flat list of themes that MapViewer needs to process. Sometimes this list contains just one theme, and yet other times it may contain dozens or even hundreds of themes.
Given the above understanding, let’s now look at how MapViewer processes the themes for a single map request.
Rendering themes of a map request
When MapViewer processes a theme, it begins by issuing a query to the database to load the data for that theme (for Spatial based themes anyway). This query typically contains a spatial filter so that it only fetches data within the current viewing window. In the first two posts we have already covered how to speed up this phase by reducing the database time.
Note that the rendering phase does not actually start until all the themes’ loading phases have completed. In other words, if your map request contains 10 themes, the rendering phase will not start until all 10 themes have finished loading their data from the database. In other words, all the data for all the themes will be held in memory until the map request itself is completed. This is why it is important to consciously limit the number of features MapViewer might load for any given theme. When a map request is completed (the final image rendered and saved or streamed back to the client), then all the loaded theme data and temporary shapes et al. are released from memory.
During the actual rendering process, MapViewer iterates through all the loaded features of each theme, converting each geometry object from its raw database format into a Java representation when necessary, followed by the creation of a Java2D shape or point. Any necessary viewport transformation is also carried out at this stage.
If a theme requires text annotation, then the labeling process occurs after all the themes have completed the shape-rendering phase. Note that rendering follows the order of themes as they are listed in the map request, but for labeling the order is actually reversed. In other words, the last theme in the list is always labeled first.
To summarize, the total time (excluding the database time) MapViewer spends on processing a single map request is affected by the following factors, roughly in decreasing order of significance:
Note that the first item (number of total features loaded) also has the biggest impact on MapViewer’s memory usage.
While understanding how MapViewer works on a single map request certainly helps, in a real world scenario your MapViewer instance is often flooded with many concurrent map requests initiated by a variety of applications and clients using different APIs. In order to improve the scalability of MapViewer and properly size its hardware platform, one needs to understand how MapViewer handles concurrent map requests. This is discussed next.
Database sessions, number of mappers and concurrent map requests
Prior to MapViewer version 10.1.3.3, themes in a single map request load their data in parallel. In other words, if a map request is composed of 10 themes, then 10 database connections are obtained right off the bat, each executing the query of one theme. This “liberal” usage of database sessions often leads to scalability issues while providing marginal if any performance gain.
Starting with MapViewer version 10.1.3.3, SQL execution of themes’ queries are carried out in a serial fashion within each map request. A theme cannot start loading its data until the theme before it in the list has finished its query execution and data loading. The end result is that only one database connection is ever needed when processing a map request, no matter how many themes may be involved.
For a given data source, the maximum number of concurrent map requests MapViewer will accept is determined by the number of mappers specified for that data source. In other words, if a data source’s number_of_mapper attribute is set to 10 (in the MapViewer config file), then a maximum of 10 concurrent map requests can be processed (for that data source) at any given time. Because each map request will use one database connection, there can be a maximum of 10 active database sessions for the data source schema. When the 11th map request arrives it must wait in the queue until a free mapper becomes available.
You may recall that in MapViewer’s data source definition, there is also a max_connection parameter. Starting with version 10.1.3.3, this parameter is no longer needed actually. This is because the maximum number of connections out there for a given data source is always bound by the number_of_mapper parameter.
Understanding the above, you may come to the conclusion that in order to increase the scalability you simply increase the number of mappers specified for a data source. The problem is that as you increase this number, you are also increasing the number of potential concurrent database sessions, the memory usage inside MapViewer and the amount of raw data it needs to push through its rendering pipeline. Finally, if you have multiple data sources defined, then the total concurrent map requests that can potentially arrive at the MapViewer is the sum of all the data sources' maximum concurrent requests.
In the end, it is difficult to provide a clear-cut formula for determining what level of hardware you will need for your MapViewer instances. For instance, to determine roughly how much heap memory you need to allocate to a MapViewer instance, you need to look through each defined data source, finding out the average number of features each theme will load into memory and their geometric complexity, the number of themes in a typical map request, and the maximum concurrent map requests you wish to sustain. You also need to factor in things such as the in-memory geometry cache which are permanent. The best approach is to simply perform load tests against your application and monitor various logs and stats from your database and MapViewer instances to see if they are being saturated.
In the next post we will dig in further and provide a number of practical tips on tuning MapViewer performance.
Lets now look at various factors within MapViewer itself that may affect overall performance. There are also settings in the J2EE container level that can affect both performance and scalability, which we will also touch upon later.
Types of map requests
In order to properly tune MapViewer performance and size your hardware for deployment, you need to first understand where and how MapViewer spends its resources. Because MapViewer resources are primarily spent on processing map requests, it is critical to understand the types of requests that MapViewer may receive.
XML map request
The most basic type of request is simply a single XML document containing a complete map request posted to a MapViewer server’s URL directly. In this request doc, you can specify a base map, one or more additional themes (pre-defined or dynamic), a map query window, and a response format, among many other things. When MapViewer receives such a request, it breaks down the basemap (if specified) into a list of composing themes, and then adds those additional themes to form a final theme list. It then queries and fetches data for all the themes on the list, rendering their data into one or more images. The result images are either directly streamed back to the client, or saved to the disk (in which case it is the URLs to the saved image files that are sent back to the client).
WMS requests
MapViewer can also act as a WMS (Web Mapping Service) server. When it receives an incoming WMS request, the request is first converted into an XML map request, followed by the same workflow as if the client had just sent the XML request directly.
Oracle Maps requests
With the Oracle Maps JavaScript API, a typical “map” displayed by your application is often composed of one or more map tile layers, plus one or more Feature Of Interest (FOI) layers. Whenever such a map is displayed or refreshed, the Mapviewer server will receive one FOI request for each visible FOI layer, plus multiple map tile image requests.
For each FOI request, you can think of it as a simplified XML map request containing just a single theme. Once the features of this theme are loaded, MapViewer can either render them into individual FOI images (one small image for each feature), or combine all the features into a single image equal to the size of the application map window (when the whole-image option is enabled on the FOI layer).
For each map tile request, if the tile (an image file stored on the server) does not already exist, MapViewer will need to generate it on the fly, by issuing itself an XML map request that contains just the name of the base map of the tile layer. The query window of this XML map request will be the data coverage area for this particular tile. The normal workflow then follows, and the result image is always stored to a specific folder on the server. It then serves this tile image file to the client.
Note that once a map tile is generated and saved, all subsequent requests to that tile can be fulfilled extremely quickly, because MapViewer no longer needs to process/render it. Serving 100s of tile images per second is a typical performance. Whether your browser can load and display that many images quickly enough however is another matter and depends on such factors as your network connection speed.
Because on the fly rendering of map tiles is expensive, you should consider tile pre-fetching by issuing an admin request to MapViewer when it is not under load (or on a non-production server). When MapViewer receives a pre-fetching request, it simply issues many concurrent map tile requests to itself, which are basically XML map requests as mentioned earlier.
JavaBean API requests
If your application uses the JavaBean API of MapViewer, what happens is that as soon as you call oracle.lbs.mapclient.MapViewer’s run, pan or zoom method, it will construct a XML request document based on the information you have provided (name of basemap, themes you added, et al), and send it to the MapViewer server for processing.
In the end it's all XML
So as you can see, no matter which API your application uses, all that the MapViewer rendering engine sees are XML map request docs. From now on we will shorten the verbose “XML map request doc” to just “map request”. Note that client side map actions (clicking, zooming, panning et al) may or may not result in a map request to the server. And sometimes one action on the client side may even lead to multiple map requests (such as when zooming an Oracle Maps map with multiple visible FOI layers).
While the structure of MapViewer’s XML map request document can be very complex (the DTD for which can be found in the MapViewer User Guide), for simplicity we can think of it simply as a flat list of themes that MapViewer needs to process. Sometimes this list contains just one theme, and yet other times it may contain dozens or even hundreds of themes.
Given the above understanding, let’s now look at how MapViewer processes the themes for a single map request.
Rendering themes of a map request
When MapViewer processes a theme, it begins by issuing a query to the database to load the data for that theme (for Spatial based themes anyway). This query typically contains a spatial filter so that it only fetches data within the current viewing window. In the first two posts we have already covered how to speed up this phase by reducing the database time.
Note that the rendering phase does not actually start until all the themes’ loading phases have completed. In other words, if your map request contains 10 themes, the rendering phase will not start until all 10 themes have finished loading their data from the database. In other words, all the data for all the themes will be held in memory until the map request itself is completed. This is why it is important to consciously limit the number of features MapViewer might load for any given theme. When a map request is completed (the final image rendered and saved or streamed back to the client), then all the loaded theme data and temporary shapes et al. are released from memory.
During the actual rendering process, MapViewer iterates through all the loaded features of each theme, converting each geometry object from its raw database format into a Java representation when necessary, followed by the creation of a Java2D shape or point. Any necessary viewport transformation is also carried out at this stage.
If a theme requires text annotation, then the labeling process occurs after all the themes have completed the shape-rendering phase. Note that rendering follows the order of themes as they are listed in the map request, but for labeling the order is actually reversed. In other words, the last theme in the list is always labeled first.
To summarize, the total time (excluding the database time) MapViewer spends on processing a single map request is affected by the following factors, roughly in decreasing order of significance:
- The number of features loaded by each theme, and total number of features loaded.
- The cost of rendering and labeling a single feature; this is in turn dictated by how complex each rendering and labeling style is, how detailed (in terms of number of vertices) are the geometries on average, et al.
- The raw speed of the server’s disk, CPU and memory access
- Whether anti-aliasing is requested
- The amount of heap memory allocated to MapViewer
- The version of JDK being used
Note that the first item (number of total features loaded) also has the biggest impact on MapViewer’s memory usage.
While understanding how MapViewer works on a single map request certainly helps, in a real world scenario your MapViewer instance is often flooded with many concurrent map requests initiated by a variety of applications and clients using different APIs. In order to improve the scalability of MapViewer and properly size its hardware platform, one needs to understand how MapViewer handles concurrent map requests. This is discussed next.
Database sessions, number of mappers and concurrent map requests
Prior to MapViewer version 10.1.3.3, themes in a single map request load their data in parallel. In other words, if a map request is composed of 10 themes, then 10 database connections are obtained right off the bat, each executing the query of one theme. This “liberal” usage of database sessions often leads to scalability issues while providing marginal if any performance gain.
Starting with MapViewer version 10.1.3.3, SQL execution of themes’ queries are carried out in a serial fashion within each map request. A theme cannot start loading its data until the theme before it in the list has finished its query execution and data loading. The end result is that only one database connection is ever needed when processing a map request, no matter how many themes may be involved.
For a given data source, the maximum number of concurrent map requests MapViewer will accept is determined by the number of mappers specified for that data source. In other words, if a data source’s number_of_mapper attribute is set to 10 (in the MapViewer config file), then a maximum of 10 concurrent map requests can be processed (for that data source) at any given time. Because each map request will use one database connection, there can be a maximum of 10 active database sessions for the data source schema. When the 11th map request arrives it must wait in the queue until a free mapper becomes available.
You may recall that in MapViewer’s data source definition, there is also a max_connection parameter. Starting with version 10.1.3.3, this parameter is no longer needed actually. This is because the maximum number of connections out there for a given data source is always bound by the number_of_mapper parameter.
Understanding the above, you may come to the conclusion that in order to increase the scalability you simply increase the number of mappers specified for a data source. The problem is that as you increase this number, you are also increasing the number of potential concurrent database sessions, the memory usage inside MapViewer and the amount of raw data it needs to push through its rendering pipeline. Finally, if you have multiple data sources defined, then the total concurrent map requests that can potentially arrive at the MapViewer is the sum of all the data sources' maximum concurrent requests.
In the end, it is difficult to provide a clear-cut formula for determining what level of hardware you will need for your MapViewer instances. For instance, to determine roughly how much heap memory you need to allocate to a MapViewer instance, you need to look through each defined data source, finding out the average number of features each theme will load into memory and their geometric complexity, the number of themes in a typical map request, and the maximum concurrent map requests you wish to sustain. You also need to factor in things such as the in-memory geometry cache which are permanent. The best approach is to simply perform load tests against your application and monitor various logs and stats from your database and MapViewer instances to see if they are being saturated.
In the next post we will dig in further and provide a number of practical tips on tuning MapViewer performance.
Thursday, January 14, 2010
MapViewer performance tuning tips - part 2
This is the second post of the performance tuning tips series.
When you identify a theme with an unusually high "database time", it's time to look for database fixes. While typical database query optimization tips and tricks still apply, spatial queries have some unique characteristics, especially when the base table of your 'troubling' theme contains a large number of rows.
From my own observations over the years, in many cases the "database time" of a MapViewer theme is actually database idle time or wait time. To be more specific, it is time the database spent waiting on disk I/Os. This is probably nothing shocking given that many MapViewer theme queries can fetch a lot of data in terms of pure volume (as measured in total bytes), especially when polygon or polyline type geometries are involved. Compared with a theme query, your typical shopping cart query that fetches a product listing is almost negligible as far as result data volume is concerned. With a huge base table, chances are the database buffer cannot easily cache all the data, so it often needs to read physical disk blocks in order to fetch and return data for a query.
When diagnosing database bottlenecks such as disk I/O waits , there are two powerful tools that come with Oracle database 10g and later versions. They are AWR (Advanced Workload Repository) and ADDM (Automated Database Diagnostic Monitor). I found them invaluable in finding out why your MapViewer application is running slow, as they can not only identify I/O bottlenecks, but also many other kind of bottlenecks in the database. Note however they do not replace other tuning tools such SQL Tuning Adviser. In fact ADDM will often suggest you run SQL Tuning Adviser on specific SQL queries it deemed problematic.
Let's jump right into action already. The following are the basic steps of using these two tools.
Step 1 Login as DBA, then create a "before" snapshot of the database:
SQL> exec dbms_workload_repository.create_snapshot;
Step 2 Run your MapViewer application load tests. When done,
Step 3 Create an "after" snapshot of the database, using the same command:
SQL> exec dbms_workload_repository.create_snapshot;
With these two snapshots, you can now execute the AWR and ADDM scripts to generate very informative and detailed reports on what happened inside the database during these two snapshots. Keep in mind that the stats gathered by AWR and ADDM cover all database activities for all sessions, not just sessions created by MapViewer to perform theme queries.
Step 4 Generate an AWR report.
To create an AWR report, execute the following as DBA:
SQL> @$ORACLE_HOME/rdbms/admin/awrrpti.sql
When prompted, use HTML as the report format, and pick the two snapshots that you just created.
After exiting SQL*Plus, you will see an HTML file containing the AWR report. Open it in your browser, and you will be greeted with a rich set of database activity reports.
One of the interesting tables is titled "Top 5 Timed Foreground Events". For instance, this is the result from a recent AWR report generated for my spatial database:
As you can see, it shows that the database spent a whopping 96.55% of time waiting for User I/Os (db file sequential reads) ! You can also check out several other tables such as "SQL ordered by Elapsed Time" and "SQL ordered by User I/O Wait Time" et al. It is also quite easy to find out which SQL is causing the most User I/Os, so on and so forth. Of course, your report may not show user I/O as an issue, but you get the idea.
While AWR is very useful in providing you with a detailed break down of the database events and timings, it is best complemented by the adviser tool, ADDM, which reads off the same set of stats between the two snapshots, but can provide you with concrete advices on how to fix the bottlenecks in your database.
Step 5 Generate an ADDM report.
While you can always use EM to view the ADDM report, here we just want to generate a simple text report using SQL*Plus by running the following command as DBA:
SQL> @$ORACLE_HOME/rdbms/admin/addmrpt.sql;
Again this script will prompt you for the before and after snapshots' IDs. Then it generates a .lst file which you can open with any text editor. In this text report you can find some very insightful suggestions. For instance, in one of my reports it detected that the disk containing the database data file of a large theme's base table has a below-average I/O speed, and suggested file striping as one way to reduce I/O waits. This is exactly the kind of advices that can immediately provide a performance boost.
While I have been stressing the negative impact of heavy user I/Os, please note that your situation may very well be a different one. For instance you may not have an I/O bottleneck at all, so please be open minded when reading these two reports and be prepared to wrestle with other kind of issues within your database.
Let's say you indeed find yourself in a similar situation where you see a lot of User I/O wait events, especially on your large tables (those with hundreds of thousands or millions of rows). Then how do you tackle this issue? One of the most obvious remedies is to partition both your data tables and associated spatial indexes, if they are not already partitioned.
The other tip, which will be described in detail now, is to reorganize the table rows based on the proximity of the geometry column. The basic idea is to store rows in neighboring database blocks if their geometries are close to each other spatially. When stored this way, rows that are to be returned for a theme query will more likely be found in neighboring data blocks on the disk, thus significantly reducing disk reads. This re-org will help most spatial queries with predicates such as SDO_FILTER or SDO_RELATE. MapViewer for instance always uses SDO_FILTER for normal map panning/zooming and WMS type map requests, thus can benefit from such a table wide re-org, sometimes tremendously.
Lets first see what happens if you do not reorganize your table. Lets say each disk block has capacity for 100 rows. In a ‘uniform but random storage’ case, for any given query window, each "hit" block may only have 10 rows that are within this window. Now lets say a particular map request’s query window interacts with 1000 rows according to the spatial index. Assuming the table is so big and the incoming query windows change so often that the database data buffer is basically useless. What the database will end up doing then is reading 100 blocks from the disk in order to fetch these 1000 rows based on their ROWIDs produced by the spatial index. Reading so many blocks for a single query is very expensive. Now imagine 20 concurrent incoming map requests, each having a different viewport, and you can easily imagine why the database spends majority of its time waiting for disk reads. As a result, the overall throughput of MapViewer plummets.
So how do you reorganize row storage? The following script is actually documented in the “Pro Oracle Spatial for Oracle Database 11g” book, Chapter 16. It uses a linear tessellation key to re-order the geometries stored in a table. For those of you don’t have access to this book, here are the (slightly modified to fix several typos in the book) scripts you can copy and use.
First, create a package in the MDSYS schema (it won't work in any other database schema!) :
CREATE OR REPLACE PACKAGE clusterit
AS
FUNCTION linear_key(location sdo_geometry, diminfo sdo_dim_array)
RETURN RAW deterministic;
END;
/
CREATE OR REPLACE PACKAGE BODY clusterit
AS
FUNCTION linear_key
(
location sdo_geometry,
diminfo sdo_dim_array
)
return RAW deterministic
AS
ctr sdo_geometry;
rval raw(48);
lvl integer;
BEGIN
ctr := sdo_geom.sdo_pointonsurface(location, diminfo);
lvl := 8;
rval:=
MD.HHENCODE(
ctr.sdo_point.x, diminfo(1).sdo_lb, diminfo(1).sdo_ub, lvl,
ctr.sdo_point.y, diminfo(2).sdo_lb, diminfo(2).sdo_ub, lvl);
return rval;
END;
END;
/
Next, log into the proper schema, and make a backup copy of the problem base table ‘my_table’ (drop its spatial index first):
SQL> rename my_table to my_table_dup;
Add a linear key column:
SQL> alter table my_table_dup add (linear_key raw);
Then pre-compute and store the linear keys:
SQL> update my_table_dup a set linear_key =
mdsys.clusterit.linear_key(a.geometry,
(select diminfo from user_sdo_geom_metadata where table_name='MY_TABLE’ and column_name='GEOMETRY'));
Finally, re-create the original table with proper storage order:
SQL> create table my_table as select * from my_table_dup where rownum <1;
SQL> insert into my_table select * from my_table_dup order by linear_key;
SQL> commit;
SQL> drop table my_table_dup;
Don’t forget to recreate the spatial index on my_table.
Now re-run your MapViewer tests and see if this reorganization has helped its overall performance. In one internal and not so rigorous experiment, after reorganizing a large table using the above approach, we observed as much as 300% increase in overall MapViewer throughput.
In summary, AWR and ADDM are two of the more useful tools at your disposal when diagnosing high database times, and pay special attention to excessive database reads. This blog also assumes that your typical MapViewer theme is a pre-defined geometry theme with relatively simple query conditions. If your theme uses super complex dynamic SQL queries, then chances are you have more than disk reads to worry about. In any case, database tuning in general, and Spatial SQL tuning in particular, is a very deep topic in itself, and I'm really in no position to offer any authoritative advice; so please don't blame me if my little tip above did not actually reduce your database time :)
- Finding bottlenecks in your Oracle database
When you identify a theme with an unusually high "database time", it's time to look for database fixes. While typical database query optimization tips and tricks still apply, spatial queries have some unique characteristics, especially when the base table of your 'troubling' theme contains a large number of rows.
From my own observations over the years, in many cases the "database time" of a MapViewer theme is actually database idle time or wait time. To be more specific, it is time the database spent waiting on disk I/Os. This is probably nothing shocking given that many MapViewer theme queries can fetch a lot of data in terms of pure volume (as measured in total bytes), especially when polygon or polyline type geometries are involved. Compared with a theme query, your typical shopping cart query that fetches a product listing is almost negligible as far as result data volume is concerned. With a huge base table, chances are the database buffer cannot easily cache all the data, so it often needs to read physical disk blocks in order to fetch and return data for a query.
When diagnosing database bottlenecks such as disk I/O waits , there are two powerful tools that come with Oracle database 10g and later versions. They are AWR (Advanced Workload Repository) and ADDM (Automated Database Diagnostic Monitor). I found them invaluable in finding out why your MapViewer application is running slow, as they can not only identify I/O bottlenecks, but also many other kind of bottlenecks in the database. Note however they do not replace other tuning tools such SQL Tuning Adviser. In fact ADDM will often suggest you run SQL Tuning Adviser on specific SQL queries it deemed problematic.
Let's jump right into action already. The following are the basic steps of using these two tools.
Step 1 Login as DBA, then create a "before" snapshot of the database:
SQL> exec dbms_workload_repository.create_snapshot;
Step 2 Run your MapViewer application load tests. When done,
Step 3 Create an "after" snapshot of the database, using the same command:
SQL> exec dbms_workload_repository.create_snapshot;
With these two snapshots, you can now execute the AWR and ADDM scripts to generate very informative and detailed reports on what happened inside the database during these two snapshots. Keep in mind that the stats gathered by AWR and ADDM cover all database activities for all sessions, not just sessions created by MapViewer to perform theme queries.
Step 4 Generate an AWR report.
To create an AWR report, execute the following as DBA:
SQL> @$ORACLE_HOME/rdbms/admin/awrrpti.sql
When prompted, use HTML as the report format, and pick the two snapshots that you just created.
After exiting SQL*Plus, you will see an HTML file containing the AWR report. Open it in your browser, and you will be greeted with a rich set of database activity reports.
One of the interesting tables is titled "Top 5 Timed Foreground Events". For instance, this is the result from a recent AWR report generated for my spatial database:
As you can see, it shows that the database spent a whopping 96.55% of time waiting for User I/Os (db file sequential reads) ! You can also check out several other tables such as "SQL ordered by Elapsed Time" and "SQL ordered by User I/O Wait Time" et al. It is also quite easy to find out which SQL is causing the most User I/Os, so on and so forth. Of course, your report may not show user I/O as an issue, but you get the idea.
While AWR is very useful in providing you with a detailed break down of the database events and timings, it is best complemented by the adviser tool, ADDM, which reads off the same set of stats between the two snapshots, but can provide you with concrete advices on how to fix the bottlenecks in your database.
Step 5 Generate an ADDM report.
While you can always use EM to view the ADDM report, here we just want to generate a simple text report using SQL*Plus by running the following command as DBA:
SQL> @$ORACLE_HOME/rdbms/admin/addmrpt.sql;
Again this script will prompt you for the before and after snapshots' IDs. Then it generates a .lst file which you can open with any text editor. In this text report you can find some very insightful suggestions. For instance, in one of my reports it detected that the disk containing the database data file of a large theme's base table has a below-average I/O speed, and suggested file striping as one way to reduce I/O waits. This is exactly the kind of advices that can immediately provide a performance boost.
While I have been stressing the negative impact of heavy user I/Os, please note that your situation may very well be a different one. For instance you may not have an I/O bottleneck at all, so please be open minded when reading these two reports and be prepared to wrestle with other kind of issues within your database.
- Reorganizing your large tables
Let's say you indeed find yourself in a similar situation where you see a lot of User I/O wait events, especially on your large tables (those with hundreds of thousands or millions of rows). Then how do you tackle this issue? One of the most obvious remedies is to partition both your data tables and associated spatial indexes, if they are not already partitioned.
The other tip, which will be described in detail now, is to reorganize the table rows based on the proximity of the geometry column. The basic idea is to store rows in neighboring database blocks if their geometries are close to each other spatially. When stored this way, rows that are to be returned for a theme query will more likely be found in neighboring data blocks on the disk, thus significantly reducing disk reads. This re-org will help most spatial queries with predicates such as SDO_FILTER or SDO_RELATE. MapViewer for instance always uses SDO_FILTER for normal map panning/zooming and WMS type map requests, thus can benefit from such a table wide re-org, sometimes tremendously.
Lets first see what happens if you do not reorganize your table. Lets say each disk block has capacity for 100 rows. In a ‘uniform but random storage’ case, for any given query window, each "hit" block may only have 10 rows that are within this window. Now lets say a particular map request’s query window interacts with 1000 rows according to the spatial index. Assuming the table is so big and the incoming query windows change so often that the database data buffer is basically useless. What the database will end up doing then is reading 100 blocks from the disk in order to fetch these 1000 rows based on their ROWIDs produced by the spatial index. Reading so many blocks for a single query is very expensive. Now imagine 20 concurrent incoming map requests, each having a different viewport, and you can easily imagine why the database spends majority of its time waiting for disk reads. As a result, the overall throughput of MapViewer plummets.
- Script for Reorganizing a large table
So how do you reorganize row storage? The following script is actually documented in the “Pro Oracle Spatial for Oracle Database 11g” book, Chapter 16. It uses a linear tessellation key to re-order the geometries stored in a table. For those of you don’t have access to this book, here are the (slightly modified to fix several typos in the book) scripts you can copy and use.
First, create a package in the MDSYS schema (it won't work in any other database schema!) :
CREATE OR REPLACE PACKAGE clusterit
AS
FUNCTION linear_key(location sdo_geometry, diminfo sdo_dim_array)
RETURN RAW deterministic;
END;
/
CREATE OR REPLACE PACKAGE BODY clusterit
AS
FUNCTION linear_key
(
location sdo_geometry,
diminfo sdo_dim_array
)
return RAW deterministic
AS
ctr sdo_geometry;
rval raw(48);
lvl integer;
BEGIN
ctr := sdo_geom.sdo_pointonsurface(location, diminfo);
lvl := 8;
rval:=
MD.HHENCODE(
ctr.sdo_point.x, diminfo(1).sdo_lb, diminfo(1).sdo_ub, lvl,
ctr.sdo_point.y, diminfo(2).sdo_lb, diminfo(2).sdo_ub, lvl);
return rval;
END;
END;
/
Next, log into the proper schema, and make a backup copy of the problem base table ‘my_table’ (drop its spatial index first):
SQL> rename my_table to my_table_dup;
Add a linear key column:
SQL> alter table my_table_dup add (linear_key raw);
Then pre-compute and store the linear keys:
SQL> update my_table_dup a set linear_key =
mdsys.clusterit.linear_key(a.geometry,
(select diminfo from user_sdo_geom_metadata where table_name='MY_TABLE’ and column_name='GEOMETRY'));
Finally, re-create the original table with proper storage order:
SQL> create table my_table as select * from my_table_dup where rownum <1;
SQL> insert into my_table select * from my_table_dup order by linear_key;
SQL> commit;
SQL> drop table my_table_dup;
Don’t forget to recreate the spatial index on my_table.
Now re-run your MapViewer tests and see if this reorganization has helped its overall performance. In one internal and not so rigorous experiment, after reorganizing a large table using the above approach, we observed as much as 300% increase in overall MapViewer throughput.
In summary, AWR and ADDM are two of the more useful tools at your disposal when diagnosing high database times, and pay special attention to excessive database reads. This blog also assumes that your typical MapViewer theme is a pre-defined geometry theme with relatively simple query conditions. If your theme uses super complex dynamic SQL queries, then chances are you have more than disk reads to worry about. In any case, database tuning in general, and Spatial SQL tuning in particular, is a very deep topic in itself, and I'm really in no position to offer any authoritative advice; so please don't blame me if my little tip above did not actually reduce your database time :)
Wednesday, January 13, 2010
MapViewer performance tuning tips - Part 1
This is the first of a series of blogs on MapViewer performance tuning.
Overview
When your MapViewer application performs poorly, there are many possible causes. Maybe your application is trying to (unnecessarily) display way too many features on a map or layer. Maybe your database query (for dynamic themes) is too complicated and poorly optimized. Or maybe the database is doing too many disk I/Os when fetching the result set for a theme. It could also be due to a poorly configured MapViewer in-memory cache, an overwhelmed J2EE container (overloaded with many other applications), or just a very slow middle-tier box. It could be all of the above. The symptom however is always the same: when a user clicks on the map to pan or zoom, she will wait a long time before the map completely refreshes itself.
So how to find the true bottleneck? Where do you start when diagnosing a slow mapping application? The short answer, is to first identify the most time consuming theme(s) in your application, by looking at the log file.
To do so set MapViewer's logging to the "finest" level, restart MapViewer, and run the application just long enough (or manually submit a single XML map request) to generate some meaningful logs. When scanning the freshly created MapViewer log file, for each theme you should see several log records like the following:
FINER: [ THEME_DEMO_HIGHWAYS ] sql exec time: 585ms, total time loading 8 features: 2679ms.
FINER: time to render theme THEME_DEMO_HIGHWAYS with 8 styled features: 72ms
FINER: time to label theme THEME_DEMO_HIGHWAYS with 8 styled features: 112ms
In the first log record, pay special attention to the last number, 2679ms. This number (in milliseconds) indicates how long it took the database to execute the theme query, fetch result rows from data blocks, and transmit the result set over the network to MapViewer. For simplicity we will call this the "database time". Likewise let's call the combined rendering and labeling time indicated by the second and third log records the "MapViewer time"; after all that is how long MapViewer actually spent doing something useful instead of just waiting for and receiving data from the database.
In many cases, the database time will be the dominant one, so you will need to poke around the database and speed things up there. We will get into it later.
If the MapViewer time is high, we will need to find out why. One recommended approach is to run a CPU profiling of the MapViewer instance. Again we will provide more details in a later post.
Reducing these two types of time will have a positive impact on your application's performance regardless of which MapViewer API you are using.
Note that starting with MapViewer 11g R1 there is another way to easily spot themes with excessive "database time". To do so you must first log in as the MapViewer
admin user. Then manually open the /mapviewer/admin.html page in your browser. You will see a "Top theme queries" text area with a Submit button underneath it. Click on the submit button to see a list of top (database) time-consuming queries. It's a good idea to first click the "Reset top theme queries" button so that MapViewer clears any existing list of themes and starts gathering fresh stats for you.
Sometimes you may find that all of your themes are loaded and rendered quickly, but your application still takes a long time to display the map. Let's simply call this the "user time", since it is the time a user spent staring at a spinning clock or progress bar. When the sum of the "database time" and "MapViewer time" accounts for only a small portion of the "user time", we suggest you use a FireFox addon called "FireBug" to find out what exactly is happening on the wire and in your browser. For instance, is your Oracle Maps application overwhelming the remote web server with too many HTTP requests for map tile and FOI images? Maybe you should look into using the Whole-Image option for your FOI layer(s) to avoid transmitting many small pieces from MapViewer to the client every time a user pans or zooms the map. We will provide more tips like this in a later blog as well.
To be continued...
Overview
When your MapViewer application performs poorly, there are many possible causes. Maybe your application is trying to (unnecessarily) display way too many features on a map or layer. Maybe your database query (for dynamic themes) is too complicated and poorly optimized. Or maybe the database is doing too many disk I/Os when fetching the result set for a theme. It could also be due to a poorly configured MapViewer in-memory cache, an overwhelmed J2EE container (overloaded with many other applications), or just a very slow middle-tier box. It could be all of the above. The symptom however is always the same: when a user clicks on the map to pan or zoom, she will wait a long time before the map completely refreshes itself.
So how to find the true bottleneck? Where do you start when diagnosing a slow mapping application? The short answer, is to first identify the most time consuming theme(s) in your application, by looking at the log file.
To do so set MapViewer's logging to the "finest" level, restart MapViewer, and run the application just long enough (or manually submit a single XML map request) to generate some meaningful logs. When scanning the freshly created MapViewer log file, for each theme you should see several log records like the following:
FINER: [ THEME_DEMO_HIGHWAYS ] sql exec time: 585ms, total time loading 8 features: 2679ms.
FINER: time to render theme THEME_DEMO_HIGHWAYS with 8 styled features: 72ms
FINER: time to label theme THEME_DEMO_HIGHWAYS with 8 styled features: 112ms
In the first log record, pay special attention to the last number, 2679ms. This number (in milliseconds) indicates how long it took the database to execute the theme query, fetch result rows from data blocks, and transmit the result set over the network to MapViewer. For simplicity we will call this the "database time". Likewise let's call the combined rendering and labeling time indicated by the second and third log records the "MapViewer time"; after all that is how long MapViewer actually spent doing something useful instead of just waiting for and receiving data from the database.
In many cases, the database time will be the dominant one, so you will need to poke around the database and speed things up there. We will get into it later.
If the MapViewer time is high, we will need to find out why. One recommended approach is to run a CPU profiling of the MapViewer instance. Again we will provide more details in a later post.
Reducing these two types of time will have a positive impact on your application's performance regardless of which MapViewer API you are using.
Note that starting with MapViewer 11g R1 there is another way to easily spot themes with excessive "database time". To do so you must first log in as the MapViewer
admin user. Then manually open the /mapviewer/admin.html page in your browser. You will see a "Top theme queries" text area with a Submit button underneath it. Click on the submit button to see a list of top (database) time-consuming queries. It's a good idea to first click the "Reset top theme queries" button so that MapViewer clears any existing list of themes and starts gathering fresh stats for you.
Sometimes you may find that all of your themes are loaded and rendered quickly, but your application still takes a long time to display the map. Let's simply call this the "user time", since it is the time a user spent staring at a spinning clock or progress bar. When the sum of the "database time" and "MapViewer time" accounts for only a small portion of the "user time", we suggest you use a FireFox addon called "FireBug" to find out what exactly is happening on the wire and in your browser. For instance, is your Oracle Maps application overwhelming the remote web server with too many HTTP requests for map tile and FOI images? Maybe you should look into using the Whole-Image option for your FOI layer(s) to avoid transmitting many small pieces from MapViewer to the client every time a user pans or zooms the map. We will provide more tips like this in a later blog as well.
To be continued...
Wednesday, January 6, 2010
MapViewer 11g patch 1 (11.1.1.2) available
Thursday, November 12, 2009
Use Mapviewer to cache WMS maps as map tiles
Warning: Not all map services providers(WMS included) allow their maps to be cached by others. Before using MapViewer to cache maps served by any third party map services provider, you should consult the map services provider to make sure you're allowed to cache their maps.
A map source adapter is basically a Java class that is called by MapViewer to construct requests understood by the external map services provider. MapViewer calls the map source adapter to construct map requests, send them to the map services provider, fetch the maps and cache them as map tiles. Mapviewer is shipped with a map source adapter for WMS, which you can use to cache maps served by a WMS server. To better understand how the map source adapter works, you can take a look at the source code of the WMS adapter, which can be found at $mapviewer_deploy_home/web/WEB-INF/tileserver/mvadapter/mcsadapter/WMSAdapter.java.
Here are the steps to create a map tile layer using the WMS adapter.
1. Login MapViewer's web admin page.
2. Enter the tile layer creation interface. Choose "External" when being asked for the the type of map source.
3. On the tile layer creation page, you need to provide the configuration information for the map tile layer. Besides regular tile layer configuration options such as name, data source, coordinate system, zoom level definition and etc, you also need to provide the following information specific to an external map tile layer.
- Map service URL: The URL of the external map services provider. In the case of WMS, it should be the URL of the WMS server, e.g. http://www.myhost.com/mywms.
- Adapter class: The path of the WMS adapter class. It should be mcsadapter.WMSAdapter.
- Jar file location: the full path of the file mvadapter.jar, which is under $mapviewer_deploy_home/web/WEB-INF/tileserver/mvadapter.
- Check "Adapter properties".
- Add WMS parameters(name and value) for your map requests. Example:
name value
----------- -----
version 1.1.1
srs EPSG:4326
layers Countries,Borders
format image/png
transparent true
4. Click "Submit" to create the tile layer.
Thursday, October 22, 2009
Google maps tiles in a MapViewer app
Earlier this year I posted an article on how to display MapViewer generated tiles in a Google Maps application. In this post I will discuss how to display Google Maps tiles in a MapViewer/Oracle Maps application.
Starting with MapViewer 11g Release 1, a built-in Google Maps tile layer class has been added to the MapViewer's JavaScript API. It is called MVGoogleTileLayer. Unfortunately it is not officially documented for 11g Release 1 (but will be in the upcoming patch and future releases).
This API class is a thin wrapper of the official Google Maps API, and as such you will need to get your own Google Maps key for your application.
The basic steps for displaying Google Maps in your Oracle Maps application are:
Step 1. Import the Google Maps JavaScript library
<script src="http://maps.google.com/maps?file=api&v=2&key=[your-Google-Maps-API-key]"
type="text/javascript"></script>
Again you will need to get your own Google Maps key from Google and set it in the above script tag.
Step 2. Add a Google Maps tile layer to the MVMapView object
mapview = new MVMapView(document.getElementById("map"), baseURL);
baseMap = new MVGoogleTileLayer() ;
mapview.addMapTileLayer(baseMap);
That's all it takes to display a basic Google Maps map in your MapViewer application.
You can also change the type of map you get from Google Maps. Here is an example:
var mapType = G_HYBRID_MAP;
basemap.setMapType(mapType) ;
Where basemap is a MVGoogleTileLayer instance. Note that G_HYBRID_MAP is a constant defined by the imported Google Maps API library and represents a hybrid Google Maps type. The full description for this method is:
setMapType(mapType)
This method sets the type of the map, which can be one of the following predefined Google Map types, G_NORMAL_MAP, G_SATELLITE_MAP, G_HYBRID_MAP and G_PHYSICAL_MAP.
The only other method defined on the MVGoogleTileLayer class is getMapType(), which returns the current map type of the Google Maps being displayed.
Finally, as part of your application cleanup, please call Google Maps' clean-up method GUnload() (such as in the HTML page's unload event listener).
Starting with MapViewer 11g Release 1, a built-in Google Maps tile layer class has been added to the MapViewer's JavaScript API. It is called MVGoogleTileLayer. Unfortunately it is not officially documented for 11g Release 1 (but will be in the upcoming patch and future releases).
This API class is a thin wrapper of the official Google Maps API, and as such you will need to get your own Google Maps key for your application.
The basic steps for displaying Google Maps in your Oracle Maps application are:
Step 1. Import the Google Maps JavaScript library
<script src="http://maps.google.com/maps?file=api&v=2&key=[your-Google-Maps-API-key]"
type="text/javascript"></script>
Again you will need to get your own Google Maps key from Google and set it in the above script tag.
Step 2. Add a Google Maps tile layer to the MVMapView object
mapview = new MVMapView(document.getElementById("map"), baseURL);
baseMap = new MVGoogleTileLayer() ;
mapview.addMapTileLayer(baseMap);
That's all it takes to display a basic Google Maps map in your MapViewer application.
You can also change the type of map you get from Google Maps. Here is an example:
var mapType = G_HYBRID_MAP;
basemap.setMapType(mapType) ;
Where basemap is a MVGoogleTileLayer instance. Note that G_HYBRID_MAP is a constant defined by the imported Google Maps API library and represents a hybrid Google Maps type. The full description for this method is:
setMapType(mapType)
This method sets the type of the map, which can be one of the following predefined Google Map types, G_NORMAL_MAP, G_SATELLITE_MAP, G_HYBRID_MAP and G_PHYSICAL_MAP.
The only other method defined on the MVGoogleTileLayer class is getMapType(), which returns the current map type of the Google Maps being displayed.
Finally, as part of your application cleanup, please call Google Maps' clean-up method GUnload() (such as in the HTML page's unload event listener).
Monday, October 5, 2009
displaying table of longitude and latitude points
While in most cases your location data are stored in an Oracle database as SDO_GEOMETRY type, sometimes you may have certain point data stored simply as two numeric columns in a table (as longitude/latitude for instance). So how can you display these points using MapViewer? In this article I will present one of the more elegant options, namely using function-based index.
The overall steps are:
1. Create a database function that returns a SDO_GEOMETRY object from two numeric values.
2. Treat this function as a SDO_GEOMETRY column by creating an entry in the USER_SDO_GEOM_METADATA view.
3. Create a Spatial index on this function
4. Create a pre-defined theme in Map Builder for this function.
5. Profit.
Lets say we have an existing table CITIES in the schema SCOTT.
SQL> desc cities;
Name Null? Type
------------------------------- -------- ----------------------------
ID NUMBER
NAME VARCHAR2(100)
ALIASE VARCHAR2(500)
COUNTR VARCHAR2(100)
POP NUMBER
LON NUMBER
LAT NUMBER
Where the two columns LON and LAT represent each city's location.
So the first step is to create a function that generates an actual SDO_GEOMETRY object when supplied with a pair of (lon, lat).
Step 1. Create function
Execute the following while logged in as SCOTT:
create or replace function get_geometry(lon in number,
lat in number)
return SDO_GEOMETRY deterministic is
begin
return sdo_geometry(2001, 8307, sdo_point_type(lon, lat, NULL),NULL, NULL);
end;
/
Note that the function must be deterministic.
Step 2. Populate Spatial metadata view
Now that we have a function, we need to make Spatial and MapViewer treat it as if it's a real SDO_GEOMETRY column. All we need to do is insert an entry for this function in the USER_SDO_GEOM_METADATA view. Again execute the following as SCOTT:
insert into user_sdo_geom_metadata values('CITIES',
'scott.get_geometry(lon,lat)',
sdo_dim_array(
sdo_dim_element('Longitude', -180, 180, 0.005),
sdo_dim_element('Latitude', -90, 90, 0.005)),
8307);
commit;
Step 3. Create a Spatial index:
We now need to create a Spatial index on the function, just like with any SDO_GEOMETRY columns. Without a Spatial index, your SQL spatial queries won't work, and MapViewer cannot display them easily.
create index CITIES_SDX on
CITIES(get_geometry(lon,lat))
indextype is mdsys.spatial_index;
Step 4. Create a Map Builder theme
Open Map Builder, click Show Data to open the data navigator. You should find the CITIES table under the user SCOTT. Right click it and choose "Create Geometry Theme". You should see that the function we just created is already picked up as the Spatial column, as illustrated in this screen shot:
Go through the normal process and complete the theme creation. You now have a pre-defined MapViewer theme that can be used to display those (X,Y) data. You can also add this theme to your AJAX map as a FoI (Feature Of Interest) layer.
The overall steps are:
1. Create a database function that returns a SDO_GEOMETRY object from two numeric values.
2. Treat this function as a SDO_GEOMETRY column by creating an entry in the USER_SDO_GEOM_METADATA view.
3. Create a Spatial index on this function
4. Create a pre-defined theme in Map Builder for this function.
5. Profit.
Lets say we have an existing table CITIES in the schema SCOTT.
SQL> desc cities;
Name Null? Type
------------------------------- -------- ----------------------------
ID NUMBER
NAME VARCHAR2(100)
ALIASE VARCHAR2(500)
COUNTR VARCHAR2(100)
POP NUMBER
LON NUMBER
LAT NUMBER
Where the two columns LON and LAT represent each city's location.
So the first step is to create a function that generates an actual SDO_GEOMETRY object when supplied with a pair of (lon, lat).
Step 1. Create function
Execute the following while logged in as SCOTT:
create or replace function get_geometry(lon in number,
lat in number)
return SDO_GEOMETRY deterministic is
begin
return sdo_geometry(2001, 8307, sdo_point_type(lon, lat, NULL),NULL, NULL);
end;
/
Note that the function must be deterministic.
Step 2. Populate Spatial metadata view
Now that we have a function, we need to make Spatial and MapViewer treat it as if it's a real SDO_GEOMETRY column. All we need to do is insert an entry for this function in the USER_SDO_GEOM_METADATA view. Again execute the following as SCOTT:
insert into user_sdo_geom_metadata values('CITIES',
'scott.get_geometry(lon,lat)',
sdo_dim_array(
sdo_dim_element('Longitude', -180, 180, 0.005),
sdo_dim_element('Latitude', -90, 90, 0.005)),
8307);
commit;
Step 3. Create a Spatial index:
We now need to create a Spatial index on the function, just like with any SDO_GEOMETRY columns. Without a Spatial index, your SQL spatial queries won't work, and MapViewer cannot display them easily.
create index CITIES_SDX on
CITIES(get_geometry(lon,lat))
indextype is mdsys.spatial_index;
Step 4. Create a Map Builder theme
Open Map Builder, click Show Data to open the data navigator. You should find the CITIES table under the user SCOTT. Right click it and choose "Create Geometry Theme". You should see that the function we just created is already picked up as the Spatial column, as illustrated in this screen shot:
Go through the normal process and complete the theme creation. You now have a pre-defined MapViewer theme that can be used to display those (X,Y) data. You can also add this theme to your AJAX map as a FoI (Feature Of Interest) layer.
Friday, September 18, 2009
Why am I getting "Request string is too long for Oracle Maps' non-AJAX remoting"?
You may get this error when your application is running in a web domain that is different from the one where mapviewer is running, e.g. your application is running on hostA and your MapViewer is running on hostB. Please note that different http ports on the same host are also two different domains.
function showMap()
{
Oracle Maps client relies on AJAX to exchange data with the MapViewer server. In the scenario described above, when the client needs to send an AJAX request to the Mapviewer server, it is sending the request across domain, which is prohibited by all major web browsers. This is known as the cross-domain constraint.
We implemented our own non-AJAX remoting mechanism to overcome this so that the Oracle Maps client can talk to the MapViewer in a different domain. This workaround has one limitation - the length of the non-AJAX remoting request can not exceed certain limit. If the requests are within the limit, everything will work great. That's why in most cases Oracle Maps application works fine across domain out of box.
But things are not always that simple. If your application displays some complex theme based FOI layers, especially those with JDBC queries and/or dynamic styles, the underlying FOI requests might become too long to be handled by the non-AJAX remoting mechanism. As the result, you will get the error message mentioned in the title.
To solve this problem, you have two options.
Option 1
Option 1
---------
Deploy a MapViewer to the same domain where your application is running. You don't have to configure this MapViewer to do any real mapping. You'll only need its proxy servlet, which can forward all requests from the client to the remote MapViewer that does the real mapping. The client will send all AJAX requests to the proxy servlet, which will then forward the requests to the remote MapViewer. Because the servlet is in the same domain as the application, there will be no cross-domain constraint when the client talks to the proxy.
In your application code, you need to call MVMapView.enableXMLHTTP(true) at the beginning of your showMap function.
function showMap()
{
MVMapView.enableXMLHTTP(true);
var baseURL = "http://hostB/mapviewer";
function showMap()
{
MVMapView.enableXMLHTTP(true);
var baseURL = "http://hostB/mapviewer";
mapview = new MVMapView(mapDiv, baseURL) ;
...
}
...
}
Option 2
---------
Configure a reverse proxy in your HTTP server to forward requests sent to it to the remote MapViewer. This reverse proxy should accept requests sent to http://hostA/mapviewer/* and forward them to http://hostB/mapviewer/*. If your HTTP server is Apache, then this can be done by adding the following two lines inside mod_proxy section in httpd.conf.
Configure a reverse proxy in your HTTP server to forward requests sent to it to the remote MapViewer. This reverse proxy should accept requests sent to http://hostA/mapviewer/* and forward them to http://hostB/mapviewer/*. If your HTTP server is Apache, then this can be done by adding the following two lines inside mod_proxy section in httpd.conf.
<IfModule mod_proxy.c>
ProxyRequests Off
<Directory proxy:*>
Order deny,allow
Deny from 127.0.0.1
Allow from all
</Directory>
ProxyPass /mapviewer/ http://hostB/mapviewer/
ProxyPassReverse /mapviewer/ http://hostB/mapviewer/
</IfModule>
Your application code should act as if the MapViewer is in the local domain, because the proxy is acting as a local MapViewer instance. For better performance, you should explicitly specify the remote tile server URL when creating a tile layer. By doing so, you're telling the Oracle Maps client to get the map tiles directly from the remote MapViewer, not via the proxy, which also means better performance.
function showMap()
{
var baseURL = "http://"+document.location.host+"/mapviewer" ;
mapview = new MVMapView(mapDiv, baseURL) ;
mapview = new MVMapView(mapDiv, baseURL) ;
tileLayer = new MVMapTileLayer("myds.my_map", "http://hostB/mapviewer/mcserver");
...
}
}
Friday, July 31, 2009
display labels with superscripts and subscripts
My colleague Albert Godfrind just sent me a nice tip on how to display map labels containing subscripts and superscripts. Thought I will share it with you all.
I found the following to work for me. I create a table (using a NVARCHAR2 column for storing the label), then I populate it with two rows: one using the standard label, one using the sub/superscript notation.
create table labels (
id number,
label nvarchar2(30),
geom sdo_geometry
);
insert into labels (id, label, geom)
values (
1,
'm3 CH4 / min',
sdo_geometry (2001, null, sdo_point_type (1,1,null), null, null)
);
insert into labels (id, label, geom)
values (
2,
unistr('m\00B3 CH\2084 / min'),
sdo_geometry (2001, null, sdo_point_type (1,2,null), null, null)
);
commit;
insert into user_sdo_geom_metadata values (
'LABELS',
'GEOM',
sdo_dim_array (
sdo_dim_element ('x', 0, 100, 0.05),
sdo_dim_element ('y', 0, 100, 0.05)
),
null
);
commit;
create index labels_sx on labels (geom) indextype is mdsys.spatial_index;
Mapviewer then nicely renders the super/sub scripts as illustrated!
I found the following to work for me. I create a table (using a NVARCHAR2 column for storing the label), then I populate it with two rows: one using the standard label, one using the sub/superscript notation.
create table labels (
id number,
label nvarchar2(30),
geom sdo_geometry
);
insert into labels (id, label, geom)
values (
1,
'm3 CH4 / min',
sdo_geometry (2001, null, sdo_point_type (1,1,null), null, null)
);
insert into labels (id, label, geom)
values (
2,
unistr('m\00B3 CH\2084 / min'),
sdo_geometry (2001, null, sdo_point_type (1,2,null), null, null)
);
commit;
insert into user_sdo_geom_metadata values (
'LABELS',
'GEOM',
sdo_dim_array (
sdo_dim_element ('x', 0, 100, 0.05),
sdo_dim_element ('y', 0, 100, 0.05)
),
null
);
commit;
create index labels_sx on labels (geom) indextype is mdsys.spatial_index;
Mapviewer then nicely renders the super/sub scripts as illustrated!
Tuesday, July 28, 2009
Java API: setting array-type binding variables for a theme
MapViewer supports pre-defined themes that contain query conditions with one or more binding variables. For instance, we can define a theme COUNTIES_BY_STATES that displays all the counties whose State abbreviations are in a list supplied by user at run time.
The query condition for this theme may look like the following:
( state_abrv in (select column_value from table(:1)) )
In other words, when this theme is to be displayed, it will require a list of State abbreviations such as 'CA', 'WA', 'NY' as the value for its query's binding variable. Note that this binding variable itself requires a SQL TABLE type.
So how do you supply such an array of state abbreviation codes to the MapViewer from your Java application? The following shows the sample code that does just that.
String [] values = { "MA","NY","ME","NH" }; //list of state abbreviations
ArrayParameter states = new ArrayParameter("MV_STRINGLIST",values);
Object [] values = new Object[1]; //array containing all binding variable values
values[0] = states;
mapViewer.setThemeBindingParameters("COUNTIES_BY_STATES", values);
Note that basically you must use MapViewer's public ArrayParameter class to supply the list of states (or list of any strings/numbers) for a single table-type binding variable. Because in this case there is only one binding variable in this theme's query, the values array contains only one element, the ArrayParameter instance.
Note also MV_STRINGLIST is a SQL list/table type created by MapViewer automatically in each data source (database schema). Please check the 11g MapViewer user's guide on more information about this pre-created type.
The query condition for this theme may look like the following:
( state_abrv in (select column_value from table(:1)) )
In other words, when this theme is to be displayed, it will require a list of State abbreviations such as 'CA', 'WA', 'NY' as the value for its query's binding variable. Note that this binding variable itself requires a SQL TABLE type.
So how do you supply such an array of state abbreviation codes to the MapViewer from your Java application? The following shows the sample code that does just that.
String [] values = { "MA","NY","ME","NH" }; //list of state abbreviations
ArrayParameter states = new ArrayParameter("MV_STRINGLIST",values);
Object [] values = new Object[1]; //array containing all binding variable values
values[0] = states;
mapViewer.setThemeBindingParameters("COUNTIES_BY_STATES", values);
Note that basically you must use MapViewer's public ArrayParameter class to supply the list of states (or list of any strings/numbers) for a single table-type binding variable. Because in this case there is only one binding variable in this theme's query, the values array contains only one element, the ArrayParameter instance.
Note also MV_STRINGLIST is a SQL list/table type created by MapViewer automatically in each data source (database schema). Please check the 11g MapViewer user's guide on more information about this pre-created type.
multi-line labels
MapViewer supports labels or annotations that contain a newline char (ASCII char code 10). However by default such characters are ignored by MapViewer during the map text labeling process, and the label text appears in a single line (unless there is a text wrap width specified.)
To force MapViewer to honor the newline chars in a theme's label texts, the theme must use a TEXT labeling style with the "Honor Newline" attribute set to true. This can be done from MapBuilder's TEXT style editor, by simply selecting the Honor newline check box. Now any theme that uses this TEXT style will automatically display any label text containing newline characters with multiple lines.
If your application is written using the MapViewer Java API, and you need to add individual geo-features with multi-line text labels to your map, then additional caution must be taken when constructing the label text with the newline char. Namely, the label text for a feature must be constructed like this:
String textLabel = "LINE 1" + "%26#10;" + "LINE 2";
Typically in an XML document (the Java API eventually will convert your current map request into an XML request before sending it to the MapViewer server), one uses to represent the newline char(10). Because we are sending the entire map request string using HTTP POST from map client to the server, the character '&' itself must be escaped as '%26', hence the odd-looking string %26#10; representing an HTTP-safe newline character.
So this is how you can construct a new feature with this text as its label:
mapViewer.addPointFeature(centreX, centreY, 8265, "C.RED", textLabel, "MY_TEXT_STYLE",
null, null, true);
where MY_TEXT_STYLE should be a TEXT type style with "Honor newline" enabled.
To force MapViewer to honor the newline chars in a theme's label texts, the theme must use a TEXT labeling style with the "Honor Newline" attribute set to true. This can be done from MapBuilder's TEXT style editor, by simply selecting the Honor newline check box. Now any theme that uses this TEXT style will automatically display any label text containing newline characters with multiple lines.
If your application is written using the MapViewer Java API, and you need to add individual geo-features with multi-line text labels to your map, then additional caution must be taken when constructing the label text with the newline char. Namely, the label text for a feature must be constructed like this:
String textLabel = "LINE 1" + "%26#10;" + "LINE 2";
Typically in an XML document (the Java API eventually will convert your current map request into an XML request before sending it to the MapViewer server), one uses to represent the newline char(10). Because we are sending the entire map request string using HTTP POST from map client to the server, the character '&' itself must be escaped as '%26', hence the odd-looking string %26#10; representing an HTTP-safe newline character.
So this is how you can construct a new feature with this text as its label:
mapViewer.addPointFeature(centreX, centreY, 8265, "C.RED", textLabel, "MY_TEXT_STYLE",
null, null, true);
where MY_TEXT_STYLE should be a TEXT type style with "Honor newline" enabled.
Monday, July 6, 2009
MapViewer 11g R1 production kit released
We are excited to announce that as part of Oracle's Fusion Middleware 11g R1 release, MapViewer 11g R1 is now available for download! Please navigate to OTN MapViewer home page here: http://www.oracle.com/technology/products/mapviewer/index.html. The 11g R1 software can be found under the Software link on the right side of the page.
Note that this release includes all the features and bug fixes of MapViewer patch 10.1.3.3 for WebLogic server, and we strongly encourage users to upgrade to the 11g R1 release to take advantage of the many new enhancements and a more stable Oracle Maps JavaScript API.
It supports not only WebLogic server (and Oracle AS/OC4J 10.1.3.x), but also Tomcat 6.x and JBoss 4.x. The full certification matrix can be found here:
http://www.oracle.com/technology/products/mapviewer/htdocs/j2ee_server_support.html
For the complete list of all the new and enchanced features of MapViewer 11g R1, please check out the New and Changed section of the MapViewer User's Guide, also downloadable from the OTN MapViewer home.
Note that this release includes all the features and bug fixes of MapViewer patch 10.1.3.3 for WebLogic server, and we strongly encourage users to upgrade to the 11g R1 release to take advantage of the many new enhancements and a more stable Oracle Maps JavaScript API.
It supports not only WebLogic server (and Oracle AS/OC4J 10.1.3.x), but also Tomcat 6.x and JBoss 4.x. The full certification matrix can be found here:
http://www.oracle.com/technology/products/mapviewer/htdocs/j2ee_server_support.html
For the complete list of all the new and enchanced features of MapViewer 11g R1, please check out the New and Changed section of the MapViewer User's Guide, also downloadable from the OTN MapViewer home.
Subscribe to:
Posts (Atom)