Wednesday, 8 December 2010

Map of Wikileaks list of facilities 'vital to US security'

One of the most interesting, and perhaps controversial, bits of information from the recent release of the Wikileaks US embassy cables has been the list of locations deemed vital to US security. Since this information is now in the public domain, we were interested in visualizing the data spatially.


The first step was to take the rather messy data and identify individual entries. In some cases it was no more specific than "Indonesia: Tin Mine and Plant". In other cases it named a pipeline, a port, or a city in which an undersea cable made landfall. Next, using the worldatlas.com geocoder (as well as some Wikipedia entries), we (along with the much appreciated help of Zach U. and Tim B.) located an approximate latitude and longitude for each of the locations mentioned in the cables.

We wish to emphasize that the locations in our mashup are only for the cities in which these critical facilities are located, and not the actual facilities themselves. In some cases, the location in the map is no more detailed than the country. Given this relative inaccuracy, this map does not present any security threat whatsoever. Moreover, all the data sets used for this geo-coding are openly available on the Internet and could easily be replicated by anyone.

Our purpose is to visualize the patterns exhibited by this particular data set, which are illustrated below. The categories in the legend are our own classifications based on the information provided by Wikileaks (you can view a larger, non-embedded version of our mashup here, or download a KMZ file of the mashup here that should automatically load into Google Earth. The KMZ version also allows you to turn on and off categories as you wish).




Map Legend

It is interesting to note that the vast majority of these facilities are not directly military-related. Even the ones that we mark as 'military' are related industrial facilities rather than actual bases. Instead, the list seems to focus on non-military topics such as telecommunications, energy and pharmaceuticals. Much of the list is also focused on supplies of important raw materials (Bauxite, Chromite, and Rare Earth Minerals), as well as the ability to move products through ports and shipping channels.

Share of Facilities by Type
Telecommunications 28%
Energy 18%
Pharmaceuticals 13%
Border_Crossing 11%
Raw_Material 10%
Port 7%
Military 5%
Industrial 4%
Shipping 3%
Dam 2%

These data offer a fascinating insight into the ways that the national security priorities of the United States span the entire globe. This global web of essential facilities goes a long way to explain the fact that the US Department of Defense has more military facilities around the world than all other nations combined. The globalization of the world economy means that facilities that are vital to the communication, health, and economic needs of the U.S. are scattered across the planet; and this ultimately means that the U.S. (as well as other developed and developing countries) have to contend with new and changing notions of what "security" means in the 21st century.

We are truly living in a network society.

Tuesday, 7 December 2010

Geospatial Information Systems Market to Experience Vibrant Growth

The worldwide market for Geospatial Information Systems (GIS) is forecast to grow 65 percent over the next five years, representing a compound annual growth rate of 10.5 percent, according to a new ARC Advisory Group study.  With the global economic downturn now predominantly in the past, capital spending on information technology has rebounded vigorously.  The GIS market has taken part in this technology rebound and is expected to experience vibrant growth.
 
Geospatial“Traditionally strong GIS market segments such as electric power, oil and gas distribution, and divisions of federal governments continue to expand their use of GIS solutions. Meanwhile, more contemporary segments such as insurance, real estate, and retail are expected to increase GIS adoption along with the global economic rebound,” according to Clint Reiser, Enterprise Software Analyst, and the principal author of ARC’s “Geospatial Information Systems Worldwide Outlook”.
 
Government Policy Drives GIS SalesA large percentage of total GIS sales is to the government and utility sectors. This characteristic makes the GIS market sensitive to changes in government policy.  The Smart Grid initiative in the US, the European 20/20/20 initiative, and India’s power development program are currently stimulating growth in GIS sales to the electric power industry.  The US Pipeline Integrity, Protection, Enforcement, and Safety Act is expected to drive GIS sales to the oil &gas distribution industry as suppliers employ GIS to support risk-based integrity management programs and improvements in asset reporting capabilities.  At the same time, government spending at the federal level on infrastructure and technology has been initiated in many regions as means of stabilizing the economy, supporting future growth, and maintaining current programs.  Some of this spending is expected to generate GIS sales to government agencies.
 
Modern IT Reshapes the GIS MarketMobile solutions, web services, and cloud computing have become influential factors within the GIS industry.  Mobile GIS has become more prevalent and shows promise for increased adoption in network design and maintenance functions.  The software-as-a-service business model is now well established as the delivery mechanism for geospatial business analysis solutions.  Meanwhile, the improvements in connectivity provided by cloud computing have also brought about an influx of GIS products that integrate imaging and GIS data with time-based information such as traffic reports to create mash-ups. Improvements in connectivity are also providing increased visibility into GIS data sets stored within organizations, thereby decreasing duplicate purchases and unnecessary costs.  The continuing adoption of mobile GIS, the SaaS delivery model, and other forms of cloud computing have great potential to transform and expand segments of the GIS market in the near future.
 
Emerging Markets Primed for GrowthEmerging markets such as China, India, and Brazil continue to invest in infrastructure for electric power distribution, water & wastewater, and telecommunications.  GIS sales to these regions will increase as the local utilities adopt GIS to improve their infrastructure management processes. 

Monday, 6 December 2010

Loqate Launches Worldwide Geocoding Solution



Loqate, the specialist in global location data announces the launch of a new worldwide geocoding software solution.

Covering over 240 countries, Loqate provides rooftop, street, city, or postal code level geocoding for almost any address or location around the world.

“This new solution is revolutionary in two ways,” comments President/CEO of Loqate Inc, Martin Turvey. “ Our advanced parsing and address verificaton engine enables us to obtain geocode matches on even the very worst data, including non-Roman alphabet based languages. Secondly, the worldwide coverage at the level of granularity provided by the solution, all through a single interface, is completely unique.”

“Geocoding is playing an increasingly important part in many businesses”, adds Loqate’s VP of Sales, Liat Perlman. “From traditional data quality markets, through freight and transportation, GIS and mapping, insurance risk assessment and more recently, the massive growth of Location Based Services, being able to pinpoint an accurate location of an address is core to so many business processes.”

The Loqate Geo-data Engine is available as a developer’s toolkit for integration into a wide variety of applications and business processes and is designed to run on most operating system platforms.

Loqate’s solution will soon be available in a Software as a Service model.

About Loqate

Loqate provides international addressing and geo-location software solutions, enabling clients to make better use of their data in areas such as data quality, direct marketing, fraud detection, insurance risk assessment, transaction monitoring, logistics & supply chain, online payment processing, as well as GIS, mapping and location based services. Using a combination of advanced algorithmic analysis and comprehensive reference sources, Loqate can identify, verify, correct and enrich geographic related client data, adding valuable information to increase its value. Loqate solutions are global, covering over 240 countries around the world - every populated world territory.

Loqate is based in Redwood City, CA, and operates through a number of strategic partnerships throughout the world.

www.loqate.com

Saturday, 4 December 2010

Earth from Space: Eye of Africa

A stunning Envisat image capturing spectacular geological phenomena – the ‘Eye of Africa’ (right) and a magnetic mountain (centre) – in the Sahara Desert of Mauritania in northern Africa.


Peeping out from a sea of golden sand, the remarkable circular Richat structure resembles an eye from space. Once thought to be formed by a meteor impact, it is now believed to be the result of geological uplift exposed by wind and water erosion.
Different rates of erosion on the varying rock types have formed concentric ridges; the more erosion-resistant rocks form high ridges (blue and purple), while the non-resistant rocks form valleys (yellow).


The surrounding dark area forms a plateau of sedimentary rock standing some 200 m above the surrounding desert sands, with the peak of the outer rim some 485 m above sea level. Sand is encroaching into the Richat structure’s southern side.
Mauritania’s highest peak (nearly 1000 m), the Kediet ej Jill Mountain, is visible northwest of Richat. It appears bluish because it is made completely of magnetite, a natural magnet. Owing to its inherent magnetic properties, the mountain disrupts navigational compasses.
Western Sahara is visible along the top and left.
This image was acquired by Envisat’s Medium Resolution Imaging Spectrometer on 1 November at a resolution of 300 m

A close-up of the Richat structure in the Sahara Desert of Mauritania in northern Africa. Credits: ESA

Friday, 3 December 2010

Google Launches Earth Engine for Carbon Accounting

Click here for a nice video:

http://www.youtube.com/watch?v=MnCf9Gjz720
As promised last year at the COP15 climate talks in Copenhagen, the Google Earth Engine launched today at the COP16 Climate Change Conference in Cancun. Earth Engine is a dynamic digital model of our planet that is updated daily, with 25 years of satellite imagery from the Landsat imagery catalog and tools to analyze the imagery. The tools were made available to rainforest nations in advance of this week’s climate talks as both a means of assessing carbon in their forests and as a testbed for system development.

Earth Engine is not only a core platform for hosting imagery and other geospatial data, it is also an analysis processing framework with an Earth Engine API for scientists to write their own code to customize the platform for their area of interest. Google is also providing 20 million CPU hours for free to scientists and developing countries to take advantage of the new analytical tool.

Earth Engine has been a collaborative effort that has included contributions from Greg Asner of the Carnegie Institution for Science, Carlos Souza of Imazon and Matt Hansen of the Geographic Information Science Center at South Dakota State University.
It is hoped that this year’s talks will result in an agreement on the U.N. initiative known as Reducing Emissions from Deforestation and Forest Degradation in Developing Countries (REDD), and this tool will act as a means to monitor and enforce such an agreement, tracking environmental change and deforestation rates in real time.

Thursday, 2 December 2010

GIS on a Stick

GIS on a USB stick. Please if you have any comments post download pop them in the comments area as it would be great to have some decent impartial reviews.


Announcement: Portable GIS Version 2 is released! Download it here

The philosophy behind this idea was to provide beginners with a ready-installed and configured stack of open source GIS tools that would run in windows without the need for emulation or a live cd. By taking out the often difficult installation and configuration, I hope to make it easier for beginners to get started with open source GIS, so they are not put off before it gets interesting and fun. Not only that, but having a fully self-contained GIS system may prove useful in a number of real-life situations.
Newly updated version 2 contains a self-contained installer, updated versions of all the constituent software packages, a new control panel, and improved documentation.
The current set of software includes:
  • Desktop GIS packages QGIS (with GRASS plugin), uDIG and gvSIG,
  • FWTools (GDAL and OGR toolkit)
  • XAMPPlite (Apache2/MySQL5/Php5),
  • PostgreSQL (version 8.4)/Postgis (version 1.4),
  • Mapserver, OpenLayers, Tilecache, Featureserver, and Geoserver web applications.
The packages and menu system are all open source, but each component has a separate license. Some of the components have been altered in order to make them drive-letter independent.
Provisoes:
  • This is not for production use. In order to keep end user configuration to a minimum there are a number of security holes and as such it should be used for demonstration and home use only.
  • It is also not “stealth GIS”- no attempt has been made to leave no trace on the host system.
Support and Assistance:

Wednesday, 1 December 2010

Awesome New Tool, Needlebase.

Needlebase allows you to view web pages through a virtual browser, point and click to train it in understanding what fields on that page are of interest to you and how those fields relate to each other. Then the program goes and scrapes the data from all of those fields, publishes them into a table, list or map, and recommends merges of cells that appear to be mistakenly separate. It's very cool and it lets non-technical people do things with data quickly and easily that we used to require the assistance of someone more technical to do.

Video is here:

http://www.youtube.com/watch?v=58Gzlq4zSDk&feature=player_embedded

Investigative journalism

Last month a local newspaper reported that a big new data center had opened in Salt Lake City with a mystery anchor client. The paper believed the client was Twitter, as the company has said it was going to open its first off-site data center in Utah at an undisclosed date.
We used Needlebase to look at all the tweets from people on the Twitter list of Twitter staff members and extract the username, message body and location, if exposed. Needlebase scraped the last 1500 Tweets in less than 5 minutes. We displayed them on a map and saw that there was just one Tweet published in that time from Utah: a Twitter Site Operations Technician who had just left San Francisco to move to Salt Lake City, complaining about Qwest router problems. That wasn't quite confirmation, but it sure felt like a valuable clue and was very easy to come by thanks to Needlebase.

Data Re-Sorting

Last night I found a solution to a long-running issue I've been struggling with. I've got this list of 300 blogs around the web that cover geotechnology (that's a whole other story) and have them all run through Postrank. That service ranks them in order of most to least social media and reader engagement per blog post.
Wouldn't it be great to extract that data over time, to track it and to turn it into blog posts? I think it would. I couldn't figure out how to get all the data out that I wanted though.
Enter Needlebase. Last night I pointed Needle to my Postrank pages for geotech blogs and in minutes it pulled down all the data I wanted. I exported that data as a CSV, uploaded it to Google Docs as a spreadsheet, did a little subtraction and now have the following chart tracking the top 300 geotech blogs on the web. Now in my handy spreadsheet, I was able to set up a function to show me which blogs jumped or fell in the rankings the most over the previous week. Thanks, Needlebase!Event Preparation
I've written here about how to use Mechanical Turk to get ready and rock an industry event. Needlebase can prove useful for that as well.

The DIY Data Hackers Toolkit

I put Needle in my mind in between two other wonderful tools. On one end of the spectrum is the now Yahoo-acquired Dapper, which anyone can use to build an RSS feed from changes made to any field on any web page. (See: The Glory and Bliss of Screen Scraping and How Yahoo's Latest Acquisition Stole and Broke My Heart)
One the other end of the spectrum is the brand-new Extractiv, a bulk web-crawling and semantic analysis tool that's also remarkably easy to use. Earlier this month I used Extractiv to search across 300 top geotech blogs for all instances of the word "ESRI," all entities mentioned in relation to ESRI and the words used to describe those relations. The service processed 125,000 pages and spit out my results in less than an hour for less than a dollar. That's incredible - it's a game changer.
Needlebase is too. It sits somewhere in between Dapper and Extractiv, I think. These tools are democratizing the ability to extract and work with data from across the web. They are to text processing what blogging was to text publishing.