Airport AWOS Frequency/Phone KML file updated

I have updated the AWOS frequency, phone, phone and type file based on the FAA's 56-day subscription data. However, instead of using latitude and longitude included in the file, I joined the main airport file by airport ID. This requires making an assumption that all AWOSes are at the airport they serve. While that might not be the case, I get 643 more airports into the file then before. The AWOS file has 2,185 entries of which 1,542 have latitude and longitude. Almost every corresponding airport in the APT file has latitude and longitude. Thus, DCA will now show up with a phone number, but no frequency. The new KML will also show the AWOS type: ASOS, AWOS-1, AWOS-2, AWOS-3, or AWOS-A.

Finally, matching airports to AWOS increases the complexity. I had hoped to write a simple script that does text manipulation for all KML files -- one script altering one FAA text file to produce one KML file. As with the Special Use Airspace, however, there wasn't a good way to do it without using a database and relations.

Find them in the KML archive.

Brand Dimensions still won't stop scraping my site

Despite having a no BDFetch robots.txt directive, Brand Dimensions has downloaded hundreds of my original pages with photos on them. None of these pages mention any brand names of any companies, so I'm curious as to what BD is really doing. I'm guessing they could also provide some serious competitive intelligence to their clients. I just wonder what happens when they represent competing companies, like Coke and Pepsi. Here are some representative entries from my log files:

/var/log/httpd/access_log.1:72.14.164.139 - - [11/Aug/2009:07:25:27 -0400] "GET /carleton/reunionweb/WebPage-Full.00001.html HTTP/1.1" 200 1394 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.140 - - [11/Aug/2009:07:25:42 -0400] "GET /carleton/reunionweb/WebPage-Full.00015.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.197 - - [11/Aug/2009:07:25:57 -0400] "GET /carleton/reunionweb/WebPage-Full.00018.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.157 - - [11/Aug/2009:07:26:12 -0400] "GET /carleton/reunionweb/WebPage-Thumb.00023.html HTTP/1.1" 200 3648 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.179 - - [11/Aug/2009:07:26:27 -0400] "GET /carleton/reunionweb/WebPage-Full.00013.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.193 - - [11/Aug/2009:07:26:42 -0400] "GET /skiing/webdest/WebPage-Full.00011.html HTTP/

Brand Dimensions switched the name of their bot to sidestep robots.txt directives. Based on my own Google Analytics info, I can safely say a lot of people are interested in what Brand Dimensions is doing and how to stop it. More LinkWalker info here. Other webmasters report that the LinkWalker agent is also used by spambots harvesting email addresses for phishing attacks and the like.

Here are my latest robots.txt lines:
User-agent: BDFetch
Disallow: /
User-agent: BPImageWalker
Disallow: /
User-agent: VoilaBot
Disallow: /
User-Agent: LinkWalker/2.0
Disallow: /
User-Agent: LinkWalker
Disallow: /

Airspace, Special Use, and Airport KML files updated (not all US Airports are in the Western Hemisphere)

I updated the Airspace, Special Use, and Airport KML files for the FAA data covering the 56 Day Subscription from August 27 2009 to October 22 2009. I have also created batch files to call all the scripts, and I'm working on a T-SQL script to clear the tables and do a bulk import from the files.

Also, I corrected one incorrect assumption. Not all US Airports are in the western hemisphere. I assumed all longitudes were W and put a minus sign in the conversion to Google/KML decimal format from the FAA's all-seconds format. Wrong. In the updated file, MAJ (MARSHALL ISLANDS INTL) is in the right place. I'm still not sure Pago-Pago is correct, however.

The index of files is here.

Class B, C, D and experimental E Airspace KML files posted

FAA subscription data includes .shp (shape) files for ESRI products. Google Earth Pro ($400/year) is capable of opening these shape files directly, but the default setting is to make them opaque. Thus the class B airspace looks like one single area with no details. FWTools is a free open-source conversion tool to change GIS formats. It also does a much better job of converting the data, showing each individual airpspace and its low and high altitudes. The FAA marks the class E files as experimental, NOT TO BE USED FOR NAVIGATION.

The FWTools command to change from the FAA's .shp format to KML is as follows:
ogr2ogr -f "KML" C:\airports\class_b.kml C:\airports\class_b.shp The FAA lists airspace in order (e.g. Washington Class B Area A, Washington Class B Area B...), and fwtools maintains this order. When Google Earth displays this information, the last one listed goes on "top" of the area. Thus when you click on the center of a Class B zone, the widest area comes up, NOT the SFC to 10000 area. You can see the correct area when you click on the corresponding area layer, however. (I am currently working on a fix for this, but it requires some custom XML and SQL programming that's going to take some time.)

However, there is a weakness in Google Earth. The KML specification does not allow the creation of floating polygons. All non-plane polygons are extruded from the surface or ocean floor. Thus, there's no way to express in KML the upside-down-wedding-cake shape of your typical Bravo airspace. This may one day change, however, because Google is already adding its own extensions to the KML specification.

If you don't want to bother, just download the Airpspace KML files here: Archive link.The usual warning applies -- DO NOT USE FOR NAVIGATION. The KML archive is here.

Special Use Airspace in Google Earth

According to the FAA, there are 977 U.S. special use airspaces around the world. (Some are international, like the Pacific off of Guam.) By Special Use airspace, I mean the following FAA Types: ALERT AREA, MILITARY OPERATIONS AREA, PROHIBITED AREA, RESTRICTED AREA, WARNING AREA. (Another side note: It's not just the FAA that loves ALL CAPS. The Department State also uses ALL CAPS in its cables.) There are many areas of the United States, in CONUS, Alaska and Hawaii, that are special use.

The FAA flat-fixed file that is the source of this data is so convoluted that I have a new-found respect for companies like Jeppesen that produce the data in a readable format every 56-day cycle. The FAA stuffs 8 different tables into one giant flat file with a well-documented layout.

Google Earth is not perfect with its polygons, either. It's got bugs, especially when it comes to KML stylesheets. I wrote the style tags with VBScript out of SQL, so they are all the same, but in Google Earth, they don't all look the way they're supposed to. It feels like troubleshooting early versions of Netscape and IE when writing HTML. And no, I didn't use some fancy XSL to transform my queries into KML (XML). I used VBScript because it's the tool I know and it's fast.

The usual warning applies: DO NOT USE FOR NAVIGATION. My code isn't perfect, and neither is the FAA's data. There is no substitute for a pre-flight briefing. I ask about the airspace every time flying out of the ADIZ.

Download the file here: US Special Use Airspace.kml.

US Airports by Region in Google Earth & Maps via KML

Given that my other x64 4GB computer hung this morning when loading the large KML file this morning, I decided to break up the KML file into the following FAA regions:ALASKA, CENTRAL, EASTERN, GREAT LAKES, NEW ENGLAND, NORTHWEST MOUNTAIN, SOUTHERN, SOUTHWEST and WESTERN-PACIFIC. (Once again, the FAA really really likes ALL CAPS.)

I also set up a directory to hold all my KML files, including the original all US airports and the AWOS file at this location. To view in Google Maps rather than Google Earth, add "http://maps.google.com/?q=" to the front of the URL.

View all US Airports on Google Earth via KML

I finally took the time to write the script to take the raw National Flight Data Center APT.TXT file and write it straight to a Google Earth and Maps-compliant KML file. It's pretty basic XML, with a little math to convert the FAA's all-seconds format into decimal coordinates.

A word of warning: DO NOT USE FOR NAVIGATION. This is unverified data from the FAA, and my math is often accurate to about a factor of ten. Also, there are 13,569 airports, which means there are 13,569 points in this file. It's enough to max out the processor running Google Earth on your computer. However, when you zoom in to a single state, it will speed up because most points are off-screen. I filtered out non-airport landing facilities: BALLOONPORT, SEAPLANE BASE, GLIDERPORT, HELIPORT, STOLPORT, ULTRALIGHT, so you will find none of those in this file. (The FAA likes things in ALL CAPS.)

The pop-up description of each data point includes the airport code, airport name, UNICOM frequencies, CTAF, and pulic or private. Download the US Airports KML file here.

Here's a screenshot of Google Earth with the KML loaded:

Displaying FAA Airport AWOS Data in Google Earth and Google Maps with KML

The FAA releases updated Airport data every 56 days via the National Flight Data Center Portal. However, since government contracts apparently don't have any usability requirements, it's difficult to navigate, and the data is still in a fixed-field format reminiscent of cobol and mainframes. (Which, by the way, is a format that plays a huge role in banking even in 2009.)

After cleaning it up and matching airports with their AWOS data, I created a KML file. KML is simple XML -- you just need to do get the elements correct. The AWOS frequency and phone number is in the pop-up for each point.

With this file you can see all airports for which the FAA has latitude and longitude data. The FAA doesn't have coordinates for all airports, so many are missing. (Why they don't have coordinates for major airports like DCA in the AWOS file is beyond me. They have them in the airport facilities file.)

To view it in Google Maps (if you don't have Google Earth), use the
http://maps.google.com/?q=http://cw.sampas.net/kml/US_AWOS_20090827-20091022.kml format, which will pull the KML from the link after the ?q=.

Coming soon: VBScript to take the raw FAA files and produce KML. VBScript isn't the best way to export text to XML, but it is easy to write and very quick.