Commuter Bicycle Review: Breezer Uptown 8

After a year-and-a-half and over 1,000 miles on the Breezer Uptown 8, I thought I'd write a review. There is a dearth of material on commuter bicycles and bicycling. The trade press is advertiser-focused and almost all of their bicycle reviews are non-critical and improve as the price increases. Most bicycle stores don't focus on the commuter cycle market segment. If one store started doing it in or around DC, I suspect they'd get a hardcore following pretty quick. There's a rush-hour on the CCT even in the winter. (Minneapolis claims 3,000 winter bicycle commuters -- they plow 50 miles of bicycle trails there.)

First: my ride. On days that I ride (I'm not a 5-day-a-week rider yet), I ride seven miles (mostly downhill) into work in 35 minutes. That same ride home (mostly uphill) takes about 45 minutes. I carry a laptop and a full set of work clothes each way in panniers on a rack.

Overall, the Breezer Uptown 8 is a great bicycle. However, there are some things to nit-pick about, mostly because commuter bicycles are relatively new in the United States. Nobody reviews them because they're not sexy.

First off, the Uptown is comfortable. I ride in an upright position. The seatpost is suspended, and overall it's a pleasure to ride. However, it's not a fast bike. It's got full fenders and a fully-enclosed chaincase. The only place I get splashed in rain is my feet.

This is my Breezer's fully enclosed chain. It keeps the chain clean and reduces the maintenance, which is a nice feature. When you do maintenance, however, it takes longer to get to the chain.

You can see photos of my nitpicks here.
I bought the bike at Bikes at Vienna in Vienna, and get some help with maintenance from Griffin Cycle in Bethesda. There's nothing wrong with the service at Bikes at Viennna, it's just that Griffin Cycles is close to home.

While the generator hub is on the front wheel, the rear light is in, well, the rear. Those two buttons are the connectors, and that's where the cables disconnected the third time I rode the bike. The cable runs inside the frame until the bottom bracket, where it connects to the two little buttons on the rear fender. Wires built into the rear fender carry the current to the rear light, until...

These two buttons are the original connections for the rear light. At least one of them came off every ride. Vibration would make it happen. I love the bike, but those connectors are engineered to fail. And it's the rear light, so it FAILS SILENTLY when I'm in traffic at night. Not that all rear lights don't fail silently -- just don't ever rely on a single one. Not that all the lights and a bright reflective jacket will make motorists "see" you. 

The generator on the Uptown 8 is in the Shimano hub. It's a great feature and the generator's always there. It's silent in operation, too, unlike the old-school tire-wheel generators I used to see in Europe on the 1970s. Some say the drag is not noticeable, but I notice it. Almost my whole ride home is uphill, and in the winter it's uphill and dark

This is the Shimano Nexus hub on the Breezer Uptown 8. The internal shifting is really nice, although it took some adjusting of the cable to get it to shift correctly every time. At my first tune-up, the bike mechanic told me the cable was misrouted. Dude, I bought the bike from you.

This is the rear rack, where I carry my laptop and work clothes each day that I ride. It's generally a good rack, but when you keep your panniers at the rear...

It can be tight with the pannier clips until you figure out that Arkel clips are laterally adjustable.

The enclosed chain and internal-shifting rear hub are great, but I can't replace the rear tire in the field. I left the super-heavy City tire on the rear, but replaced the front tire with something much lighter. Creating an enclosed chain that's easy to deal with and lets me replace a tire/tube in the field is an engineering challenge.

The built-in Shimano front hub generator and lights are great. The headlight is pretty bright. I did get a Niterider light because the trail is not lit and pitch black. The rear light connections are weak and the light failed after going over a few bumps. Some new wire fixed that. The generator does add a little drag when it turns itself on. It's great having lights without ever having to worry about a charger or batteries. I have two Planet Bike flashies (one of them even works consistently)  mounted on my panniers. That worked out especially well after I wrecked and my black laptop pannier fell off and onto the dark, unlit trail at night.

The stock tires, Schwalbe City, are heavy. I replaced the front with a Conti 1" which works fine. The rear tire I haven't replaced because the rear wheel is a pain to take off. You need to shift to 4th gear, remove the tiny screws in the chaincase, and then use a 15mm wrench. Not something to do on the trail after dark.

Update: When I started my new job, my commute grew from 7 miles each way to 11.5 miles each way. I'm now riding my 20-year-old no-suspension mountain bike back and forth to work. However, my Breezer Uptown still sees plenty of use on the weekends with an Adams Trail-a-Bike or a Burley Trailer. The Uptown's Fenders are perfect for protecting the kids behind the bike from road spatter.

I also ride it to the local Metro stop on days I don't ride all the way in to work. When I'm wearing pants for work, the chainguard is awesome. This is also my guest bike.

Airspace KML files updated for the December 17 cycle

I updated the airspace KML files again -- skipping one release. The latest covers from December 17, 2009 throughFebruary 11, 2010. You can find them in the archive here.

Migrating Movable Type and Gallery2 to a new (Fedora) server

I loved my Dell 1750 server. It has plenty of power and a great 3Ware RAID card, two 500 GB RAID-1 drives, 4 GB RAM, and ran Red Hat Linux. I bought it from the Dell Outlet site several years ago when my condo fee included electric. Since then I have moved. I pay my own electric bill, and my 1750 consumes 150 watts at idle. When I publish with Movable Type or Gallery, power consumption exceeds 200 watts. I pay $0.150845894 per kilowatt-hour. (That's summing the separate generation, transmission, distribution, demand-side surcharge, and adding the gross-receipts tax. Pepco doens't make it easy to figure out what you're paying.) With a thirty-day month, that's 108 kwh, which comes to $16.29135658. That's $16/month, just for idling. And that doesn't use the noise of the server in my office or the additional AC required in summer.


That may seem expensive, but it's far cheaper than getting that much server capacity at Rackspace. However, I don't need that much capacity. I can trade processor power for power savings, keep the disk space and RAID card, and switch to an Atom-based server. My current FreePBX Atom server runs at 40 watts with an analog card powering two FXS modules. I bet I can match that on a new server. I'd get the new Supermicro Atom Server, but it has space for only one 3.5 inch hard drive. I need two. Thus I'll be using another miniITX case. In the meantime, everything's running on another old Dell tower box.


How did I move it? First I though reinstalling all the software from scratch would be a good idea. I'd get a nice clean, efficient build. But that took way too long, and I'd have to re-customize my templates and tweaks. I had MySQL backups running for a while, why not start testing the restores? I used rsync:

rsync -avz /var/www/ -e ssh:user@mynewserver /var/www/

(Note: Please study rsync syntax. Those / at the ends make a big difference.)
It worked. Next I had to tweak the new httpd.conf file. I couldn't just copy the old one, because I was using the latest Apache version. But I could use almost all of the old file. I just needed to adjust the modules it loaded, because several have changed names.


Then I restored the databases:

mysql -u root -p

enter your password. (You ARE using a PW for MySQL root, aren't you?)

mysql> create database mynewdb

then

mysql> quit;

then

$ mysql -u root -p [mynewdb] < [backupfile.sql]

But that generally does not restore your user privs on the db. Back to mysql:

mysql -u root -p

then

mysql> use mynewdb;



mysql> GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'localhost' 



IDENTIFIED BY 'pAssW0rd' WITH GRANT OPTION;



mysql> flush privileges;

Don't forget the above step or you'll need to restart MySQL to get it to work.

mysql> quit;

Then double check that the user and pass from above match your config files.


Finally, test your applications. Gallery2 and MovableType worked fine. Your milage may vary. My office is almost silent now.

Using Logparser to dump Bluecoat log files into SQL

Working with Bluecoat files in the raw can be time-consuming. Findstr and grep only work so fast. Windows grep is slow. I know SQL syntax OK, so I tend to dump logfiles into databases to analyze them for activity. There are certainly other ways to do it, such as using a reporting tool for Bluecoat. (Splunk's free Bluecoat application, e.g.).

Theoretically, Bluecoat logfiles are the same as W3C web server log files that logparser can consume via the -i:W3C directive.

You can see the fields in a Bluecoat log below.

#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id

For some reason, Bluecoat leaves two spaces between cs(Referrer) and sc-Status, so all the columns to the right of sc(Referrer) past that will be one off. BlueCoat also leaves spaces in cs-categories and surrounds them with quotation marks, so you need to specify -dQuotes:on. Logparser doesn't have a quick and easy way to handle the double-spaces issue, so I wrote a VB Script to handle it. (VBScript is pretty quick at text handling and it's much faster than using search and replace in WordPad or Notepad on a 500-1000 MB File.)

Here's the VBScript:
'start

Set objFSO = CreateObject("Scripting.FileSystemObject")
'change this line to wherever you want to read the input from.
Set objTextFile = objFSO.OpenTextFile("c:\myBluecoatlog.log",1)
Set objNewFile = objFSO.CreateTextFile("c:\myCleanBlueCoatlog.log")
Do Until objTextFile.AtEndOfStream

myString = objTextFile.Readline
objNewFile.WriteLine(Replace (myString, " ", " "))
Loop
'end vbscript
Here's the logparser file:
-------------------start
SELECT TO_LOCALTIME(TO_TIMESTAMP(date, time)) AS date,
time-taken,
c-ip,
cs-username,
cs-auth-group,
x-exception-id,
sc-filter-result,
cs-categories,
cs(Referer) AS Referer,
sc-status AS scStatus,
s-action,
cs-method,
rs(Content-Type) AS ContentType,
cs-uri-scheme,
cs-host,
cs-uri-port,
cs-uri-path,
cs-uri-query,
cs-uri-extension,
cs(User-Agent) AS UserAgent,
s-ip,
sc-bytes,
cs-bytes,
x-virus-id

INTO BlueCoat4
FROM c:\myCleanBlueCoatlog.log
------------------end
And here's the command line for logparser. (Save the logparser file as c:\scripts\log\bluecoat.sql)

logparser file:c:\scripts\log\bluecoat.sql -i:W3C -o:SQL -server:sqlservername -database:BLUECOAT -createtable:ON -dQuotes:ON


Statistics:
-----------

Elements processed: 613076
Elements output: 613076
Execution time: 241.20 seconds (00:04:1.20)
About 2500 lines/sec. Processor utilization is almost zero for SQL and logparser, so it's all about disk time.

The above is from a file that's 310,935,417 bytes large. That means BlueCoat logs are about 507 bytes per line, or 0.5k per line before compression. The last time I checked BlueCoat gz compression, it was about 15% of the original file size. Compressed, the line would cost you 76 bytes.

How I compiled Darkice

Usually, installing an application from source on Linux/Solaris/BSD is easy:

1. ./configure --help (Always look at the help to see the options. It makes a difference if, for instance, you compile php without support for MySQL.)


2. ./configure


3. make


4. make install

However, with Darkice, it's prerequisites are numerous, and Darkice's configure doesn't find it's prereqs if they're installed in the standard locations. I've done this twice so far without documenting how I did it, so this time, I'm writing it down.

Here's my configure line:
./configure --with-vorbis-prefix=/usr/local/ --with-lame-prefix=/usr/local/lib/ --with-twolame=prefix=/usr/local/lib/ --with-faac-prefix=/usr/local/lib/

Then you'll get this when you launch darkice:
darkice: error while loading shared libraries: libmp3lame.so.0: cannot open shared object file: No such file or directory

So you need to link that, and when you link that you'll get the next error, so here are both:
ln -s /usr/local/lib/libmp3lame.so.0 /usr/lib/libmp3lame.so.0
ln -s /usr/local/lib/libfaac.so.0 /usr/lib/libfaac.so.0.
If you're wondering what links you're missing, try
ldd /usr/local/bin/darkice
If one of the links to the libraries reads "missing" then that's the one you need to link.

Yum install darkice might work for you, but then again, if you need all the features, it probably won't.
Prereq links are below. Generally ./configure, make, make install works well with all of them, but you really want to track exactly where each lib gets installed -- usually /usr/local/lib/.
Lame
Twolame
libogg
libvorbis
faac

Preqrequisite for faac or twolame -- I forget which:
libsndfile
Prereqs I didn't neeed:
Alsa
Jack

How to tell when someone Googles you

Case 1: You Google me and click on my page

Yes, I'm using google as a verb. If you Google me and click on one of my pages, my web server logs the information:
1.2.3.4 - - [01/Oct/2009:10:23:41 -0400] "GET / HTTP/1.1" 200 7186 "http://www.google.com/search?hl=en&source=hp&q=larry+s&aq=f&aqi=g10&oq=&fp=7d15299a959dbb33" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)"

As you can see, I get your IP address, a date, an offset to Universal Time (-0400), a verb (GET, in this case / means my default site page), a status code (200=OK), and a referrer. From the referrer, I can tell you Googled me with the phrase "larry s". Finally, I also get some information about the browser you used, Firefox, and the operating system, Windows XP with service pack 2. There's a chance you may have used a anonymizing proxy, but I'd still get an entry. (Generally, Anonymizer says "TuringOS," so I know it's them.)

Case 2: You Google me and don't click on my page.

That's more difficult but not impossible, because I have a Google AdWords account. I bought my own name as a keyword. Google AdWords works by selling keywords for search insertion. It's an open market, with the second-highest bidder winning in a dutch auction that is Google's revenue machine. When you buy a keyword, you get two measures back from Google:

1. how many impressions it got (viewing)


2. how many clickthroughs it got. (someone clicks on the ad)

A keyword ad's success is measured by the ratio of impressions to clickthroughs. The more clickthroughs per impression, the better. So if you don't click on my ad link, which I have made irresistable by promising dirt on me, I still know that someone Googled me, because the impression counter increments with each search. If you click on a regular page on my server rather than the keyword ad (Google calls this "organic"), we're also back to case one above. If you don't click on any of my links, I don't get any of the details from case one.


And that's how I know that you Googled me. If you're wondering if you've been Googled, but don't have a web site with logs you can comb through, or don't want to set up a Google AdWords account, try Google's external keyword tool. Just don't forget to un-check the synonyms box.

Did you know October is Cybersecurity Awareness Month?

Niether did I. Hardly anyone knows, because few people take DHS seriously, and nobody outside of the Federal government has said "Cyber" since the nineties. I attended a computer security conference recently and listened to a panel of current and former federal officials speak about "Cyber" security. They might one day be able to secure government systems, but they're a long way off from protecting you and me online. One of the few things they can do to protect us is to stage a public awareness campaign -- thus we have Cybersecurity Awareness Month.

Why doesn't Google have a Cybersecurity graphic? Online providers don't want you to think about security. Banks don't want you to think about online security. If you thought about security when you signed up for online banking, you might not do it. Without the regulatory agencies, the banks would leave you liable for all losses -- event those caused by the bank's own security lapses, as happened in the UK.

A banking-industry consultant at the same conference said two striking things:

1. Bank marketers fought tooth and nail against FFIEC regulations requiring two-factor authentication for online banking logons. (That means you need your password AND something else to log on.) Banking marketers want to make easy for you (or a hacker) to log on and transfer funds.


2. Banking customer service representatives are just as dumb as the customers when it comes to online security.

If your bank account gets hacked, your bank isn't going to be of much help. They might get some money back, but in most cases, they won't. Your money's gone. The same goes for any other account of yours that gets hacked, whether it's Facebook, GMail, or Yahoo. Nobody's going to help you much.

So take the time now to do a few things to ensure your online security.

1. Use antivirus and make sure it's up to date. If you're on Windows, there are several free antivirus packages available, such as Microsoft Security Essentials , Avast , and Avira . Password-stealing viruses infect computers every day. If you want to tweak out on antivirus effectiveness comparisons, go here.


2. Patch your computer. It doesn't matter if you're windows, mac, unix, linux or bsd. Patch.


3. Change your banking password. Change your email password, because all your password resets go there. Change your security questions, because those reset your passwords. If you're using the same password from college, and your college system gets hacked and reveals your password, then they will find your other accounts.


4. Realize that you are a hundred times more likely to fall for a phishing email than you are to click on an online ad. (Phishing emails are now so common that you might get one that coincides with a recent transaction, making you think it's real.) Now that banks have increased their online security, the hackers are targeting you -- the soft spot.


5. Also realize there are are now office buildings full of professional hackers working in shifts trying to get to your money. (Another panelist, Chris Roberts, talked about research he had done observing the building in an unnamed country in Eastern Europe. Some of his work is available on McAfee's hacker-commerce site.)


6. Don't use unsecured wireless networks. Secure your home wireless network. (Replace WEP encryption with WPA or WPA2.)
   

Online Backups with Backblaze: Does it work?

Just because disk space is getting cheap, don't think that storage is cheap. A referral from Slashdot to Backblaze's blog charted the situation out accurately. The cost of a petabyte of storage on raw SATA hard drives is $81,000. On Amazon and EMC, it's $2.8 million. If Backblaze really could create their own enterprise storage devices, then it would be possible to offer backups at $5/month for unlimited storage. Backblaze even offers a 15-day free trial, so I tried it, although I was skeptical.

Catches:
1) You need to use their client.
2) Their client doesn't run on Windows Server or Linux -- just WinXP, Vista, and Macintosh. (Even if you run the installer in XP compatible mode on Windows server, it still doesn't install.)
3) The $5/month is for one computer, not all the computers in my house.
4) rsync doens't work with Windows/samba shares. (You may, however, be able to get rsync to work to a Macintosh. I haven't tested yet.) (Update below: you can install an NFS server onto WindowsXP/Vista to get rsync to work, or you could do it from Windows via an SSH rsync script.)

Solution: I installed it on a Vista workstation, created a share, and copied the few things I really need backed up to it. I also wrote scripts to transfer my PBX backups to my backup and log host and then copy the files from the backup server to the windows share via smbclient. I'll skip the part about configuring password-less logins for SSH via ssh-keygen keys, as well as the kinit for logging into windows via smbclient. (I also never was able to mount.cifs via kinit, just smbclient.)

So what happened when I tried to back up 15 GB on my Vista box to Backblaze? Not much -- the files just transferred. iPod library -- check. Photos -- check. My mrtg indicated that bandwidth increased to about 310 kbps for four days. I was still able to make phone calls via my SIP trunk to vitelity with no problems. (g729 to my SIP provider and alaw to my friends' PBX servers via IPSec VPN.)

Bandwidth used:
`Weekly' Graph (30 Minute Average)


Max Average Current
In 501.9 kb/s (0.5%) 56.4 kb/s (0.1%) 47.9 kb/s (0.0%)
Out 1360.2 kb/s (1.4%) 168.6 kb/s (0.2%) 29.3 kb/s (0.0%)

Security comments: Backblaze says it encrypts files, but doesn't offer details on the algorithm or implementation. (e.g AES-CBC, etc.) Backblaze does offer you the option of using a private key, so that only you (assuming you don't forget the key) can access your files.

My advice: If it needs to stay secure, encrypt the files yourself before they hit the local hard disk. You can even do a loopback mount (Super-awesome tutorial there) to an AES-encrypted file on a samba share, and rsync will work, but the whole file will change, requiring it all to be sent to Backblaze.

Update: You can use rsync to get your Linux/BSD/Unix files over to your windows box, but you'll need to install an NFS Server on your windows box. You could also use Microsoft's Services for Unix, but it's easier just using the Allegro server.

Running pfSense on a WatchGuard x700 firewall

The original Firebox X series is nearing its end of life, so I was able to purchase an x700 on eBay for a song. Watchguard is no longer providing updates for it as of October 2009, so you might start seeing more of them on eBay. The original Watchguard X-series consisted of the x500, x700, x1000, and x2500. Since they were software-upgradable, I am assuming the hardware for all is identical. (Warning: WatchGuard has many stickers on the box and more inside indicating that opening the box or removing any hardware voids the warranty.) Why would I buy an end-of-life firewall? Because it's great hardware to run pfSense, one of the best open-source firewall packages available.

WARNING: You should not pay more than $100 for the x500 through x2500. They're end-of-life. And don't let anyone confuse an even older, 2RU model, with the x-series.

What's the hardware? (Boot console text here, photos here.) It's a 1.2 GHz Celeron processor, 256 MB of PC133 RAM, a SanDisk 64MB "Industrial Grade" compact flash card, an Intel Motherboard with six RealTek (!) LAN ports. There's a serial port, two fans, a PCI slot, and a mini-PCI slot occupied by a SafeNet SafeXcel 1141 v.1.1 card. Unfortunately, the SafeXcel 1141 is not supported in FreeBSD even thought the boot shows that it found something. (Maybe OpenBSD...)

Just to be sure, I did the OpenSSL speed check from the Watchguard's console after I installed pfSense.

Firebox Xseries, SafeNet 1141 v1.1 installed:
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128 cbc 26917.66k 27987.75k 28248.74k 28359.04k 28376.05k
aes-192 cbc 22900.88k 23917.93k 24122.93k 24210.67k 24213.70k
aes-256 cbc 20624.38k 21210.58k 21364.18k 21430.52k 21439.17k

Firebox Xseries, Safenet removed:
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128 cbc 26924.00k 27986.35k 28249.58k 28358.33k 28374.19k
aes-192 cbc 22911.56k 23924.39k 24126.03k 24212.23k 24220.05k
aes-256 cbc 20624.28k 21211.41k 21360.39k 21429.22k 21439.88k

While there's no difference, it blows away my Alix 2d3 Board with the Soekris mini-PCI HiFn 7955 card. This is
what I'm currently running pfSense on. To be fair, it was still live and sending 310k/sec to Backblaze, but that's another story.

Alix board w/Soekris VPN Card:
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128 cbc 5186.19k 5310.60k 5423.67k 5473.56k 5487.10k
aes-192 cbc 4548.62k 4671.94k 4780.79k 4802.54k 4801.88k
aes-256 cbc 4117.25k 4157.12k 4243.48k 4263.61k 4257.15k

Finally, some commentary on the original Watchguard Firebox series:
I used one of these at a former client's main site. The user interface was great, and it offered superb logging and even offered a realtime view of connections that I have seen no-one duplicate since. That much let me identify info leak attempts in real time, because I did not leave all outbound ports open.

However, Watchguard included the encryption acceleration hardware, but didn't let you use it without an additional licen$e. pfSense is free.

Update -- Receiving HDTV signals in Washington, DC

I was disappointed with my reception using my old Radio Shack 15-1868 indoor rabbit-ears antenna. I didn't receive channel 66, and I wanted to at least see what was on channel 66. I received the three major networks and Fox fine, but none of the fringe-UHF stations the FCC said I should be able to receive at my location.

I bought another Radio Shack antenna -- this time the "Outdoor HTDV Antenna," SKU number 15-2152 with the last money left on my two-year-old Radio Shack gift card. Since it was a huge, 80-something-inch antenna, it must receive better, right? I mounted it in my attic, two floors above the indoor rabbit ears, and hooked it up via quad-shield RG58. Excitedly, I ran back down to my man-cave and scanned through the channels again.

At first, I lost channels 4, 26, and 32. After minor direction adjustments, I received all previous channels as well as 23 (analog), 25 (analog), 30, 47 and 66. While there's nothing I'm going to be watching on channel 66, I definitely receive more stations than before. Best of all, the new stations are free, so I'm one step closer to eliminating my cable bill.

If you can run a cable to your attic, and you have a traditional, non-steel roof (like the asphalt shingle here), then you can mount your antenna there. I'm not about to distract from the beauty of my Icom Discone mounted on my chimney with a cheap aluminum TV antenna. And that antenna is cheap and I got aluminum dust all over my hands assembling it. Since it's going to stay in a corner of my attic, I don't mind. If you're looking for the 15-2152 antenna on Radio Shack's site, it's gone. It was discontinued and cheap to buy.

Airport AWOS Frequency/Phone KML file updated

I have updated the AWOS frequency, phone, phone and type file based on the FAA's 56-day subscription data. However, instead of using latitude and longitude included in the file, I joined the main airport file by airport ID. This requires making an assumption that all AWOSes are at the airport they serve. While that might not be the case, I get 643 more airports into the file then before. The AWOS file has 2,185 entries of which 1,542 have latitude and longitude. Almost every corresponding airport in the APT file has latitude and longitude. Thus, DCA will now show up with a phone number, but no frequency. The new KML will also show the AWOS type: ASOS, AWOS-1, AWOS-2, AWOS-3, or AWOS-A.

Finally, matching airports to AWOS increases the complexity. I had hoped to write a simple script that does text manipulation for all KML files -- one script altering one FAA text file to produce one KML file. As with the Special Use Airspace, however, there wasn't a good way to do it without using a database and relations.

Find them in the KML archive.

Brand Dimensions still won't stop scraping my site

Despite having a no BDFetch robots.txt directive, Brand Dimensions has downloaded hundreds of my original pages with photos on them. None of these pages mention any brand names of any companies, so I'm curious as to what BD is really doing. I'm guessing they could also provide some serious competitive intelligence to their clients. I just wonder what happens when they represent competing companies, like Coke and Pepsi. Here are some representative entries from my log files:

/var/log/httpd/access_log.1:72.14.164.139 - - [11/Aug/2009:07:25:27 -0400] "GET /carleton/reunionweb/WebPage-Full.00001.html HTTP/1.1" 200 1394 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.140 - - [11/Aug/2009:07:25:42 -0400] "GET /carleton/reunionweb/WebPage-Full.00015.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.197 - - [11/Aug/2009:07:25:57 -0400] "GET /carleton/reunionweb/WebPage-Full.00018.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.157 - - [11/Aug/2009:07:26:12 -0400] "GET /carleton/reunionweb/WebPage-Thumb.00023.html HTTP/1.1" 200 3648 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.179 - - [11/Aug/2009:07:26:27 -0400] "GET /carleton/reunionweb/WebPage-Full.00013.html HTTP/1.1" 200 1468 "-" "LinkWalker/2.0"
/var/log/httpd/access_log.1:72.14.164.193 - - [11/Aug/2009:07:26:42 -0400] "GET /skiing/webdest/WebPage-Full.00011.html HTTP/

Brand Dimensions switched the name of their bot to sidestep robots.txt directives. Based on my own Google Analytics info, I can safely say a lot of people are interested in what Brand Dimensions is doing and how to stop it. More LinkWalker info here. Other webmasters report that the LinkWalker agent is also used by spambots harvesting email addresses for phishing attacks and the like.

Here are my latest robots.txt lines:
User-agent: BDFetch
Disallow: /
User-agent: BPImageWalker
Disallow: /
User-agent: VoilaBot
Disallow: /
User-Agent: LinkWalker/2.0
Disallow: /
User-Agent: LinkWalker
Disallow: /

Airspace, Special Use, and Airport KML files updated (not all US Airports are in the Western Hemisphere)

I updated the Airspace, Special Use, and Airport KML files for the FAA data covering the 56 Day Subscription from August 27 2009 to October 22 2009. I have also created batch files to call all the scripts, and I'm working on a T-SQL script to clear the tables and do a bulk import from the files.

Also, I corrected one incorrect assumption. Not all US Airports are in the western hemisphere. I assumed all longitudes were W and put a minus sign in the conversion to Google/KML decimal format from the FAA's all-seconds format. Wrong. In the updated file, MAJ (MARSHALL ISLANDS INTL) is in the right place. I'm still not sure Pago-Pago is correct, however.

The index of files is here.

Class B, C, D and experimental E Airspace KML files posted

FAA subscription data includes .shp (shape) files for ESRI products. Google Earth Pro ($400/year) is capable of opening these shape files directly, but the default setting is to make them opaque. Thus the class B airspace looks like one single area with no details. FWTools is a free open-source conversion tool to change GIS formats. It also does a much better job of converting the data, showing each individual airpspace and its low and high altitudes. The FAA marks the class E files as experimental, NOT TO BE USED FOR NAVIGATION.

The FWTools command to change from the FAA's .shp format to KML is as follows:
ogr2ogr -f "KML" C:\airports\class_b.kml C:\airports\class_b.shp The FAA lists airspace in order (e.g. Washington Class B Area A, Washington Class B Area B...), and fwtools maintains this order. When Google Earth displays this information, the last one listed goes on "top" of the area. Thus when you click on the center of a Class B zone, the widest area comes up, NOT the SFC to 10000 area. You can see the correct area when you click on the corresponding area layer, however. (I am currently working on a fix for this, but it requires some custom XML and SQL programming that's going to take some time.)

However, there is a weakness in Google Earth. The KML specification does not allow the creation of floating polygons. All non-plane polygons are extruded from the surface or ocean floor. Thus, there's no way to express in KML the upside-down-wedding-cake shape of your typical Bravo airspace. This may one day change, however, because Google is already adding its own extensions to the KML specification.

If you don't want to bother, just download the Airpspace KML files here: Archive link.The usual warning applies -- DO NOT USE FOR NAVIGATION. The KML archive is here.

Special Use Airspace in Google Earth

According to the FAA, there are 977 U.S. special use airspaces around the world. (Some are international, like the Pacific off of Guam.) By Special Use airspace, I mean the following FAA Types: ALERT AREA, MILITARY OPERATIONS AREA, PROHIBITED AREA, RESTRICTED AREA, WARNING AREA. (Another side note: It's not just the FAA that loves ALL CAPS. The Department State also uses ALL CAPS in its cables.) There are many areas of the United States, in CONUS, Alaska and Hawaii, that are special use.

The FAA flat-fixed file that is the source of this data is so convoluted that I have a new-found respect for companies like Jeppesen that produce the data in a readable format every 56-day cycle. The FAA stuffs 8 different tables into one giant flat file with a well-documented layout.

Google Earth is not perfect with its polygons, either. It's got bugs, especially when it comes to KML stylesheets. I wrote the style tags with VBScript out of SQL, so they are all the same, but in Google Earth, they don't all look the way they're supposed to. It feels like troubleshooting early versions of Netscape and IE when writing HTML. And no, I didn't use some fancy XSL to transform my queries into KML (XML). I used VBScript because it's the tool I know and it's fast.

The usual warning applies: DO NOT USE FOR NAVIGATION. My code isn't perfect, and neither is the FAA's data. There is no substitute for a pre-flight briefing. I ask about the airspace every time flying out of the ADIZ.

Download the file here: US Special Use Airspace.kml.

US Airports by Region in Google Earth & Maps via KML

Given that my other x64 4GB computer hung this morning when loading the large KML file this morning, I decided to break up the KML file into the following FAA regions:ALASKA, CENTRAL, EASTERN, GREAT LAKES, NEW ENGLAND, NORTHWEST MOUNTAIN, SOUTHERN, SOUTHWEST and WESTERN-PACIFIC. (Once again, the FAA really really likes ALL CAPS.)

I also set up a directory to hold all my KML files, including the original all US airports and the AWOS file at this location. To view in Google Maps rather than Google Earth, add "http://maps.google.com/?q=" to the front of the URL.

View all US Airports on Google Earth via KML

I finally took the time to write the script to take the raw National Flight Data Center APT.TXT file and write it straight to a Google Earth and Maps-compliant KML file. It's pretty basic XML, with a little math to convert the FAA's all-seconds format into decimal coordinates.

A word of warning: DO NOT USE FOR NAVIGATION. This is unverified data from the FAA, and my math is often accurate to about a factor of ten. Also, there are 13,569 airports, which means there are 13,569 points in this file. It's enough to max out the processor running Google Earth on your computer. However, when you zoom in to a single state, it will speed up because most points are off-screen. I filtered out non-airport landing facilities: BALLOONPORT, SEAPLANE BASE, GLIDERPORT, HELIPORT, STOLPORT, ULTRALIGHT, so you will find none of those in this file. (The FAA likes things in ALL CAPS.)

The pop-up description of each data point includes the airport code, airport name, UNICOM frequencies, CTAF, and pulic or private. Download the US Airports KML file here.

Here's a screenshot of Google Earth with the KML loaded:

Displaying FAA Airport AWOS Data in Google Earth and Google Maps with KML

The FAA releases updated Airport data every 56 days via the National Flight Data Center Portal. However, since government contracts apparently don't have any usability requirements, it's difficult to navigate, and the data is still in a fixed-field format reminiscent of cobol and mainframes. (Which, by the way, is a format that plays a huge role in banking even in 2009.)

After cleaning it up and matching airports with their AWOS data, I created a KML file. KML is simple XML -- you just need to do get the elements correct. The AWOS frequency and phone number is in the pop-up for each point.

With this file you can see all airports for which the FAA has latitude and longitude data. The FAA doesn't have coordinates for all airports, so many are missing. (Why they don't have coordinates for major airports like DCA in the AWOS file is beyond me. They have them in the airport facilities file.)

To view it in Google Maps (if you don't have Google Earth), use the
http://maps.google.com/?q=http://cw.sampas.net/kml/US_AWOS_20090827-20091022.kml format, which will pull the KML from the link after the ?q=.

Coming soon: VBScript to take the raw FAA files and produce KML. VBScript isn't the best way to export text to XML, but it is easy to write and very quick.

Logparser: find which computers are locking your windows accounts.

To find account lockout events on multiple domain controllers, download logparser 2.2 and execute the following command in a domain admin context (e.g. runas user:domain\administrator logparser.exe), where the part below the command is in "lockouts.sql". The account lockout event is 644 -- if you need to find others, read Microsoft's KB174074 Also, this script will access each domain controller's security event log sequentially, so if you're in a hurry, execute several different logparser processes for each domain controller.

logparser.exe file:c:\scripts\logparser\lockouts.sql -i:EVT -o:datagrid

------stick this part in lockouts.sql
SELECT
timegenerated AS LogonTime,
extract_token(strings, 0, '|') AS UserName,
message as Message
FROM \\domaincontroller1\security, \\domaincontroller2\security, \\domaincontrolle2\Security
WHERE EventID = 644
-----end here

If you want the output to go into a database instead of a datagrid (Excel-type) table, make the logparser command look like this:

logparser.exe file:c:\scripts\logparser\lockouts.sql -o:SQL -server:myDBservername driver:"SQL Server" -database:myDBname -createtable:ON

Table name will end up matching your dbname. Set -createtable to off after you run it once.

Props to: Microsoft's Log Parser Toolkit, by Gabriele Giuseppini and Mark Burnett.

If you're going to be doing anything with windows logs, buy the book. It's more useful than several log management software packages I've demo'ed.

Download Log Parser here.

Brand Dimensions' bot stops, but Brand Dimensions doesn't.

Now that I added user-agent: BDFetch, disallow / to my robots.txt, all the BDFetch bot gets is robots.txt. However, some people Brand Dimensions is now browsing my blog:

72.14.164.134 - - [08/Jun/2009:13:58:26 -0400] "GET /blog HTTP/1.1" 301 314 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
72.14.164.133 - - [08/Jun/2009:13:58:26 -0400] "GET /blog/ HTTP/1.1" 200 57901 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
72.14.164.196 - - [08/Jun/2009:13:58:42 -0400] "GET /blog/2009/06/comcast-is-collecting-data-on.html HTTP/1.1" 200 16335 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"

A little research reveals that they have a class C block of IPs:
CustName: Brandimensions Inc.
Address: 5090 Explorer Drive
Address: Suite 203
City: Mississauga
StateProv: ON
PostalCode: L4W-4T9
Country: CA
RegDate: 2008-06-25
Updated: 2008-06-25
NetRange: 72.14.164.0 - 72.14.164.255
CIDR: 72.14.164.0/24

Thus blocking 72.14.164.0/24 at the firewall will prevent them from seeing anything.

Update on Washington DC Television Transition: OTA channel assignments

The information below is what the FCC has for the Washington DMA. If you ask most television stations what their frequency is, they don't have a clue. The FCC also says I should receive all of these stations at my house. These stations are VHF and UHF, so your standard TV antenna should work fine (see study). I contend there is no such thing as a "high-def" antenna. Radio waves are radio waves. The frequency is the same. If you're experiencing multi-path errors, then try using a directional antenna. (How do you determine if you're reception problems are multi-path errors? Well in the analog days, multi-path was ghosting. Generally, you should be getting good reception because you're close, but you can't lock on.)

If you're wondering why you can't receive a station, here's a study by the FCC on DTV reception in Washington, DC. If you have a hundred thousand dollars in equipment you'll be able to replicate their results: You're less likely to receive DTV signals than you were analog signals. You can improve your chances with a thirty-foot mast.

For the tripod-mounted, indoor-type antennas, SPI was 86% for WUSA and 84% for WRC when the better of either the bowtie antenna or the Silver Sensor directional antenna was used. These SPIs for the combined indoor antenna types exceed the above values for mast-mounted antenna reception computed in the 1998 study.

Station	Network	Analog Channel	Digital Channel Pre Transition	Digital Channel Post Transition	Virtual Channel	Transition Date
WRC	NBC	4	48	48	4-1	6/12/2009
WTTG	FOX	5	36	36	5-1	6/12/2009
WJLA	ABC	7	39	7	7-1	6/12/2009
WUSA	CBS	9	34	9	9-1	6/12/2009
WFDC	UNIVISION	14	15	15	14-1	6/12/2009
WDCA		20	35	35	20-1	6/12/2009
WHAG	NBC	25	55	26	25-1	6/12/2009
WETA	PBS	26	27	27	26-1	6/12/2009
WWPB	PBS	31	44	44	31-1	6/12/2009
WHUT	PBS	32	33	33	32-1	6/12/2009
WVPY	PBS	42	21	21	42-1	6/12/2009
WDCW	CW	50	51	50	50-1	6/12/2009
WWPX	ION	60	12	12	60-1	6/12/2009
WFPT	PBS	62	28	28	62-1	4/16/2009
WPXW	ION	66	43	34	66-1	6/12/2009

Comcast is collecting data on my blog: How to stop Brand Dimensions

I just noticed some interesting entries in my logs from a new bot: BDFetch. Brand Dimensions is a company that collects information from the Internet, looking for bad things people say about Comcast's poor service. Apparently, they're conducting private investigations for U.S. clients from Canada by collecting files on everyone that says something about Comcast online. Personally, I'd rather not be investigated by a Canadian company in a state that requires licensing for such activities.

Here are the entries from my web server's access log. Clearly, they're looking only at content that mentions Comcast. I mention other brand names, but they're not interested in them.

72.14.164.176 - - [05/Jun/2009:16:49:17 -0400] "GET /robots.txt HTTP/1.1" 200 289 "www.brandimensions.com" "BDFetch"
72.14.164.150 - - [05/Jun/2009:16:49:39 -0400] "GET /blog/2008/11/comcast-strikes-back.html HTTP/1.1" 200 17006 "www.brandimensions.com" "BDFetch"

Since then, I have two new lines in my robots.txt file:

User-agent: BDFetch
Disallow: /

If that doesn't work, I'm going to cut off the 72.14.164.0/24 network at my firewall.

Wondering if Brand Dimension is watching you? Here's the grep command to find them:
grep BDFetch access_log
You'll need access to your web log. Also, remember that grep and unix are case-sensitive.

Washington DC OTA DTV Channel Line-up: cheaper than cable with more stations

Many cable companies have a government-mandated monopoly. In exchange, they offer "basic cable" which was supposed to be a modestly priced service that includes little more than you could receive over the air. With the advent of DTV, does basic cable grow to include high-definition television? Of course not! Cable companies now charge more for you to receive lower-quality signals than you can receive over the airwaves. They also require you to rent an HD set-top box. If you have an HDTV manufactured in recent years, you can get it free over the air. Also, most of the local stations in DC offer more than one programming feed over the air. Cable companies charge extra for those same free signals.



WUSA and WRC are changing their frequencies on June 12, so be sure to scan through on your HDTV/receiver. Others may be changing, but information is not easy to find on their sites.

Station/Affiliation	Old Analog Number	Digital Channel	Programming, Format	Online Listings	Reponse to questions?
WRC/NBC	4	48 (674-680 MHz UHF)	4.1 HD 1080i 16:9 4.2 Weather 480i 4:3 4.3 Sports 480i 4:3 (mostly skiing this winter)	No	no response
WTTG/FOX	5	36 (602-608 MHz UHF)	Single DTV channel 720p 16:9	No	Fullest, fastest response to email inquiry.
WJLA/ABC	7	39 (620-626 MHz UHF)	7.1 HD 720p 16:9 7.2 Weather 480i 4:3 7.3 Classic TV 480i 4:3	No	no response
WUSA/CBS	9	34 (590-596 MHz UHF)	9.1 HD 1080i 16:9 9.2 Weather 480i 4:3 Transitioning back to VHF channel 9 on June 12th.	No	no response
WFDC	14		14.1 480i 4:3	?
WDCA	20	35 (596-602 MHz)	20.1 HD 720p 16:9	Yes
WETA/PBS	26 Analog goodnight 6/12/2009	27 (548-554 MHz)	26.1 HD 26.2 Create 26.3 Kids 26.4 WETA	Yes	Reponded, but without frequency information.
WHUT	32	33 (584-590 MHz)	32.1 480i 4:3	Yes
WDCW	50	51	?		unreceivable

Why the telemarketers aren't stopping.

June 16, 2009 update: Fox News has a story on the auto-warranty Scammers.

With the implementation of the National Do-Not-Call registry, you'd think that telemarketing activity would decrease. Instead, telemarketing activity is increasing. Of course, it's impossible to measure, because there are no reliable statistics for illegitimate telemarketing activity. By illegitimate, I mean not just that they're not supposed to call you, but they're trying to scam you with a bogus auto warranty, fake sweepstakes winnings, or fake identity-theft protection services.

The cost of calling has dropped dramatically. With SIP trunking and the g729 codec, I can squeeze a hundred calls across a T1. The SIP trunk will cost me $.01 per call-minute. An open-source PBX will cost a few hundred dollars for server hardware. Robocalling and autodialing scripts are free. Add in a kilobuck hardware codec card, and I can start calling every number there is. A thousand sixty second robocalls cost me $10.00 and take only ten minutes to complete at a hundred concurrent calls a minute. My caller ID is whatever I enter into the caller id field. Faking caller id is trivial and legal. Even if only one in a thousand calls hooks me up to a sucker, I'm making money.

If I'm a telescammer, I'm not really concerned with the do not call list. If I'm faking my ID, what are you going to do -- report a number? A company that doesn't exist? I'm practically untraceable. If you *69 me, all you get is the faked caller ID number. You'd need a trap-and-trace from your phone company, and you can't do that without a threat. Even if you do get one, I've already called. I'm not going to call again, and if I do, it'll be from a different number. In Canada, the do-not-call list is a service for scammers to get Canadian phone numbers.

The economics make it almost as cheap as spam, and spam is a LOT easier to block than roboscammers. Blacklisting phone numbers, or even faked caller-id numbers is not easy. There are several free web services tracking this type of information, like whocalled.us and 800notes.com. However, there's no update service to add this to phone blocklists, which don't exist. Vonage won't let you blacklist numbers unless you get their Wifi hardware phone. Even Verizon's anonymous call blocking permits obviously bad 000-000-0000 numbers through.

Freepbx has a great blacklist, but it blacklists only known bad numbers. What we really need is a shared database of bad originating numbers.

Enumerate many Active Directory Groups at once

Here's another simple script that will simply write out your AD group memberships to a csv file with the name of a group. Input is a simple text file with one group name per line. This script is adapted from the original at "WiseSoft":http://www.wisesoft.co.uk/scripts/vbscript_list_group_members.aspx.

' VBScript source code
' takes a list of groups in a text file and dumps out a text file with each group's membership.
Set objFSO = CreateObject("Scripting.FileSystemObject")
'change this line to wherever you want to read the input from.
Set objTextFile = objFSO.OpenTextFile("c:\scripts\groups\groups.txt",1)

Do Until objTextFile.AtEndOfStream

groupName = objTextFile.Readline
'Debug.WriteLine groupname
If groupName = "" Then
wscript.quit
End if

groupPath = getgrouppath(groupName)
'Debug.WriteLine groupPath
If groupPath = "" then
wscript.echo "Unable to find the specified group in the domain"
wscript.quit
End if

Set objGroup = getobject(grouppath)
Set objFSO2 = createobject("scripting.filesystemobject")
'change the path to where you want the output files to go.
Set objFile = objFSO2.createtextfile("c:\scripts\groups\" & groupname & ".csv")
q = """"


objFile.WriteLine(q & "sAMAccountName" & q & "," & q & "Surname" & q & "," & q & "FirstName" & q)
For each objMember in objGroup.Members
objFile.WriteLine(q & objmember.samaccountname & q & "," & q & objmember.sn & _
q & "," & q & objmember.givenName & q)
Next

Loop
Set objFile=nothing
'***** Users who's primary group is set to the given group need to be enumerated seperatly.*****
getPrimaryGroupMembers groupName


wscript.echo "Completed"

Function getGroupPath(byval GroupName)
Set cmd=createobject("ADODB.Command")
set cn=createobject("ADODB.Connection")
set rs=createobject("ADODB.Recordset")

cn.open "Provider=ADsDSOObject;"

cmd.commandtext = "SELECT adspath from 'LDAP://" & getnc & _
"' WHERE objectCategory = 'Group' and sAMAccountName = '" & groupname & "'"
cmd.activeconnection = cn

set rs = cmd.execute

if rs.bof <> true and rs.eof<>true then
getgrouppath=rs(0)
else
getgrouppath = ""
end if
cn.close

End function

Function getNC
set objRoot=getobject("LDAP://RootDSE")
getNC=objRoot.get("defaultNamingContext")
End function

Function getPrimaryGroupMembers(byval GroupName)
set cn = createobject("ADODB.Connection")
set cmd = createobject("ADODB.Command")
set rs = createobject("ADODB.Recordset")

cn.open "Provider=ADsDSOObject;"
cmd.activeconnection=cn

'***** Change the Page Size to overcome the 1000 record limitation *****
cmd.properties("page size")=1
cmd.commandtext = "SELECT PrimaryGroupToken FROM 'LDAP://" & getnc & _
"' WHERE sAMAccountName = '" & GroupName & "'"
Set rs = cmd.execute

If rs.eof<>true and rs.bof<>true Then
PrimaryGroupID = rs(0)
Else
Err.Raise 5000, "getPrimaryGroupMembers", "Unable to find PrimaryGroupToken property"
end If

cmd.commandtext = "SELECT samaccountname, sn, givenName, distinguishedName FROM 'LDAP://" & getNC & _
"' WHERE PrimaryGroupID = '" & PrimaryGroupID & "'"

set rs = cmd.execute

while rs.eof<>true and rs.bof<>true
objFile.WriteLine(q & rs("samaccountname") & q & "," & q & rs("sn") & q & _
"," & q & rs("givenName") & q & "," & q & rs("distinguishedName"))
rs.movenext
Wend
cn.close

End Function

How to dump your web hosting provider in one line.

I used to host with Interland, which is now web.com. They just sent me an email telling me my monthly Windows hosting plan monthly price would double because FrontPage extensions are end-of-life on Linux. Thus I needed to download everything via FTP, because FrontPage will always choke before it completes. Windows GUI FTP clients are no longer free -- and they also choke before finishing 300 MB and thousands of files. (I checked and ChiliSoft ASP isn't supported/sold by Sun anymore, either.)

wget -r ftp://userxxx:password@ftp.mysite.com

Who knew wget could use ftp and be recursive?

Verizon Rejects My Port Order

After a couple of months, Verizon rejected my port order. I don't really understand why, other the Verizon's internal processes can't handle it, because, well, some number are portable but not others. Below is what Vitelity told me:

This order was rejected by the losing carrier. The rejection reason from them is \"The customer had the number converted to a dry loop, it is not portable now. What the customer needed to do was have the DSL transferred to a dry loop and the number converted to a PSTN. Instead, the customer converted the number itself to a dry loop.\"

Thank you, Verizon.