Monday, March 26, 2012

Beginning NE Tallahassee

I have reviewed and/or mapped, in Tallahassee, all roads east of Monroe Street, North & West of Capital Circle Southeast, and south of Tennessee Street. Last weekend I began moving north of Tennessee Street, where I will focus on everything South and West of Capital Circle, and East of Monroe Street.

My ultimate goal is to have all streets within the Capital Circle cup and South of the Interstate reviewed and mapped. The idea is that within Tallahassee, the maps should by 99.999% reliable. Once this area has been mapped I'll begin seriously moving outside of that circle and getting areas like Buck Head and Killearn. I will also begin an attempt at getting all buildings and POIs marked within the circle.

I've added some building and businesses in areas I frequent, such as around Apalachee Parkway.

Tuesday, January 10, 2012

Copy and paste from the system clipboard with vim

Often, when using vim, highlighting text in the terminal to copy-and-paste it around is plausible.  This is definitely true when on a true terminal.  I use the rnu option so that I have relative line numbers on each line.  So copying multiple lines with the mouse grabs the line number with unwanted indentation.  When I need to move code around it is annoying to have to manually remove the numbers.

To understand how to get vim's clipboard to match your system's you need to understand vim registers.  On a computer we usually only get one clipboard.  Every time we ctrl-c, the contents of the clipboard are discarded and replaced with whatever is highlighted.  We don't have any option to copy multiple objects and then paste them around.  However vim has multiple "registers" where text can be copied and pasted from.  To see the registers type :reg in command mode.

The register that we are interested in is register +.  To test it out, copy some text from another application then run :reg and see the contents displayed in register +.

So now we just need to know how to access the contents of register +.  Register access is done with ".  To paste from they keyboard we type "+p  To copy, for example an entire line, into the system clipboard we type "+yy  To delete the current line and store it in the system clipboard we type "+dd

We can also use registers with visual mode.  If we wanted to copy the line underneath the cursor we would type V"+y  We could copy the next three lines into the system clipboard by typing Vjj"+y

To sum it up, to use the system clipboard to copy and paste in vim simply do what you would usually do in vim, then prepend "+ to your y (yank) or p (paste) command.

Update:
Depending on the version of vim you are using, the register for the clipboard may work differently. In some cases the * and + registers are the same register. If you want to find out which register is for your system clipboard, simply copy some text from anywhere into your clipboard, then run :reg. Whichever register has the text you copied is the register that holds the contents of your clipboard.

Changing priorities for a bit.

I'm going to cease work on beefing up the Tallahassee map and fixing MapDust bug in Florida to start addressing area that the OSM Inspector shows needs to be addressed before the licensing change.  Given that this is time sensitive it will be my new effort up until April 1.

Thursday, December 15, 2011

Finding duplicate MapDust bugs.


One big issue that the MapQuest app seems to have is that it sometimes will report a single button multiple times at the exact same lat/long. Often one of the duplicates will have the description the user entered, if one was entered at all, and the other bugs will have seemingly random Types, with the default text for that type in the bug report.

In my effort to squash all reported in Florida, sometimes I just go looking for invalid bugs so that when I'm in the mood to do some real fixing its easier to find actual problems. In that spirit I have whipped up a quick way to find duplicate bugs so that the dupes can easily be marked invalid so that the original can later be individually verified.

To find duplicate bugs, first download the latest MySQL database extract from http://www.mapdust.com/dumps/latest_mapdust.sql.tar.gz Unzip the database file in the tarball to wherever is convenient for you.

Install MySQL on your system and then run the following command:
CREATE DATABASE mapdust

Now edit the database extract and add the following to the first line. 
use mapdust;

Now, while in the same directory as the extract file run the following command
mysql -u root -ppassword < latest_mapdust.sql

Replace the word password above with your root password, being careful to not put a space between -p and your password.

Now we are ready to query our new database for duplicates.
use mapdust;
SELECT id,
    latitude,
    longitude,
    bug_count

    FROM
    (
        select latitude, 
            longitude, 
            count(*) bug_count 
            FROM osmexportbug 
            GROUP BY latitude, 
                longitude 
    ) foo
        JOIN osmexportbug 
            USING (latitude, longitude)
    WHERE bug_count > 1
        ORDER BY bug_count, 
            id;

You should now have the bug number and lat/long of the duplicates. The ones that belong together are easy to spot because they are sequentially together. I use the following query to only find duplicates in Florida.
SELECT id,
    latitude,
    longitude,
    bug_count
    FROM     ( 
        select latitude, 
            longitude, 
            count(*) bug_count 
            FROM osmexportbug 
            WHERE administrative_area = 'Florida' 
            GROUP BY latitude, 
                longitude 
    ) foo 
        JOIN osmexportbug 
            USING (latitude, longitude) 
    WHERE bug_count > 1 
    ORDER BY bug_count, 
        id;

After locating a strand of duplicate bugs I go through and mark all but the lowest numbered bug as a duplicate. I use language like the following:
Duplicate of bug 85064.
http://www.mapdust.com/detail/85064


Later when I come across the original bug, I'll only have the one to verify and squash, not a bunch after it to go through and tediously mark invalid.

I hope this helps any other bug squashers out there looking to address the bugs the awesome MapQuest users are out there reporting for us.

Sunday, December 11, 2011

Out of town mapping

This weekend I went down to the West Palm Beach area to celebrate my sister-in-laws graduation from FAU. My father-in-law noticed that his son's neighborhood was not completely mapped in either Bing or on Google. Bing is missing some streets, and Google is missing some street names. I pulled up OSM and the situation was even worse. No streets where mapped at all. Apparently this place was only built around two years ago.

Like a good OSMer, I began tracing the missing streets using the areal imagery. When I was done with that I downloaded the area into JOSM, loaded the kids into the car, and drove around, marking the names of the streets as we campused the neighborhood. When we got back we uploaded the edit, and now OSM has the only 100% mapping of the area.

I mapped some other missing streets, but there is no chance of me getting the chance to visit the areas to get street names, mostly because of time issues, but also, these are gated communities where I wouldn't be allowed in to anyways.

Thursday, November 10, 2011

Steve Jobs was wrong on Flash

With Adobe's recent announcement that they will discontinue supporting Flash on mobile devices, a lot of people are declaring that Jobs was right on Flash. I say he was wrong and here's why.
Before we really start, lets set the record straight about Flash. Flash allowed web developers to do things that where impossible in HTML. This was good, but many developers got carried away and created monstrosities, building websites void of HTML and completely annoying to use. This gave Flash a bad name amongst real web developers who saw it as a cop-out for developers. However it allowed games and videos to exist in a space where it would have otherwise to deliver such content. As HTML5 has been rolled out amongst every major browser web developers have been looking forward to the death of Flash.
I fall in the "death to Flash" camp, but I still believe Jobs was wrong. Computing is a user-centric experience. If the computer isn't doing what the user wants, then the computer isn't really doing its job. Designers of software are in a position where they have to balance making everything excessively configurable, such that a user try to set settings becomes confused, and making decisions for the user. This is a pretty delicate balance, and one that is fraught with inevitably some minority of users becoming upset at the choices that are made.
The direction Apple has chosen with the iPhone is to be the ultimate gate keeper, keeping their users behind a walled garden where Apple dictates what will be allowed, and how what is allowed will behave. Break Apple's rules and you get the boot, or are not even let in in the first place. This is a fundamental design decision Apple has made that has driven their product every step of the way. Because Apple holds the key they keep the bad stuff out and only allow in what they deem to be good enough for their users. Before the iPhone users have had a chance to speak their voice, Apple has decided if a voice will even be allowed in the first place. This is what Apple did with Flash. They declared Flash on mobile to be a dead end, and didn't allow their users to choose for themselves if Flash on their iPhone was good or bad. Jobs, and through him Apple, may have been right that Flash on mobile devices isn't a good idea. They where wrong, however, to not allow Flash on the iPhone. If users didn't like it, nobody had to make them use it. If nobody used it then no harm-no foul.
We can only hope that over time Flash goes the way of Shockwave, but in the meantime, lets let those purchase computing devices make that decision, not the manufacturers of those devices. It is my inherent freedom, as the purchaser of a device, to use it in any lawful way I choose. It is unethical for the makers of that device to micro-manage those choices for me, barring me from the decision making process.

Saturday, October 29, 2011

UEFI Secure Boot Controversy Explained

There's been a lot of buzz and hyperbole spread out over the Internet regarding a new UEFI specification called secure boot. The concern isn't with secure boot itself but with its potential to prevent computers from running any operating system other than Microsoft Windows 8. I've discussed, read, and debated this issue quite a bit and wanted to throw formalize the situation as I understand it. I hope that this explanation is clear, factual and not favoring any political agenda.

Secure Boot is a new feature where a computer will only boot an operating system that contains a digital signature known to the motherboard. But what does that even mean? In digital cryptography there is an idea of what is called a public key and a private key. The author of a digital file can sign that file with their own private key. This key, as the name suggests, is private to the author. Only the person with that key can sign the file as long as that key stays private. Once it is made public the game is off.

The public key can be used to authorize the file. Using the public key I can verify that it is indeed signed by the person claiming to be the author. To play that explanation back to the real world, Microsoft would sign its Windows 8 operating system with its private key. Any operating system with this signature attached to it must have come from Microsoft as only they have the key. The computer hardware itself would have a copy of Microsoft's public key, which it can then use to verify that the operating system is genuinely from Microsoft. It any unauthorized modifications have been made to it the verification process would fail, and the computer would fail to boot. This would both insure that a malicious program hasn't modified Windows 8 in a way that is harmful to the user, and would also help verify that the system is not pirated.

Here's where the controversy comes in. What if I buy a machine configured with secure boot and I wanted to run something else, such as a GNU/Linux distribution, Solaris, BSD, or my own homemade operating system? I couldn't, unless it was signed with a private key, and the computer had the public key to verify it with. Or, if the user could simply disable secure boot, then they could run a different operating system. Let's pause for a moment now. Secure boot can be a good thing. Ensuring that a verified operating system is running on your machine can only be good for you. Boot sector viruses can cause all sorts of mayhem. You only want on your machine the things that you have authorized to be on your machine. If something gets installed on your machine that you didn't authorize, the fact it found a backdoor into your machine is a good sign that was only put there with malicious intent. With that, let's continue.

So let's explore the option of disabling secure boot. The first problem with this is that the ability to disable secure boot is not a requirement, and it is likely that many vendors will ship hardware without this option. The desire to disable secure boot to install something other than Windows is, admittedly, a niche desire amongst consumers. Vendors have a tendency to not cater to the niche. They focus on delivering the pieces that the masses will want. How this will really play out is yet to be seen. Perhaps all PCs shipping with secure boot (which is likely to soon be all PCs) will have the ability to disable it, but it seems reasonably likely this may not be the case.

But even if all PCs shipped with secure boot have the ability to disable it, we still aren't in the scenario we want to be in. Remember, secure boot is a good thing. If the only way to install GNU/Linux is to disable secure boot, it means that I am missing out on an important feature of my hardware. Why can't I enjoy the freedoms and security of Linux, and the security of secure boot? There's no technical reason why this can't happen. But first, my computer needs the public key to whatever third party OS I choose. This leads me into my next piece, and is absolutely the most crucial part.

How does my PC with secure boot receive the public keys that list the operating systems that are allowed to boot? Microsoft's recommendation, and requirement to be Windows 8 certified, is for the hardware vendors to provide this. This is called being a certificate authority, or CA. A CA has a list of whitelist, or allowed, public keys and who they belong to. They also provide a blacklist of, or denied, public keys. This puts the hardware vendor in control of what can and cannot run on the hardware they ship. But what's wrong with that? The problem are many, but I'm going to only focus on the two major issues.

The first problem is that you own that hardware. You bought it. It is yours and it should obey your commands. If you can only do with your hardware what the manufacturer says, then it hardly really belongs to you. Like having to ask your landlord permission to paint your living room, you will have to ask Dell, or Intel, or whomever anytime you want to do something that they didn't preapprove. This is a horrible way to move forward with technology. It is a fundamental freedom of ownership that you get to do what you want with it, within the limits of the law. If you wanted to run something they didn't preapprove, what kind of hurdles do you think you would run through trying to get in touch with your computer manufacturer and getting them to add new keys, which they could only push out to your machine, because you wouldn't want them to be adding keys for any consumer who just calls in and asks them to do so.

This could have a disastrous effect on the young tinkerers who like to play with hardware and make it do all the things it's not supposed to do so that they can one day because the future Steve Jobs and Larry Schmidts of the world. The ability of the young to toy with hardware and software in weird ways is how visionaries are born. The second problem with making computer manufacturers the CA is the barrier it creates to new entrants in the market. Imagine that you created a really great products, say a new operating system, and your just this geek living in your mother's apartment, but you know if you can just get it out there the world will be at your feet. But how? How are you, Mr(s). Nobody, going to convince all the different manufacturers out there to review your software, to make sure you aren't just some schmoe trying to get bad things on people's computers, and then get them all to add your key to their customer's hardware? The answer is simple. You won't. And whether or not your idea would have flopped, or transformed the world doesn't matter. You where stopped before you where even allowed to try. This is a horrible artificial barrier to progress and innovation.

The good news is that it is not necessary. All we simply need to do is for hardware vendors to make consumers the CA. This may sound difficult, what does the average human being know about being a certificate authority? People want to spend their time in front of their computer using it, not messing around with blacklists and whitelists for public keys. That's simply not feasible.

However, it is possible that your computer can have a standard means of accepting and denying keys upon request. Let's start with the scenario of a new PC owner booting up their Windows 8 machine for the first time. The bootup initial bootup process of the installer could request the computer to add the Microsoft key. This would then create a prompt for the user to either accept or deny the key. Most people would just accept it and continue, no big deal. Someone like myself may deny it, I don't want to run Window (actually I wouldn't purchase a PC that even comes with Windows, but that's another story.) For the person who denies it, they may put the CD into their machine that has the installer for their OS of choice. That installer would have its own public key, which would then prompt the user to accept, and then could boot that OS. It's even reasonable that antivirus programs could help the user keep the whitelist and blacklist up to date, requesting the computer to add whitelist and blacklist keys, and each time the user would accept or deny the request.

This puts the user in control. Only the owner of the hardware has the authority to accept or deny software from running on their own system, and it doesn't have to be complicated. It also allows all software vendors to play ball on an equal field.

I hope you find this post to clear and lacking of hyperbole. It is not an attempt to call foul on Microsoft or anyone else. I do not believe anyone is acting in bad faith with secure boot, but I do believe we can have user freedom, security, and robust software competition all at the same time.

Sunday, October 2, 2011

Big Fix

I came upon MapDust bug 209335, http://www.mapdust.com/detail/209335, which noted some misplaced and missing streets. Further review of satellite imagery showed that there was a massive issue in this area where the Tiger data was missing lots of streets, and the streets it did have where grossly misplaced and mis-shaped. The problem only seemed to exist on Pleasent Hill Road north of Southport Road and South of John Young Parkway, but that area is pretty large.

Anyhow, after many hours of fixing, redrawing, and adding roads, I was finally able to close out the bug today.

After the hiatus of squashing a bunch of MapDust bugs, and marking many more invalid, I'm back to mapping out my homeland. I've already fixed up a bunch of roads, and added others. Soon I will need to do some on-the-ground checking to get some road names and find out what's going on with roads that, from the aerial imagery, don't seem to actually exist.

Saturday, September 3, 2011

Squashing bugs

Today I spent a good deal of time addressing bugs in Florida reported mapdust.com MapDust is a great tool for crowd-sourcing the location of problems, and providing the tools for cartographers to address them. There are a number of issues with MapDust. There is no ability to have a back-and-forth dialogue with the person who reported the issue. Lots of bugs do not contain sufficient information to know what issue the reporter may have been wanting resolved. Sometimes a scan of the area with the Bing aerial imagery where the bug report was filed will show where there could be improvements made. In those scenerios I just do cleanup of the area and then close the bug.

I have cleared out all bugs in the Florida Panhandle, and did spent most of my efforts today in South-West Florida. There where lots of very useful bug reports in the Fort Meyers and Naples area, so I addressed every single one of them. I would like to see a day where Florida's bug count was at zero.

I think having a fast turn-around time for bug resolution will help with the perception that OSM is a good source for mapping data, and that it's open nature empowers problems to be resolved quicker than proprietary mappers can provide.